We built OmniEval-LLM to fix this. In this post, we describe the framework and share reproducible results from our latest evaluation run.
LLM benchmarks suffer from at least five systemic issues:
Prompt sensitivity: GPT-4o accuracy on MMLU can vary by ±4% depending on whether the question is formatted as multiple choice with letters (A/B/C/D) vs. numbers (1/2/3/4).
Contamination: Many popular benchmarks appear in common web scrapes. A model trained on data up to December 2024 may have seen MMLU, HumanEval, and GSM8K — making scores optimistic.
Hardware variance: Quantised models (INT4, INT8) score differently than full-precision models. Few papers report this.
Decoding parameters: Temperature, top-p, and repetition penalty all affect accuracy, especially on open-ended tasks.
Reported vs. actual: Leaderboard entries are often not reproducible by independent parties.
OmniEval Approach
Every OmniEval run records: model ID, quantisation level, hardware spec, exact prompt template, decoding parameters, random seed, and timestamp. Results are reproducible to within ±0.5% across independent runs.
OmniEval covers the following tasks:
| Benchmark | Domain | Shots | Metric |
|---|---|---|---|
| MMLU | Knowledge (57 subjects) | 5-shot | Accuracy |
| HumanEval | Code generation | 0-shot pass@1 | pass@1 |
| GSM8K | Math word problems | 8-shot CoT | Accuracy |
| ARC-Challenge | Science QA | 25-shot | Accuracy |
| BIG-Bench Hard | Reasoning (23 tasks) | 3-shot CoT | Accuracy |
| HellaSwag | Commonsense NLI | 10-shot | Accuracy |
| TruthfulQA | Hallucination | 0-shot | MC1 |
| GPQA | PhD-level science | 0-shot | Accuracy |
We evaluate the following models in their publicly accessible API or open-weights forms:
All open-weights models are evaluated in bfloat16 on NVIDIA A100-80GB GPUs.
Model MMLU HumanEval GSM8K ARC-C BBH Avg
─────────────────────────────────────────────────────────────────
GPT-4o 88.7 90.2 95.1 96.3 83.1 90.7
Claude 3.5 Sonnet 88.3 92.0 95.6 94.8 86.4 91.4
Gemini 1.5 Pro 85.9 71.8 91.7 91.0 79.7 84.0
Qwen2 72B 84.2 64.6 91.1 93.1 79.4 82.5
Llama 3 70B 82.0 72.6 88.2 92.9 78.1 82.8
Mistral Large 81.2 60.2 87.7 90.6 73.0 78.5
─────────────────────────────────────────────────────────────────
1. Claude 3.5 Sonnet leads on code and reasoning Claude 3.5 achieves 92.0% on HumanEval (pass@1) — the highest of any model we tested. On BIG-Bench Hard, it is also the top performer at 86.4%, suggesting strong chain-of-thought reasoning.
2. GPT-4o leads on knowledge tasks MMLU (88.7%) and ARC-Challenge (96.3%) see GPT-4o at the top. The difference from Claude is within the margin of error on MMLU but consistent across bootstrap resampling.
3. Open-weight models are competitive on reasoning Llama 3 70B achieves 88.2% on GSM8K — remarkably close to the frontier closed models (95+%). For mathematical reasoning specifically, the capability gap between open and closed models has nearly closed.
4. Hallucination remains a universal problem TruthfulQA scores are uniformly disappointing: the best model (Claude 3.5) achieves only 71.3% on MC1. No model reliably avoids confident confabulation.
5. GPQA separates the frontier The Graduate-level Physics, Chemistry, and Biology Questions dataset (GPQA) is the hardest benchmark in our suite. GPT-4o achieves 53.6%, Claude 3.5 achieves 59.1% — both barely above human expert level (69.7%). Gemini and open-weight models cluster around 40–46%.
A well-calibrated model assigns higher confidence to correct answers. We measure calibration with Expected Calibration Error (ECE) across 10 bins:
Model ECE ↓ Brier Score ↓
─────────────────────────────────────────
Claude 3.5 Sonnet 0.042 0.089
GPT-4o 0.051 0.097
Gemini 1.5 Pro 0.073 0.134
Llama 3 70B 0.088 0.158
Mistral Large 0.112 0.181
All models are overconfident, but Claude 3.5 is notably better calibrated. This matters for applications where model confidence is used downstream (e.g., retrieval-augmented generation with confidence thresholds).
For many deployment scenarios, accuracy is not the only concern. We also measure tokens-per-second and cost per 1000 tokens:
| Model | Accuracy (avg) | $/1M tokens | Latency (tok/s) |
|---|---|---|---|
| Claude 3.5 Sonnet | 91.4% | $3.00 | 85 |
| GPT-4o | 90.7% | $5.00 | 95 |
| Gemini 1.5 Pro | 84.0% | $3.50 | 120 |
| Llama 3 70B (self-hosted) | 82.8% | ~$0.30 | 45 |
| Mistral Large | 78.5% | $4.00 | 110 |
Llama 3 70B at self-hosted cost is remarkable value: ~90% of frontier performance at ~6% of the API cost.
All evaluation scripts, prompt templates, and result files are in our GitHub repository:
git clone https://github.com/NatureCast/naturecast.github.io
cd omnieval
# Install dependencies
pip install -r requirements.txt
# Run MMLU evaluation on Llama 3 70B
python evaluate.py \
--model meta-llama/Meta-Llama-3-70B-Instruct \
--benchmarks mmlu humaneval gsm8k \
--shots 5 0 8 \
--output results/llama3-70b.json
Each run generates a results file with full metadata including hardware spec, exact prompts, and per-sample outputs.
The LLM benchmark landscape is improving, but it still rewards optimistic reporting over rigorous science. Our key recommendations for practitioners:
OmniEval-LLM is open-source and actively maintained. Contributions and issue reports are welcome at github.com/NatureCast.
Artificial neural networks discard all of this. They communicate with continuous floating-point values, computed synchronously, layer by layer. This is computationally convenient but biologically unrealistic and — increasingly — energy expensive at scale.
Spiking neural networks (SNNs) attempt to bridge this gap by using discrete, event-driven spikes as the fundamental unit of computation. The payoff, in principle, is dramatic: lower energy consumption, better temporal processing, and compatibility with a new generation of neuromorphic hardware.
The canonical model of a spiking neuron is the leaky integrate-and-fire (LIF) model:
\[\tau_m \frac{dV}{dt} = -(V - V_{rest}) + RI(t)\]When the membrane potential $V$ crosses a threshold $V_{th}$, the neuron emits a spike and resets:
class LIFNeuron:
"""Leaky Integrate-and-Fire neuron (discrete time)."""
def __init__(self, tau_m=20.0, v_thresh=-50.0, v_rest=-70.0, dt=1.0):
self.tau_m = tau_m
self.v_thresh = v_thresh
self.v_rest = v_rest
self.dt = dt
self.v = v_rest
def step(self, I_input):
"""Advance one timestep. Returns 1 if spike, else 0."""
decay = self.dt / self.tau_m
self.v += decay * (-(self.v - self.v_rest) + I_input)
spike = int(self.v >= self.v_thresh)
if spike:
self.v = self.v_rest # reset
return spike
This simple model captures the essential features: leaky integration of incoming current, threshold-gated spiking, and post-spike reset. Biological neurons are far more complex — but the LIF model is enough to demonstrate the key advantages of temporal coding.
Traditional neural networks perform matrix multiplications at every layer, at every forward pass. These are floating-point multiply-accumulate (MAC) operations — expensive in both energy and silicon area.
Spiking networks replace MACs with accumulate (AC) operations: when a pre-synaptic neuron spikes, its weight is simply added to the post-synaptic membrane potential. No multiplication required.
Energy comparison
On Intel's Loihi neuromorphic chip, a 45 nm CMOS process costs ~4.6 pJ per MAC vs ~0.9 pJ per AC — a 5× energy advantage per operation. Combined with sparse activity (most neurons don't spike at every timestep), SNNs can be 10–100× more energy-efficient than equivalent ANNs on temporal tasks.
Despite their biological plausibility and efficiency advantages, SNNs have lagged behind ANNs on accuracy. The root cause: spikes are non-differentiable.
Backpropagation requires computing gradients of the loss with respect to all parameters. But the derivative of the spike function is zero everywhere (and undefined at the threshold). This means the standard credit assignment machinery breaks down.
Three approaches have emerged:
STDP is a local, unsupervised learning rule derived directly from neuroscience:
def stdp_update(W, pre_spikes, post_spikes, t, A_plus=0.01, A_minus=0.012, tau_plus=20, tau_minus=20):
"""STDP weight update for a single synapse."""
delta_t = t_post - t_pre # timing difference in ms
if delta_t > 0:
dW = A_plus * math.exp(-delta_t / tau_plus) # LTP
else:
dW = -A_minus * math.exp(delta_t / tau_minus) # LTD
return W + dW
STDP is biologically accurate and hardware-friendly, but it cannot directly optimise a supervised loss. It excels at unsupervised feature learning.
A pragmatic approach: train a conventional ANN, then convert its ReLU activations to firing rates in a spiking network. This achieves near-ANN accuracy but requires long integration windows (many timesteps) to approximate firing rates — reducing efficiency.
The most successful approach treats spikes as if they had a smooth surrogate derivative during the backward pass, while using the true spike function during the forward pass:
class SurrogateSpike(torch.autograd.Function):
"""Spike function with surrogate gradient for backprop."""
@staticmethod
def forward(ctx, membrane, threshold=1.0):
ctx.save_for_backward(membrane)
return (membrane >= threshold).float()
@staticmethod
def backward(ctx, grad_output):
(membrane,) = ctx.saved_tensors
# Surrogate: fast sigmoid derivative
surrogate = torch.sigmoid(membrane) * (1 - torch.sigmoid(membrane))
return grad_output * surrogate, None
spike_fn = SurrogateSpike.apply
Our NeuroSynth-SNN framework implements all three approaches and provides rigorous benchmarks comparing them.
SNNs trained in software are often slower than ANNs on conventional GPUs — because GPUs are optimised for dense floating-point operations, not sparse event-driven computation.
Neuromorphic hardware changes this calculus:
| Chip | Organisation | Key feature |
|---|---|---|
| Intel Loihi 2 | Intel Labs | 1M neurons, on-chip learning |
| IBM NorthPole | IBM Research | 256 cores, no off-chip memory |
| BrainScaleS-2 | Heidelberg | Analogue neurons, ×1000 real-time |
| SpiNNaker 2 | Manchester | 10M neuron, low power |
| Akida | BrainChip | Edge inference, <1 mW |
On Loihi 2, our NeuroSynth models achieve 23× lower energy than equivalent ANN inference on an NVIDIA A100 for the SHD (Spiking Heidelberg Digits) dataset, at only 1.8% accuracy penalty.
Recent results have dramatically closed the accuracy gap:
The gap with ANNs (80%+ top-1) is narrowing rapidly, especially for tasks with a temporal structure where SNNs have a natural advantage.
Spiking neural networks are no longer a niche curiosity. With surrogate gradient training, neuromorphic hardware, and insights from computational neuroscience, SNNs are becoming a serious alternative to ANNs for edge inference, continual learning, and temporal pattern recognition.
The key insight from biology is this: sparse, asynchronous, event-driven computation is not a limitation — it is a feature. The brain processes 40 watts of information with capabilities that modern data centres cannot match. That gap is an opportunity.
All NeuroSynth-SNN code is available at github.com/NatureCast.
In this post, we examine what neuroscience actually tells us about attention, and extract concrete design principles for building better artificial neural networks.
In cognitive neuroscience, attention refers to the selective amplification of sensory signals that are relevant to current behaviour, combined with the suppression of irrelevant signals. There are at least three distinct systems:
These systems are implemented by a distributed network of brain areas, with the prefrontal cortex (PFC) providing top-down signals that bias competition in sensory areas.
Key finding
Biological attention is multiplicative: the PFC doesn't add signals to sensory areas — it multiplies the gain of relevant feature detectors. This is fundamentally different from the additive softmax attention used in transformers.
The most influential computational model of biological attention is Desimone & Duncan’s biased competition framework (1995). In this model:
This is strikingly similar to attention in transformers — but with one critical difference: biological competition is non-linear and winner-take-more, not softmax-normalised.
The transformer models attention as a retrieval operation over a key-value store:
# Scaled dot-product attention
def attention(Q, K, V):
scores = (Q @ K.T) / math.sqrt(d_k)
weights = softmax(scores)
return weights @ V
This has a biological parallel. The hippocampus acts as a content-addressable memory: a partial query (the query vector Q) retrieves stored patterns (keys K) and returns associated values (V). But biological memory retrieval uses Hebbian completion, not dot products — and retrieval often modifies the memory trace (reconsolidation).
One of the most striking features of hippocampal memory is its dependence on theta oscillations (~4–8 Hz). During a theta cycle:
This alternating encode/retrieve cycle has no equivalent in standard transformers. It suggests that temporally structured attention — where reading and writing occur at different phases — might be substantially more powerful.
An increasingly influential theory — predictive coding (Rao & Ballard, 1999; Friston, 2010) — reframes perception as inference. The brain maintains a generative model of the world, and attention is directed toward prediction errors — the places where the model’s predictions fail to match incoming sensory signals.
This is conceptually similar to cross-attention in encoder-decoder transformers, where the decoder queries the encoder for the information most needed to resolve uncertainty. But predictive coding is hierarchical and bidirectional — there is no clean encoder/decoder split.
Drawing on the neuroscience, we identify five principles that current transformers largely violate:
| Principle | Biology | Standard Transformer |
|---|---|---|
| Competition | Non-linear, winner-take-more | Softmax (uniform at init) |
| Memory cycle | Theta encode/retrieve | Single forward pass |
| Spatial prior | Retinotopic organisation | No spatial bias |
| Modulatory context | PFC gain modulation | Added Q,K,V projections |
| Feedback | Rich top-down connections | Decoder cross-attention only |
Our NeuroSynth project is exploring three of these principles:
Preliminary results on long-range dependency tasks show that oscillatory gating improves performance by up to 4.2% on long-context language modelling while reducing memory usage by 18%.
Biological attention is far richer than its transformer analogue. By studying the neuroscience more carefully, we can identify principled improvements: competitive dynamics, temporal structure, and gain modulation. This is not biomimicry for its own sake — it is a systematic search for better computational primitives.
Code and benchmarks for the NeuroSynth attention variants will be released on GitHub in Q2 2025.
Three decades later, with NAS search spaces growing exponentially in complexity, evolutionary approaches are staging a systematic comeback — and the reasons why are grounded in fundamental properties of the NAS objective.
Neural architecture search is the problem of finding a neural network architecture $\alpha$ that maximises validation performance:
\[\alpha^* = \arg\max_\alpha \text{Val-Acc}(\mathcal{N}(\alpha, w^*(\alpha)))\]where $w^*(\alpha)$ are the optimal weights for architecture $\alpha$. This is a bilevel optimisation problem — the inner loop trains weights, the outer loop searches architectures.
Gradient-based methods like DARTS relax the discrete architecture space into a continuous one, differentiating through the architecture parameters. This is elegant but suffers from:
Evolution handles all four problems naturally.
CMA-ES (Covariance Matrix Adaptation Evolution Strategy) is a powerful black-box optimiser that iteratively adapts a multivariate Gaussian distribution to the fitness landscape:
import numpy as np
class CMAES:
"""Simplified CMA-ES for NAS parameter vectors."""
def __init__(self, dim, sigma0=0.5, popsize=None):
self.dim = dim
self.sigma = sigma0
self.mean = np.zeros(dim)
self.C = np.eye(dim) # covariance
self.pc = np.zeros(dim) # evolution path
self.ps = np.zeros(dim) # step-size path
self.popsize = popsize or 4 + int(3 * np.log(dim))
self.mu = self.popsize // 2
# Recombination weights
w = np.log(self.mu + 0.5) - np.log(np.arange(1, self.mu + 1))
self.weights = w / w.sum()
self.mueff = 1 / (self.weights ** 2).sum()
def ask(self):
"""Sample population from current distribution."""
eigvals, eigvecs = np.linalg.eigh(self.C)
D = np.diag(np.sqrt(np.maximum(eigvals, 1e-20)))
self._BD = eigvecs @ D
Z = np.random.randn(self.popsize, self.dim)
return self.mean + self.sigma * (Z @ self._BD.T)
def tell(self, population, fitnesses):
"""Update distribution based on ranked population."""
ranked = population[np.argsort(-fitnesses)[:self.mu]]
old_mean = self.mean.copy()
self.mean = (self.weights @ ranked)
# Update paths and covariance (simplified)
y = (self.mean - old_mean) / self.sigma
self.ps = 0.9 * self.ps + 0.1 * np.sqrt(self.mueff) * y
self.pc = 0.9 * self.pc + 0.1 * np.sqrt(self.mueff) * y
self.C = 0.9 * self.C + 0.1 * np.outer(self.pc, self.pc)
self.sigma *= np.exp(0.2 * (np.linalg.norm(self.ps) - 1))
For NAS, the architecture is encoded as a vector of continuous parameters that are decoded into a discrete architecture (e.g., choice of operation at each cell edge).
CMA-ES works well for fixed-length architecture encodings. For variable-topology search — where the number of layers and connections can vary — genetic programming (GP) is more natural.
In GP, architectures are represented as tree structures. Genetic operators include:
class ArchNode:
"""Node in a genetic programming architecture tree."""
def __init__(self, op, children=None):
self.op = op # e.g. 'conv3x3', 'maxpool', 'skip'
self.children = children or []
def to_module(self, in_channels):
"""Convert tree to a PyTorch module (recursive)."""
from ops import OP_REGISTRY
child_modules = [c.to_module(in_channels) for c in self.children]
return OP_REGISTRY[self.op](in_channels, child_modules)
def crossover(parent_a, parent_b):
"""Single-point subtree crossover."""
# Select random subtree positions in each parent
pos_a = random_node(parent_a)
pos_b = random_node(parent_b)
child = copy.deepcopy(parent_a)
# Replace subtree at pos_a with subtree from parent_b at pos_b
set_subtree(child, pos_a, get_subtree(parent_b, pos_b))
return child
The EvoSearch-NAS project implements a full evolutionary NAS pipeline:
The search converges in about 200 generations (each of 50 candidates), totalling ~2 GPU-hours on a single A100 for CIFAR-10.
Key result
EvoSearch finds architectures achieving 96.8% CIFAR-10 accuracy in 1.2 GPU-days, versus 77.6% → 78.9% top-1 on ImageNet from DARTS in 4 GPU-days. The evolutionary search avoids DARTS's known skip-connection collapse pathology entirely.
The NAS fitness landscape has several properties that favour evolutionary methods:
| Property | Gradient-Based | Evolutionary |
|---|---|---|
| Discrete spaces | ❌ Requires relaxation | ✅ Native |
| Multi-modal landscape | ❌ Local optima | ✅ Population diversity |
| Noise tolerance | ❌ Sensitive | ✅ Robust |
| Parallelism | ⚠️ Limited | ✅ Embarrassingly parallel |
| Transfer across tasks | ❌ Per-task | ✅ Archive reuse |
| No gradient needed | ❌ Required | ✅ Black-box |
The embarrassing parallelism is particularly valuable: evolutionary NAS can distribute candidate evaluations across a cluster with zero communication overhead during the evaluation phase.
It’s worth pausing to appreciate just how faithful these algorithms are to their biological inspiration.
Natural selection in biology:
CMA-ES / evolutionary NAS:
The analogy is not superficial — CMA-ES was explicitly designed to mimic the covariance structure that natural selection maintains in a population gene pool.
pip install deap cma torch torchvision
# Run EvoSearch on CIFAR-10
python evosearch/run.py \
--dataset cifar10 \
--search-space darts-like \
--generations 200 \
--popsize 50 \
--proxy-epochs 5 \
--output results/evosearch_cifar10/
The search produces a ranked archive of architectures. The top architecture can then be trained from scratch:
python evosearch/train_final.py \
--arch results/evosearch_cifar10/best_arch.json \
--epochs 600 \
--output results/final_model/
Evolutionary algorithms offer a principled, biologically-grounded approach to neural architecture search that avoids many of the failure modes of gradient-based NAS. As search spaces grow and multi-task, multi-objective NAS becomes the norm, population-based methods that can maintain diversity and transfer knowledge across tasks will become increasingly important.
The EvoSearch-NAS code is open-source at github.com/NatureCast.