Benchmarks & Leaderboards

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page describes benchmarks and leaderboards as they stood on 2026-09-28: it names no model’s actual score on any of them, because those numbers move with every model release and would be stale within weeks — what stays useful is knowing what each benchmark measures, why raw leaderboard positions need a skeptical reading, and where to look up whichever numbers are current when you read this.

What the common benchmarks measure

A benchmark is a fixed set of tasks with a scoring rule, used to compare models on one narrow capability. None of them measure "how good is this model" in general; each targets one slice of behavior, and a model can rank very differently across two benchmarks that sound similar:

  • MMLU-Pro is a multiple-choice knowledge-and-reasoning benchmark spanning dozens of academic and professional subjects (law, medicine, physics, history, and more). It is a harder, less saturated successor to the original MMLU: it raises the number of answer options per question from four to ten (cutting the odds of a lucky guess) and removes trivial or ambiguous questions that models had learned to answer without real reasoning.

  • GPQA ("Graduate-Level Google-Proof Q&A") is a smaller, expert-written multiple-choice benchmark in biology, physics and chemistry, deliberately built so that domain non-experts score little better than random chance even with unrestricted web search — it targets genuinely hard reasoning rather than retrievable facts.

  • SWE-bench measures whether a model can resolve real, historical GitHub issues from open-source Python repositories: given an issue and the repository at the commit before the fix, the model has to produce a patch that makes the repository’s own real test suite pass. It is an agentic coding benchmark, not a single-turn question-answering one — the model typically has to explore the repository, run commands and iterate.

  • HumanEval+ (from the EvalPlus project) extends the original HumanEval benchmark of short, self-contained programming problems by adding many more automatically generated test cases per problem. The original HumanEval accepted a solution as long as it passed a handful of example tests; HumanEval+ catches solutions that were only correct on those few examples but fail on edge cases the smaller test set missed.

  • IFEval ("Instruction-Following Eval") measures whether a model obeys precise, mechanically checkable instructions embedded in a prompt — for example "answer in exactly three paragraphs", "do not use the word X", or "wrap the answer in a JSON object with these two keys". Each instruction is verified by a small program rather than graded for content quality, isolating instruction-following from raw knowledge or reasoning.

  • Needle-in-a-haystack tests long-context recall: a fact is inserted at a controlled position inside a long document, and the model is asked a question only that fact answers. It is covered in more depth, alongside the related "lost in the middle" effect, on Context Windows & Memory.

  • MTEB ("Massive Text Embedding Benchmark") is not a benchmark for a chat model at all — it evaluates embedding models across many task types (retrieval, classification, clustering, semantic similarity, re-ranking) and languages, so an embedding model’s MTEB score is comparable only against other embedding models. As of 2026-09-28 the current generation is MTEB v2 — benchmark sets such as MTEB(eng, v2) — together with MMTEB (Massive Multilingual Text Embedding Benchmark), a community-driven expansion to more than 500 tasks in over 250 languages that supersedes the largely English-centric original task set, so check which benchmark version and task subset a quoted MTEB score refers to. The embeddings page covers what an embedding model produces and why retrieval quality depends on it.

Saturation and contamination

Two problems undermine a benchmark’s usefulness over time, and both push in the same direction: a rising score that no longer reflects a rising real capability.

  • Saturation happens once most current frontier models already score very close to the maximum on a benchmark — the remaining gap is too small to distinguish models that actually differ in capability, and small scoring differences become mostly noise. The original MMLU and the original HumanEval both saturated this way over time, which is exactly why MMLU-Pro and HumanEval+ exist: harder variants that restore a meaningful gap between models.

  • Contamination happens when a benchmark’s own questions (or close paraphrases of them) leak into a model’s training data, so the model can appear to "solve" a task it has effectively memorized rather than reasoned through. Contamination is hard to rule out for any benchmark built from public web text, and it is one reason a benchmark score published by a model’s own creator deserves more scrutiny than an independently reproduced one run by a third party under a fixed, published evaluation harness.

Both problems are why a single leaderboard snapshot should never be read as a permanent verdict: a benchmark that looked discriminating a year ago may already be saturating, and a model’s reported number is only as trustworthy as the harness and data hygiene behind it.

Arena leaderboards: pairwise preference and the Bradley-Terry model

Static benchmarks like the ones above run a fixed test set once. Arena leaderboards instead collect ongoing pairwise preferences: a person is shown two anonymous models' answers to the same prompt and picks the one they prefer (or ties), across a large volume of independent comparisons. Turning many pairwise votes into a single per-model score is a ranking problem, and the standard tool for it is the Bradley-Terry model, the same statistical model behind the Elo rating system used in chess: each model i is assigned a latent strength score , and the probability that model i is preferred over model j in a single head-to-head comparison is

where is the logistic function, . Only the difference in strength between the two models drives the win probability: two models with the same gap in have the same expected win rate regardless of where that gap sits on the scale, which is what lets the ratings be reported on an arbitrary, chess-like numeric scale (higher score, more often preferred) instead of a raw percentage. Fitting the scores that best explain the observed match outcomes — typically by maximum likelihood over all the collected pairwise votes — is what an arena leaderboard’s published rating actually is: not a benchmark score on fixed test questions, but a strength estimate derived from how often humans preferred one model’s answer over another’s, across many different prompts and topics.

How to read Arena, Artificial Analysis, and SWE-bench

  • Arena (https://arena.ai/leaderboard; formerly LMArena, and before that Chatbot Arena, renamed to Arena in January 2026) publishes Bradley-Terry-style ratings from human pairwise votes across several separate leaderboards (general chat, coding, agentic tasks, image and video generation, and more). Because the rating comes from human preference rather than a graded answer key, it captures things like tone, formatting and perceived helpfulness that a fixed-answer benchmark cannot, but it also means the ranking reflects what human raters prefer, which is not always the same thing as which answer is correct — a fluent, confident, wrong answer can win a vote against a terser but correct one. Read a category’s ranking as "preferred by this platform’s rater population on this category of task", filtered to whichever models and categories you actually care about, rather than as a single global ordering.

  • Artificial Analysis (https://artificialanalysis.ai/leaderboards/models) combines benchmark-style "intelligence" scores with independently measured operational metrics for the same models — output speed (tokens per second), latency (time to first token), cost per task, and context window — and lets you filter and sort by whichever dimension matters for a given use case. Its value for model selection is less "which model is smartest" and more the trade-off surface: which models sit near the Pareto frontier of intelligence versus cost or latency for your workload, a trade-off the Cost, Latency & Throughput page covers in more depth.

  • SWE-bench (https://www.swebench.com/) publishes separate official leaderboards per benchmark variant (the original set, "Verified" — a human-filtered subset removing under-specified or unsolvable issues — plus "Lite", "Multilingual" and "Multimodal" variants), each scored by the fraction of issues an entrant resolved. Because entries mix a base model with a particular agent scaffold (the tool-calling loop, prompting and retrieval strategy wrapped around it), a SWE-bench leaderboard position reflects that whole system, not the underlying model in isolation — two entries built on the same model but a different scaffold can land far apart.

The Hugging Face Open LLM Leaderboard is retired: its first version was archived in June 2024 and replaced by a v2 leaderboard, which was itself retired in March 2025. The v1 documentation is kept only as a historical archive at Hugging Face’s Open LLM Leaderboard archive. Do not link or cite either version as a current, maintained leaderboard. Embedding models, by contrast, still have an actively maintained MTEB v2 / MMTEB leaderboard (https://huggingface.co/mteb).

Why your own evals still matter

Every leaderboard above measures models against tasks someone else chose, on data your own application never touches. None of MMLU-Pro, GPQA, SWE-bench, HumanEval+, IFEval or an arena rating tells you how a candidate model performs on your prompts, your documents, your users' phrasing, or the specific failure modes your application cannot tolerate. A model that leads every public leaderboard can still underperform a lower-ranked one on a narrow, domain-specific task it was never evaluated on publicly, and a model that regresses on a public benchmark between versions can still be a strict improvement for a task that benchmark does not cover at all. Building and running your own evaluation suite — a small, representative set of your own prompts with a scoring method you trust — is the only way to close that gap, and is covered in depth by LLMOps: Building Eval Sets; Choosing a Model covers where that step fits in an overall model-selection workflow.

Example: a Bradley-Terry / Elo update from pairwise preferences

The following script takes a handful of pairwise preference outcomes between three toy models and fits Bradley-Terry strength scores by maximizing the log-likelihood of the observed outcomes, using scipy.optimize rather than a hand-rolled gradient step. The data deliberately includes one comparison that model_a loses: if a model won every comparison it appeared in, the likelihood would keep rising as its rivals' scores fell, so no finite maximum-likelihood estimate would exist and the optimizer would drift towards minus infinity.

import numpy as np
from scipy.optimize import minimize

models = ["model_a", "model_b", "model_c"]

# Each row is (winner_index, loser_index); a real arena logs many thousands of these.
outcomes = [
    (0, 1),  # model_a beat model_b
    (0, 1),
    (1, 0),  # model_b beat model_a
    (1, 2),  # model_b beat model_c
    (0, 2),  # model_a beat model_c
    (2, 1),  # model_c beat model_b
    (0, 1),
]


def negative_log_likelihood(scores: np.ndarray) -> float:
    """-log P(observed outcomes) under the Bradley-Terry model P(i > j) = sigmoid(s_i - s_j)."""
    total = 0.0
    for winner, loser in outcomes:
        gap = scores[winner] - scores[loser]
        # -log(sigmoid(gap)), written via log1p(exp(-gap)) for numerical stability.
        total += np.log1p(np.exp(-gap))
    return total


# Fix model_a's score at 0 as an arbitrary reference point -- only score *differences* matter.
initial_scores = np.zeros(len(models))
result = minimize(
    lambda free_scores: negative_log_likelihood(np.concatenate(([0.0], free_scores))),
    initial_scores[1:],
    method="BFGS",
)
fitted_scores = np.concatenate(([0.0], result.x))

for name, score in sorted(zip(models, fitted_scores), key=lambda pair: -pair[1]):
    print(f"{name}: {score:.3f}")

# Predicted win probability of model_a over model_c under the fitted scores:
sigmoid = lambda x: 1.0 / (1.0 + np.exp(-x))
p_a_over_c = sigmoid(fitted_scores[0] - fitted_scores[2])
print(f"P(model_a beats model_c) = {p_a_over_c:.3f}")

Running it with NumPy 2.0 and SciPy 1.13 prints:

model_a: 0.000
model_b: -1.326
model_c: -1.651
P(model_a beats model_c) = 0.839

This mirrors, at toy scale, how an arena leaderboard turns a large log of pairwise human votes into a single per-model rating: fix one model as an arbitrary zero point (since only score differences enter ), then find the scores that make the observed win/loss pattern most likely. See SciPy — minimize (BFGS method) for the optimizer used here.

Related pages

References