Comparative Evaluation and Arenas

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against Python 3.12 and the Python standard library (math, random, statistics, hashlib), SciPy’s scipy.stats.binomtest, and the LiteLLM 1.103.2 gateway and Langfuse platform 4.49.0 mentioned in passing; the full list is the version baseline on the landing page. The Python module in this page uses only the standard library, and its numeric results were produced by running it once on synthetic battles generated inside the script, which shows that the arithmetic is internally consistent and nothing else. The other snippets were written from the official documentation of the libraries they call and were not run from this repository. The sample sizes and ratings in the text come from that synthetic data or from formulas; they are not measurements of any real model and no benchmark figure appears on this page.

Absolute scores (a 1-to-5 rating, a pass rate against a rubric) answer "how good is this answer?", a question that humans and judge models both answer inconsistently. A comparison answers "which of these two is better for this question?", which is much easier to decide and much more stable between raters. That is the idea behind public arenas, and it works just as well as an internal tool for the choices the DocsAssistant team makes every month: new prompt or old, hosted model or local one, reranker on or off, chunk size 400 or 800. Chip Huyen’s AI Engineering presents comparative evaluation in this role alongside the other evaluation approaches, and the Chatbot Arena work by Zheng et al. and Chiang et al. is the reference for ranking models from crowdsourced pairwise votes. The estimators below are textbook statistics; this page paraphrases the ideas and writes its own code.

Why compare instead of score

Scoring one answer in isolation forces the rater to hold a hidden scale in mind. Two raters disagree about what a "4" means, the same rater drifts during a session, and a judge model tends to give most answers the same generous score. Putting two answers side by side removes the need for a scale: the rater only has to decide which one is better, and when they are equally good, to say so. The cost is that you get relative information only. A comparison tells you that variant B beats variant A, not that either is good enough to ship; it complements the absolute gates from Evaluation in CI, and it cannot replace a golden set built as on Building Eval Sets.

Comparisons fit these decisions well:

  • Choosing between candidates when no rubric captures the difference (tone, completeness, helpfulness of a citation).

  • Tracking a leaderboard of many variants in one number per variant, so a new candidate is slotted in with a few hundred battles instead of being scored against every old one.

  • Auditing a judge model. Pairwise judging by a model and by humans can be compared on the same pairs; the biases and the calibration of the judge are covered in LLM-as-a-Judge.

They fit these decisions badly: regression gating against a fixed threshold, anything where the criterion is a hard fact (did the answer cite the right page) and anything where both candidates are wrong in different ways, because "less wrong" is not "right".

Pairwise preference data

A battle is a record of one comparison. The minimum schema is a question, the two competitors, the two answers, the verdict and who gave it:

battle_id, ts, question_id, model_a, model_b, answer_a_id, answer_b_id,
position_of_a, verdict (a | b | tie | both_bad), rater_type (human | judge), rater_id

Four details decide whether the data can be trusted:

  • Blind presentation. The rater never sees which variant produced which answer. Names, formatting quirks and signature phrases of a model can leak identity, so normalise formatting where you can.

  • Randomised position. Raters and judge models both favour one slot (usually the first). Assign the order at random and store it, so the effect can be measured and cancelled; the judge-side version of this is described in LLM-as-a-Judge. With a model judge, run each pair twice with the order swapped and count an inconsistent pair as a tie.

  • Ties and double failures are data. Keep tie ("equally good") and both_bad separate. A tie is evidence that two variants are close; both_bad points at a gap in the system rather than a ranking, and it belongs in the backlog for the golden set.

  • Same question, same context. Both answers must come from the same input, the same retrieved passages when the variants differ only in the generator, and the same conversation state. Otherwise the comparison measures the luck of retrieval.

The questions themselves are sampled the way the golden set is, from real traffic and from the documentation, so that the ranking reflects what users ask. A ranking computed on a handful of easy questions says little about the hard ones.

Win rate as the simplest summary

For two variants A and B and a set of decisive battles, the win rate of A is

where and count the battles each variant won. Ties are either dropped (the win rate is then about decisive battles only) or counted as half a win for each side, which pulls the estimate towards one half; pick one rule and keep it. For two variants this is already the whole analysis, and the interval in Confidence interval for a win rate says how much to trust it. With more than two variants a table of pairwise win rates grows quadratically and is hard to read, which is where a single rating per variant helps.

The Elo rating

The Elo system was designed to rank chess players from game results. It keeps one number per competitor and updates it after every game. The expected score of A against B depends only on the rating difference:

so a 400-point advantage means roughly ten to one odds of winning. After a battle with outcome (1 for a win, 0.5 for a tie, 0 for a loss) the rating moves in proportion to the surprise:

and is updated symmetrically with . is the step size: large values react quickly and are noisy, small values are stable and slow. Elo is online: ratings are available after every battle, a new variant starts at a base rating and converges as it plays. That makes it convenient for a live internal arena, and its weaknesses come from the same property:

  • The result depends on the order of battles. Shuffle the same battles and the ratings change a little, because early battles are weighted by ratings that were still wrong.

  • A constant means ratings never settle; they keep moving with the latest results even if the underlying strengths are fixed, which suits chess players who improve and is a nuisance for a frozen model.

  • It carries no uncertainty: two ratings that differ by 20 points may or may not be distinguishable.

The Bradley-Terry model

The Bradley-Terry model describes the same situation statistically and removes the order dependence. Every competitor has a positive strength and the probability that beats is

where is the logistic function. Writing the strengths on the log scale shows that this is a logistic regression whose only features are indicators of "which two competitors played". Given the battle counts, the log-likelihood to maximise is

where is how many times beat . The maximum is found by an iteration that is simple enough to write in a few lines (the minorization-maximization form studied by Hunter, and going back to Zermelo):

with the total wins of and the number of battles between and . After every sweep the strengths are rescaled, because only ratios matter, for example so that their geometric mean is 1 (with the pseudo-count prior used in the code below, a fixed reference player sets the scale instead). Ratings are then displayed on the familiar Elo scale:

so that a 400-point gap still means ten to one odds. The fit uses all the battles at once, does not depend on their order and supports confidence intervals by resampling (see Intervals for ratings). Its assumptions are the ones you must check:

  • A single strength per competitor. If variant A is better on how-to questions and B on troubleshooting, one number hides that. Compute ratings per question category as well as overall, using the intent labels of the golden set.

  • Transitivity. The model implies that if A beats B and B beats C, A beats C. Rock-paper-scissors cycles in the data show up as a poor fit; counting them is a quick sanity check of an arena.

  • Ties. Counting a tie as half a win for each side is the simple convention used in the snippet below. Extensions with an explicit tie parameter exist and are described in the statistics literature; this page does not use them.

  • Connectivity. Every competitor must be linked to the others by some chain of battles, and a competitor that has never lost (or never won) gets an infinite strength estimate. Collect enough battles before reading a rank.

The Chatbot Arena reports (Zheng et al.; Chiang et al.) describe ranking models from crowd votes with this family of methods; consult those papers for the exact estimator and the corrections they apply, which were not re-verified for this page. In day-to-day use the rule of thumb is: Elo for a live dashboard that updates battle by battle, Bradley-Terry for the number you quote in a decision.

An internal arena for the DocsAssistant

The DocsAssistant team does not need a public leaderboard. It needs a controlled loop that compares candidate configurations on its own questions and its own raters.

flowchart LR Q["Question pool
golden set + sampled traffic"] --> P["Pair sampler
random pair, random order"] P --> A["Answer A
variant X"] P --> B["Answer B
variant Y"] A --> R["Blind rater
human or judge model"] B --> R R --> L[("Battle log
verdict + position")] L --> E["Elo (live)
Bradley-Terry (reported)"] E --> D["Dashboard + decision"]

There are three sources of questions and raters, each with its own trade-off:

  • Offline battles on the golden set. Each variant answers every question once; a judge model (or a human panel) compares the answers. It is cheap, repeatable and runs in CI, and it is only as good as the judge.

  • Staff arena. An internal page shows a question and two anonymous answers and asks "which is better?". Engineers and support staff supply real questions and real taste. Participation is the limiting factor, so keep it to one click and show the answers in the same layout.

  • Live A/B in production. Real users are assigned to a variant; the comparison uses behaviour or explicit feedback rather than a side-by-side verdict. That is a different design, described in Deployment and Release Patterns and Online Evaluation and Feedback.

For an A/B test the assignment must be sticky (the same user or conversation always sees the same variant), random (unrelated to anything that affects the outcome) and logged. A deterministic hash of a stable identifier gives all three without storing anything:

import hashlib


def assign_variant(unit_id: str, experiment: str, variants: list[str]) -> str:
    """Deterministic, sticky assignment: the same unit always lands in the same bucket."""
    digest = hashlib.sha256(f"{experiment}:{unit_id}".encode("utf-8")).digest()
    bucket = int.from_bytes(digest[:8], "big") % len(variants)
    return variants[bucket]


variant = assign_variant("conv-8f3a", "answer-style-2026-10", ["control", "concise-prompt"])

Official reference for the hashing module: Python documentation: hashlib. Include the experiment name in the hash so that two experiments do not assign the same users to the same arms, and pick the unit (user, conversation or tenant) according to what must not be mixed: a user who switches variant mid-session contaminates both. When the gateway routes the traffic, record the variant as a label on the request and on the trace so that every metric can be split by it; the gateway side is covered under routing in the LiteLLM and Langfuse documentation listed in the landing page bibliography, and the trace attributes in Observability with OpenTelemetry.

A few operating rules keep an arena honest:

  • Freeze everything that is not the variant under test: retrieval index version, temperature, system prompt, tool set. Record these in the battle log.

  • Mix question sources and keep them stable between rounds, or a rating shift may be a change of questions.

  • Retire stale battles. A model update or an index rebuild changes a variant, so battles against the old version belong to a different competitor; give each version its own identity (docsassistant-prompt-v7).

  • Keep the rater pool diverse and calibrated. If one staff member casts most of the votes, the ranking is that person’s taste; cap votes per rater and compare the verdicts of different raters on a shared set of pairs.

  • Watch for known judge-side biases (length, position, self-preference) before trusting a model-judged arena; they are listed with mitigations on LLM-as-a-Judge.

Computing ratings in Python

The module below is self-contained. elo replays battles in order, bradley_terry fits strengths with the iteration above (regularised by a small prior, described below), wilson and the two sample-size functions implement the formulas of the next sections, and bootstrap_bt resamples the battles to get an interval for each rating. A battle is a tuple (a, b, s) where s is 1.0 if a won, 0.0 if b won and 0.5 for a tie.

The plain maximum-likelihood fit breaks when a variant never wins: its strength converges to 0 and the logarithm that converts it to the Elo scale is undefined (a variant that never loses has the opposite problem). Both cases are common in small arenas and in bootstrap resamples, where a weak variant can easily draw no wins. bradley_terry therefore adds prior pseudo-counts: every player gets 0.5 virtual wins and 0.5 virtual losses against a fixed reference player of strength 1. This acts like a weak prior that pulls ratings with little data towards the middle and keeps every strength finite; with hundreds of real battles per variant its effect on the ranking is negligible. Pass prior=0 to recover the unregularised fit when every variant has both wins and losses.

import math
import random
from collections import defaultdict
from statistics import NormalDist


def elo(battles, k=16.0, base=1000.0):
    """Online Elo: replay battles in order, one update per battle."""
    ratings = defaultdict(lambda: base)
    for a, b, s in battles:
        e_a = 1.0 / (1.0 + 10 ** ((ratings[b] - ratings[a]) / 400.0))
        ratings[a] += k * (s - e_a)
        ratings[b] += k * ((1.0 - s) - (1.0 - e_a))
    return dict(ratings)


def bradley_terry(battles, iters=500, tol=1e-9, prior=0.5):
    """Bradley-Terry strengths via the MM iteration, shown on the Elo scale.

    `prior` adds that many virtual wins and virtual losses for every player against a fixed
    reference of strength 1, so a variant that never wins (or never loses) still gets a finite
    rating instead of a strength of 0 (or infinity). Set prior=0 for the plain maximum-likelihood fit.
    """
    wins = defaultdict(float)
    games = defaultdict(float)  # games[(i, j)] with i < j
    players = set()
    for a, b, s in battles:
        players.update((a, b))
        wins[a] += s
        wins[b] += 1.0 - s
        games[tuple(sorted((a, b)))] += 1.0
    p = {i: 1.0 for i in players}
    for _ in range(iters):
        new = {}
        for i in players:
            denom = 2.0 * prior / (p[i] + 1.0)  # virtual games against the reference player
            for (x, y), n in games.items():
                if i == x:
                    denom += n / (p[i] + p[y])
                elif i == y:
                    denom += n / (p[i] + p[x])
            new[i] = (wins[i] + prior) / denom if denom > 0 else p[i]
        if prior == 0:  # without the reference player only ratios matter: fix the geometric mean at 1
            g = math.exp(sum(math.log(v) for v in new.values()) / len(new))
            new = {i: v / g for i, v in new.items()}
        delta = max(abs(new[i] - p[i]) for i in players)
        p = new
        if delta < tol:
            break
    return {i: 1000.0 + 400.0 * math.log10(v) for i, v in p.items()}


def bootstrap_bt(battles, rounds=200, seed=7):
    """Percentile interval for each Bradley-Terry rating by resampling battles with replacement."""
    rng = random.Random(seed)
    samples = defaultdict(list)
    for _ in range(rounds):
        resample = [rng.choice(battles) for _ in battles]
        for name, r in bradley_terry(resample).items():
            samples[name].append(r)
    out = {}
    for name, xs in samples.items():
        xs.sort()
        out[name] = (xs[int(0.025 * len(xs))], xs[int(0.975 * len(xs)) - 1])
    return out


def wilson(wins, n, confidence=0.95):
    """Wilson score interval for a binomial proportion."""
    z = NormalDist().inv_cdf(0.5 + confidence / 2.0)
    p = wins / n
    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return centre - half, centre + half


def n_for_margin(margin, confidence=0.95, p=0.5):
    """Battles needed so that the normal-approximation half-width is at most `margin`."""
    z = NormalDist().inv_cdf(0.5 + confidence / 2.0)
    return math.ceil(z * z * p * (1 - p) / margin ** 2)


def n_for_win_rate(p1, p0=0.5, alpha=0.05, power=0.8):
    """Decisive battles needed to detect a true win rate p1 against p0 (two-sided, normal approximation)."""
    nd = NormalDist()
    za, zb = nd.inv_cdf(1 - alpha / 2), nd.inv_cdf(power)
    num = za * math.sqrt(p0 * (1 - p0)) + zb * math.sqrt(p1 * (1 - p1))
    return math.ceil(num ** 2 / (p1 - p0) ** 2)

Official references: Python documentation: statistics.NormalDist and Python documentation: random. The Elo, Bradley-Terry and Wilson formulas come from the statistics literature listed under References rather than from a library page.

Using it on the DocsAssistant means turning the battle log into tuples. The usage below stays in plain Python; the data source is whatever store holds the log (a table, a Langfuse score export, a JSONL file):

# Each row: variant_a, variant_b, verdict in {"a", "b", "tie"} (both_bad rows are excluded)
rows = [
    ("prompt-v6", "prompt-v7", "b"),
    ("prompt-v6", "local-qwen", "a"),
    ("prompt-v7", "local-qwen", "a"),
    # ... hundreds more
]
score = {"a": 1.0, "b": 0.0, "tie": 0.5}
battles = [(a, b, score[v]) for a, b, v in rows]

print(sorted(bradley_terry(battles).items(), key=lambda kv: -kv[1]))
print(bootstrap_bt(battles, rounds=500))

Official reference: Python documentation: data structures. Run on synthetic battles among three configurations with known strengths (600 battles, 10 percent ties), the script returned Elo and Bradley-Terry ratings that both ordered the three correctly and differed from each other by a few points, as expected from the order effect and the different estimators. A bootstrap with 100 resamples gave intervals of roughly 20 to 40 points on either side, wide enough that two neighbours 50 points apart were not clearly separated. Those figures describe the synthetic data only; the useful lesson is that a rating without an interval invites over-reading. Adding a fourth configuration that lost all of its eight battles left the earlier ratings in the same order and gave it a finite rating at the bottom, with a much wider bootstrap interval than the others, instead of the ValueError the unregularised fit raises.

Confidence interval for a win rate

Each decisive battle between A and B is a Bernoulli trial with success probability , the true win rate of A. With wins out of decisive battles, the point estimate is and its sampling noise shrinks only as . The textbook Wald interval, , performs badly for small samples and for win rates near 0 or 1. Use the Wilson score interval instead:

where is the standard normal quantile for the chosen confidence level (about 1.96 for 95 percent). The wilson function above implements it. For 62 wins in 100 decisive battles it gives approximately 0.52 to 0.71, so a 62 percent win rate is compatible with a true rate barely above one half; with 310 wins in 500 battles (the same observed rate) the interval narrows to about 0.58 to 0.66. The same rate, five times the data, a far clearer verdict.

The decision rule for "is B really better than A" is: B is preferred when the whole interval for its win rate lies above one half (the lower bound excludes 0.5). Equivalently, an exact binomial (sign) test of the null hypothesis can be run with SciPy:

from scipy.stats import binomtest

result = binomtest(k=62, n=100, p=0.5, alternative="two-sided")
print(result.pvalue)
print(result.proportion_ci(confidence_level=0.95))  # default method is the exact (Clopper-Pearson) interval

Official reference: SciPy: scipy.stats.binomtest. proportion_ci documents an 'exact' default; this page did not verify which other method values the installed version accepts, so use the Wilson function above if you need that interval, or check the SciPy page for your version.

Three cautions apply to every interval here. Battles on the same question by the same rater are not independent, so the effective number of independent observations is lower than the row count; bootstrap by question rather than by battle when questions are repeated. Looking at the interval after every battle and stopping the moment it excludes 0.5 inflates the false-positive rate, so fix the sample size in advance (next section) or use a sequential method designed for early stopping. And testing many variants against each other multiplies the chances of a spurious winner; the more comparisons you make, the stricter each one should be.

How many comparisons are enough

Two questions with different answers.

Precision: how narrow should the interval be? For a win rate near one half, the normal-approximation half-width at confidence and sample size is ; solving for the sample size needed for a half-width of gives

where the worst case is . With 95 percent confidence that is about 385 decisive battles for percentage points, about 1 068 for points and about 97 for points (n_for_margin in the module).

Power: how many to detect a real difference? To distinguish a true win rate from one half with significance and power , the usual normal-approximation formula is

with . For a two-sided 5 percent test with 80 percent power, a true win rate of 70 percent needs about 47 decisive battles, 60 percent needs about 194 and 55 percent needs about 783 (n_for_win_rate). The pattern matters more than the digits: halving the effect size roughly quadruples the data. Ties do not count towards these totals, so a pair of near-identical variants that produces 80 percent ties needs five times the raw battles.

For the DocsAssistant this gives a practical plan:

  1. Decide the smallest win rate that would change the decision. If a prompt rewrite must win at least 60 percent of decisive battles to justify a rollout, plan roughly 200 decisive battles, and more if a large share will be ties.

  2. Sample the questions to be representative; the guidance on size and coverage of the question pool is on Building Eval Sets.

  3. For a many-variant leaderboard, each additional variant adds uncertainty to all of its comparisons; budget battles per pair that matters (the new candidate against the current champion), not per variant.

  4. Offline judge-model battles are cheap, so use them to collect enough data, and spend the scarce human votes on a calibration sample that checks the judge, as in LLM-as-a-Judge.

Intervals for ratings

A rating is a fitted quantity, so its uncertainty comes from the whole dataset. The bootstrap in bootstrap_bt resamples the battles with replacement, refits the model for each resample and reports the 2.5th and 97.5th percentiles of every rating. Two variants whose intervals overlap heavily cannot be ranked with confidence; two whose intervals are far apart can. For a more direct statement, bootstrap the difference between two ratings in each resample and check whether the interval excludes zero. When questions are repeated across battles or raters vote many times, resample questions (and raters) instead of individual battles, as noted above.

Pitfalls

  • Position and verbosity bias. A slot-one preference or a taste for longer answers can decide a ranking. Randomise position, swap orders for judge models, and report ratings with and without a length control (for example comparing answers inside length bands).

  • Self-preference. A judge model tends to favour answers from its own family. Do not let a model judge battles it also competes in, or use judges from several families.

  • Leaderboard overfitting. If the same questions are reused indefinitely, variants get tuned to them. Rotate part of the pool and keep a holdout.

  • Non-stationary competitors. A hosted model that changes silently under a fixed name is a different competitor from one week to the next. Pin versions or timestamp the ratings, and see Drift and Regression Monitoring.

  • Ignoring ties. Dropping ties exaggerates differences; counting them as half-wins understates them. Report the tie rate next to every win rate.

  • Winning on style, losing on facts. A preferred answer may be fluent and wrong. Pair arena results with the reference-based and groundedness checks from Evaluation Methodology and RAG and Agent Evaluation before shipping.

  • Rating for the wrong audience. Staff and public raters like different things. Say whose preferences the number represents; safety and misuse testing of a candidate belongs to AI Red Teaming in the AI Security & Responsible AI sub-section.

Choosing an approach

Situation Use Why

Two variants, one decision

Win rate with a Wilson interval, or a sign test

No model needed; the interval says whether more data is required.

Many variants and a live dashboard

Elo with a small

Online, cheap, order-dependent; good for monitoring, not for the final number.

Many variants and a ranking you will quote

Bradley-Terry with bootstrap intervals

Order-independent, uses all battles, comes with uncertainty.

Different strengths per topic

Ratings per intent category

One number hides that a variant wins on how-to and loses on troubleshooting.

Automated, repeated comparison in CI

Judge-model battles with swapped order

Cheap and repeatable; calibrate the judge against human votes first.

Real users, real outcomes

A/B test on behaviour and explicit feedback

Measures what users do; see the online evaluation page.

References