Evaluation Methodology

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against Python 3.12 with sacrebleu 2.6.0, rouge-score 0.1.2, bert-score 0.3.13, sentence-transformers 6.1.0, transformers 5.18.0 and PyTorch 2.14.1 (see the version baseline for the rest of the sub-section). The package versions were read from PyPI on that date. Every snippet below was written from the official documentation of its library and was documented from official sources only: none was run here, because the semantic metrics download model weights and the perplexity example needs a GPU-sized model. The inputs in the worked examples are hand-written DocsAssistant answers, and no score output is quoted, because a number printed here would suggest a benchmark. The ideas about the difficulty of open-ended evaluation, the split between component and system evaluation and the reference-based versus reference-free distinction are paraphrased from Chapters 3 and 4 of Huyen’s _AI Engineering, listed in the bibliography.

This is the first page of the evaluation group. It builds the vocabulary used by the rest: LLM-as-a-Judge for open-ended answers, Building Eval Sets for the data, and Evaluation in CI for turning scores into a gate.

Why open-ended evaluation is hard

A classifier has a label, a translation system once had a handful of reference translations, and a unit test has an expected value. A DocsAssistant answer has none of these. Ask "how do I enable search on the site?" and a dozen different texts are correct: they differ in length, order, examples and tone, and some include a step the others leave out. Four properties make this difficult.

  • Many correct answers. The space of acceptable outputs is huge, so comparing with one reference text penalises answers that are right and phrased differently. This is the central problem of every lexical metric below.

  • Correctness has several axes. An answer can be faithful to the retrieved passages but irrelevant to the question, relevant but unfaithful, correct but unsafe to show, or right but too long for a chat bubble. A single number hides which axis moved.

  • Outputs are sampled. With a temperature above zero the same input yields a different output on each run, so a score is a random variable. The page on Sampling and Decoding explains where the randomness comes from; later sections show how to deal with it.

  • The reference can be wrong or stale. A golden answer written for last year’s documentation is a wrong reference today. Evaluation data needs maintenance like code, which is the subject of Building Eval Sets.

A further trap is that metrics become targets. Once a team optimises a prompt until a number rises, the number stops measuring quality and starts measuring fit to the eval set (the observation usually called Goodhart’s law). The defence is not a better metric but several independent ones, a held-out set and periodic human review.

Component evals and system evals

An LLM application is a pipeline, and a failure can come from any stage. Two levels of evaluation answer different questions.

Level Question DocsAssistant example

Component eval

Does this one stage do its job on inputs of a known shape, in isolation?

Does the retriever return the page that contains the answer in its top five? Does the query rewriter keep the product name? Does the citation formatter emit valid links?

System (end-to-end) eval

Given the real user input, is the final output good enough for the user?

For "how do I enable search?", does the streamed answer give correct steps with a working citation, within the latency budget?

Component evals are cheap, fast, precise about the culprit and deterministic when the component is (a retriever, a parser, a router). They cannot tell you whether the parts compose well. System evals are the only measure of what the user experiences but are slower, often need a judge, and say little about where a failure came from. Use both: component evals to localise and iterate quickly, system evals as the release gate. The metrics for the retrieval and generation components of a RAG pipeline (recall at , MRR, faithfulness) are defined on Evaluating RAG Systems, and agent-specific levels (final response, single step, trajectory) on Agent Evaluation; this sub-section’s RAG and Agent Evaluation page ties them to the gates.

The same logic gives the usual order of work. Start with a small set of end-to-end cases from real questions, find the failing ones, then write a component eval for the stage responsible so the fix cannot silently regress.

Reference-based and reference-free evaluation

A reference-based metric compares the output with a known-good answer (a reference, or gold answer). It is reproducible and cheap but needs labelled data, and its validity is bounded by how well one reference represents the space of correct answers. A reference-free metric scores the output from the input and the output alone, or from the input, the output and the retrieved context. Examples are a judge rubric ("is every claim supported by the passages?"), a perplexity-style likelihood, a format validator, a toxicity classifier, or a check that every cited URL resolves. It needs no gold answers, so it can run on live traffic, but it has no ground truth to anchor it and can drift or be biased, which is why judges are calibrated against human labels (see LLM-as-a-Judge).

Reference-based Reference-free

Needs gold answers

Yes.

No.

Runs on production traffic

Only on the labelled subset.

Yes, on a sample.

Typical metrics

Exact match, F1 over tokens, BLEU, ROUGE, BERTScore, embedding similarity to the reference, pass@k with tests.

Rubric judges, groundedness against retrieved context, format and schema checks, citation validity, refusal detection.

Main risk

Penalises correct paraphrases; rewards copying.

Judge bias and drift; no anchor to the truth.

Most real suites mix them: deterministic checks first (they are free and unambiguous), reference-based metrics where gold answers exist, and judged reference-free metrics for the open-ended remainder.

Functional correctness

When the correctness of an output can be executed, use that. Functional correctness evaluates an output by what it does instead of what it looks like: generated code passes its unit tests, a generated SQL query returns the expected rows, a JSON output validates against its schema, a tool call has the right arguments. It is the most trustworthy kind of evaluation, because it is indifferent to wording. Its limitation is coverage: it applies only where a check can be written.

For the DocsAssistant, which answers questions rather than writing programs, functional checks still cover a large share of the contract. The snippet below is illustrative; the page list and the rules are placeholders for your own site.

import re

KNOWN_PAGES = {  # in practice: load the page list from the Antora build output
    "ai/llmops/index.html",
    "ai/rag-systems/index.html",
    "ai/spring-ai/index.html",
}

URL_RE = re.compile(r"\]\(/?([\w./-]+\.html)\)")


def check_answer(answer: str) -> dict[str, bool]:
    """Deterministic, reference-free checks on one DocsAssistant answer."""
    cited = URL_RE.findall(answer)
    return {
        "has_citation": bool(cited),
        "all_citations_exist": all(c in KNOWN_PAGES for c in cited),
        "within_length": len(answer.split()) <= 250,
        "no_empty_code_fence": "```\n```" not in answer,
    }

This uses only the Python standard library; see the re module documentation.

Unit-test style and the pass@k estimator

For code generation the standard formulation comes from the HumanEval work of Chen et al. A model is asked to produce a solution; the solution is run against unit tests; the pass@k metric is the probability that at least one of sampled solutions passes. Sampling attempts and checking whether any passed gives a high-variance estimate. The paper’s unbiased estimator instead draws samples per problem, counts the that pass, and computes:

The fraction is the probability that a random subset of of the samples contains no passing solution, so one minus it is the probability that it contains at least one. With the expression reduces to , the plain pass rate. The binomial coefficients overflow for large , so the paper gives the numerically stable product form, which is what the code below implements:

import numpy as np


def pass_at_k(n: int, c: int, k: int) -> float:
    """Unbiased estimate of pass@k for one problem.

    n: samples generated, c: samples that passed, k: budget being estimated.
    """
    if n - c < k:  # every size-k subset contains a passing sample
        return 1.0
    return 1.0 - float(np.prod(1.0 - k / np.arange(n - c + 1, n + 1)))


# Mean over a benchmark: one (n, c) pair per problem.
results = [(20, 3), (20, 0), (20, 20), (20, 11)]
score = float(np.mean([pass_at_k(n, c, k=5) for n, c in results]))

This is the estimator from Chen et al., Evaluating Large Language Models Trained on Code; the reference implementation is in the openai/human-eval repository.

Two practical points. First, pass@k answers "can the model solve this given tries and a way to pick the winner"; it is the right metric when a verifier (tests, a type checker) is available to select the passing sample at inference time, and the wrong headline metric for a chat product where the user sees one answer. For that case use . Second, report as well: an estimate from samples per problem is far noisier than one from .

Perplexity

Perplexity measures how well a language model predicts a text. It is the exponentiated average negative log-likelihood of the tokens:

where is the number of tokens and is the probability the model assigns to token given the preceding tokens. It is equal to the exponential of the cross-entropy loss, so lower is better: a perplexity of means the model is, on average, as uncertain as if it chose uniformly among tokens at each step.

Perplexity is a reference-free metric on a held-out text and an excellent one for language-model development: it is how pre-training and fine-tuning runs are monitored, and it is cheap, because it needs one forward pass and no sampling. In application evaluation it has narrow uses: checking that a fine-tuned or quantised model has not degraded on in-domain text (compare perplexity on the DocsAssistant’s own documentation before and after quantisation), and detecting out-of-distribution or garbled input. It does not measure helpfulness or truth, and it depends on the tokenizer: perplexities of models with different vocabularies are not comparable, as the Hugging Face documentation points out.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "openai-community/gpt2"  # replace with the model under test
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).eval()

text = "Antora builds a documentation site from AsciiDoc files in several repositories."
enc = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    out = model(**enc, labels=enc["input_ids"])  # loss = mean negative log-likelihood

perplexity = torch.exp(out.loss).item()

The pattern follows the Hugging Face guide Perplexity of fixed-length models, which also explains why a text longer than the context window must be scored with an overlapping (strided) window instead of disjoint chunks, and gives the sliding-window loop. Use that loop for any real measurement; the snippet above only fits inside one context window. Perplexity cannot be computed this way for a hosted API that does not return token log-probabilities.

Lexical metrics

Lexical metrics compare surface tokens between a candidate and one or more references. They are cheap, deterministic and reproducible, and they are the only ones available when no model can be run. They also share one weakness: a correct paraphrase scores badly, and an incorrect answer that reuses the words of the reference scores well.

Exact match and token F1

Exact match is one if the normalised candidate equals the normalised reference and zero otherwise. It is right for short factual answers (a version number, a command name, a yes/no) and wrong for anything longer. Token F1 relaxes it by comparing the bags of tokens: with the fraction of candidate tokens found in the reference and the fraction of reference tokens found in the candidate,

Both need a normalisation step (lower-casing, stripping punctuation and articles) fixed once and recorded, or the same answers score differently from run to run.

BLEU

BLEU was designed for machine translation. It combines the modified (clipped) n-gram precisions for n-gram orders up to (usually 4) with a brevity penalty that stops very short outputs from scoring well:

where are weights (uniform, ), is the candidate length and the reference length. Precision counts how much of the candidate appears in the reference, so BLEU is precision-oriented. It is a corpus-level statistic: a score on a single sentence is unstable, and a candidate with no matching 4-gram scores zero unless smoothing is applied. Tokenisation changes the number, which is why papers should report the SacreBLEU signature.

from sacrebleu.metrics import BLEU

candidates = [
    "Enable search by adding the lunr extension to the playbook.",
    "Run the build with the fetch flag to pull remote branches.",
]
# One reference set per reference translation; each set aligns with candidates by index.
references = [
    [
        "Add the lunr extension to the playbook to turn search on.",
        "Use the fetch flag when building so that remote branches are updated.",
    ]
]

bleu = BLEU()
result = bleu.corpus_score(candidates, references)
print(result)                 # corpus-level score
print(bleu.get_signature())   # record this with the score so it can be reproduced

The call shape (a list of hypotheses and a list of reference sets) is documented in the SacreBLEU repository; the method is described in Papineni et al., BLEU: a Method for Automatic Evaluation of Machine Translation, linked in the references.

ROUGE

ROUGE was designed for summarisation and is recall-oriented: how much of the reference appears in the candidate. ROUGE-N is n-gram overlap; ROUGE-L uses the longest common subsequence (LCS), which rewards in-order matches without requiring adjacency. With the length of the longest common subsequence of candidate (length ) and reference (length ):

from rouge_score import rouge_scorer

scorer = rouge_scorer.RougeScorer(["rouge1", "rougeL"], use_stemmer=True)

reference = "Add the lunr extension to the playbook to turn search on."
candidate = "Enable search by adding the lunr extension to the playbook."

scores = scorer.score(reference, candidate)  # argument order: reference first, then candidate
for name, s in scores.items():
    print(name, s.precision, s.recall, s.fmeasure)

The reference is the first argument to score; swapping the arguments silently swaps precision and recall. The API is in the rouge-score package documentation. The default tokenizer of this package keeps only Latin alphanumeric characters, so it is not suitable for text in other scripts without a custom tokenizer.

Semantic metrics

Semantic metrics compare meaning instead of tokens, by embedding the texts with a neural network. They fix the paraphrase problem and introduce a model dependency: the score changes when the embedding model does, so the model and its version are part of the metric definition.

BERTScore

BERTScore (Zhang et al.) embeds each token of the candidate and of the reference with a pre-trained contextual model, then matches tokens greedily by cosine similarity. With reference token embeddings and candidate token embeddings :

The embeddings are normalised, so the inner product is a cosine similarity. The paper optionally weights tokens by inverse document frequency, and rescales raw scores against a baseline so that they spread over a more readable range.

from bert_score import score

candidates = ["Enable search by adding the lunr extension to the playbook."]
references = ["Add the lunr extension to the playbook to turn search on."]

P, R, F1 = score(candidates, references, lang="en", rescale_with_baseline=True)
print(F1.mean().item())

bert_score.score returns precision, recall and F1 tensors, one entry per pair; lang selects the default model for the language, and rescale_with_baseline=True rescales with the baseline for that model. For repeated calls, create a BERTScorer object once so the model is not reloaded each time. See the BERTScore repository and Zhang et al.. The first call downloads a model of several hundred megabytes, which matters in CI (see Evaluation in CI).

Embedding similarity

The simplest semantic metric embeds the whole candidate and the whole reference with a sentence-embedding model and takes the cosine of the angle between the two vectors:

It is one vector per text, so it is cheap and easy to cache, and it is the same machinery used for retrieval (the Similarity Metrics and Semantic Search page discusses cosine, dot product and distance), which makes it convenient to reuse the embedding model already in the pipeline. The cosine scale is not uniform across models: a value of 0.8 can mean "paraphrase" in one model and "loosely related" in another, so a threshold has to be chosen against labelled pairs for the model in use.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

candidates = ["Enable search by adding the lunr extension to the playbook."]
references = ["Add the lunr extension to the playbook to turn search on."]

cand_emb = model.encode(candidates)
ref_emb = model.encode(references)

similarity = model.similarity(cand_emb, ref_emb)  # cosine by default; matrix of all pairs
print(similarity.diagonal())                      # pair i with pair i

The encode and similarity calls are shown in the Sentence Transformers documentation page Semantic Textual Similarity. The model name above is the one used in that page; choose a model that matches your language and domain, and record its identifier and revision with every score.

Putting the metrics side by side

Metric Compares Needs Best used for

Functional checks, pass@k

Behaviour

Executable checks or tests

Code, structured output, tool calls, citation validity.

Exact match, token F1

Normalised tokens

Short gold answers

Factoid answers, extraction, classification-like outputs.

BLEU

Clipped n-gram precision

One or more references

Translation-like tasks, compared at corpus level.

ROUGE

N-gram or LCS recall

Reference

Summaries and long-form answers where covering the reference matters.

BERTScore

Contextual token embeddings

Reference and an embedding model

Paraphrase-tolerant comparison with a reference.

Embedding similarity

Sentence embeddings

Reference and an embedding model

Cheap similarity screen; near-duplicate and drift detection.

Perplexity

Likelihood of a text

Open weights with log-probabilities

Language-model and quantisation regressions on in-domain text.

Rubric judge

Output against criteria

A judge model (and optional reference)

Open-ended quality, tone, groundedness; see LLM-as-a-Judge.

When each metric misleads

Every metric above has a known failure mode. Knowing it is what turns a number into evidence. The table uses DocsAssistant examples; the same patterns appear in any domain.

Metric How it misleads Defence

Exact match

Scores a correct answer zero for a trailing full stop, an extra word or a synonym; scores an answer one only if it equals the reference, including when the reference itself is wrong.

Normalise, report token F1 alongside, and use only for short closed answers.

BLEU

Rewards copying phrases from the reference; scores a correct paraphrase near zero; is unstable at sentence level; tokenisation choices change the score. It is built for corpora of translations and is a poor fit for open-ended answers.

Use a standard implementation and record its signature; never use it as the headline metric for chat.

ROUGE

An answer that repeats the reference’s words plus extra wrong ones keeps its recall; a long answer gains recall for free; an answer that negates the reference ("search does not require the extension") shares most of its words with the correct one.

Pair with a precision-side metric or a length limit; add a faithfulness or contradiction check.

BERTScore

Insensitive to small, meaning-flipping edits (negation, a changed number or version), because most tokens still match closely; scores depend on the model and on whether baseline rescaling is used; long texts are truncated to the model’s input limit.

Add exact checks for numbers, names and commands; fix the model and settings, and report them.

Embedding similarity

Related-but-different texts score high ("how to enable search" versus "how to disable search"); the scale is not comparable across models; a long candidate and a short reference compress to the same vector direction.

Calibrate a threshold on labelled pairs; use it as a screen, not a verdict.

Perplexity

Says nothing about correctness: fluent false text has low perplexity. Depends on the tokenizer, so cross-model comparisons are invalid; memorised benchmark text scores deceptively well.

Use only for same-model before/after comparisons on held-out in-domain text.

pass@k

Optimistic when is large and no verifier exists in production; weak tests pass wrong programs; contaminated benchmark problems may be memorised.

Report for products without a verifier; strengthen tests; use private problems.

Rubric judges

Biased towards long, confident, first-listed and self-generated answers; see the next page.

Calibrate against human labels.

A worked example shows why no single reference metric is safe. Take the reference "Search needs the antora-lunr extension registered in the playbook" and two candidates: a correct paraphrase, "Register the extension for lunr in your playbook and search is available", and a wrong answer, "Search does not need the antora-lunr extension registered in the playbook". The wrong answer shares nearly every token with the reference, so ROUGE and BLEU score it above the paraphrase, and BERTScore and embedding similarity place the two close to each other, because negation changes few embedding dimensions. Only a metric that reads the claims — an entailment check or a judge — ranks them correctly. Add a regression case for this exact pattern to the eval set: a minimal pair that differs in one meaning-flipping word, which every metric you adopt must separate. The idea of testing a metric on minimal pairs before trusting it is the same validation step as calibrating a judge.

Choosing metrics for the DocsAssistant

A practical stack, ordered by cost, that later pages build on:

  1. Deterministic checks on every answer: citations exist, length, format, refusal when the retriever returned nothing. Free, reference-free, can run on live traffic.

  2. Component metrics on retrieval: recall at and MRR against labelled passages (see Evaluating RAG Systems).

  3. Reference-based screens on the golden set: token F1 and an embedding similarity, as inexpensive tripwires that flag a large change in answer content and not as quality scores.

  4. Judged, reference-free metrics: faithfulness and relevance with a calibrated rubric judge, the real quality signal.

  5. Human review of a small stratified sample, to calibrate the judge and to catch what none of the metrics see.

Evaluate a metric before relying on it: score a few dozen answers with the metric and with a human, and check the rank correlation. If the metric does not track the humans on your data, remove it from the gate, whatever the literature says. The same discipline for judges, with an agreement statistic, is on LLM-as-a-Judge; the LangChain and Spring AI sides are on Evaluation (LangChain) and Evaluation (Spring AI). Failure modes that metrics cannot catch, such as fabricated claims, are discussed on Hallucinations and Limitations. Security testing and red teaming belong to AI Red Teaming in the AI Security & Responsible AI sub-section and are not covered here.

References