Building Eval Sets

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against RAGAS 0.4.3, langsmith 0.14.2, MLflow 3.16.1, LangChain 1.4.3 on Python 3.12, Spring AI 2.0.1 on Spring Boot 4.1.x with Java 21, and JUnit Jupiter 5.x; the full list is the version baseline on the landing page. The snippets were written from the official documentation pages linked after each one and were not run from this repository. The RAGAS, LangSmith and MLflow calls shown are the ones those pages document; where a page did not show a parameter (dataset versions and splits in LangSmith, the exact scorer options in MLflow) the prose says so instead of guessing. The numbers in the worked examples are placeholders to show the arithmetic, not measurements of any real system.

An evaluation is only as good as the questions it asks. The metrics and judges in Evaluation Methodology and LLM-as-a-Judge answer "how good is this answer?"; this page answers the earlier question "which answers do we grade?". The ideas follow the treatment of evaluation datasets in Chip Huyen’s AI Engineering and the production-RAG chapters of Mendelevitch and Bao’s Hands-On RAG for Production (see the bibliography); the code is written from the tools' own documentation.

What an eval set is for

The DocsAssistant needs one artefact that every later page reuses: a golden set, a versioned collection of questions, each with whatever reference is needed to grade an answer. It serves three jobs, and a set built for one often fails at another:

  • Regression gate. Small, fast, stable, run on every pull request (see Evaluation in CI).

  • Model and prompt selection. Larger, with enough items per category to tell two candidates apart.

  • Diagnosis. Tagged so that a drop can be traced to an intent, a document or a failure mode, and so that drift monitoring can compare today’s traffic with the set.

An eval set is not a benchmark. Public leaderboards (see Benchmarks and Leaderboards) measure general ability on someone else’s tasks; the golden set measures your application on your users' questions, which is the only thing a release decision should rest on.

The pipeline from raw material to a gated set is the same each time:

flowchart LR T[Production traffic] --> S[Scrub PII and sample] D[Documentation pages] --> G[Synthetic generation] S --> L[Label and review] G --> L L --> Q{Agreement and
dedup checks} Q -->|pass| V[Version and publish] Q -->|fail| L V --> E[Offline evals and CI gates] E --> F[Failures and feedback] F --> T

Sources of questions

Real traffic

Real questions beat invented ones because they contain the typos, half-questions, product nicknames and wrong assumptions that users actually produce. The DocsAssistant already logs every ask(conversationId, userId, text) call as a trace (see Observability with OpenTelemetry), so the raw material is a query over those traces. Rules that keep the set healthy:

  • Scrub before you store. Remove or replace names, emails, account identifiers and anything pasted from a user’s own environment. Microsoft Presidio is an open-source analyser and anonymiser built for this; automated redaction misses cases, so sample the scrubbed output by hand. Retention and consent obligations belong to your legal and security process; the AI Security & Responsible AI sub-section, for example Sensitive Information and Hidden Context, is the place to read about them.

  • Sample for diversity, not volume. Taking the most recent thousand questions over-represents whatever happened this week. Cluster the question embeddings, or group by the intent label described below, and draw from every cluster.

  • Keep the failures. Questions that received a thumbs-down, an escalation or a retry are worth more per item than the ones that went well. The loop that feeds them back automatically is the evaluation flywheel on Online Evaluation and Feedback.

  • Do not label with the system under test. If the current DocsAssistant produced the "expected" answer, the set measures agreement with itself. Use it to propose a draft and have a person correct it.

  • Record provenance. Keep the trace id, the date range and the sampling rule in each record’s metadata, so the item can be traced and later removed if a deletion request arrives.

Documentation pages

New products and new pages have no traffic yet. For those, derive questions from the documentation itself, with the page as the reference: for each sampled page a person (or a model, reviewed by a person) writes questions whose answer is stated on that page, and records the page URL and the section anchor. This gives a retrieval label for free, which the RAG metrics in RAG and Agent Evaluation need, and the same pages are the corpus described in Ingestion Pipelines and Freshness. Documentation-derived questions are cleaner and more formal than real ones, so treat them as a complement: they check that every page is reachable, not that users phrase things that way.

Synthetic generation

A model can draft questions from your documents at a scale no team can write by hand. RAGAS ships a TestsetGenerator for this; its documented usage loads documents, builds a generator from a language model and an embedding model, and returns a test set that converts to a DataFrame.

from langchain_community.document_loaders import DirectoryLoader
from langchain_openai import ChatOpenAI
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import OpenAIEmbeddings
from ragas.testset import TestsetGenerator
import openai

docs = DirectoryLoader("docs/pages/", glob="**/*.md").load()

generator_llm = LangchainLLMWrapper(ChatOpenAI(model="<generator-model>"))
generator_embeddings = OpenAIEmbeddings(client=openai.OpenAI())

generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
testset = generator.generate_with_langchain_docs(docs, testset_size=50)

df = testset.to_pandas()
df.to_json("drafts/synthetic-draft.jsonl", orient="records", lines=True)

The code derives from the RAGAS page Generate a test set for RAG. Replace <generator-model> with a model you are allowed to send documentation to; any model with a LangChain chat wrapper works, including a local one (see Running LLMs Locally). RAGAS evolves quickly, so confirm the import paths against the version you pin. The columns it returns (user input, reference contexts, reference answer and a synthesizer label) are the RAGAS naming, not a standard; map them to your own schema before storing.

Synthetic data has well-known failure modes. Treat the output as a draft that a person reviews, not as ground truth:

Pitfall What it looks like Mitigation

Same-model bias

The generator and the system under test (or the judge) share a model family, so questions and answers fit that family’s style and the score is inflated.

Use a different model family to generate, or to judge, than the one under test; keep a human-written slice for comparison.

Leakage

The generated questions quote the source sentence, so a retriever matches on shared words and the test is trivial; or the documents later appear in a fine-tuning or prompt-example set.

Rewrite or paraphrase questions; hold documents used in few-shot prompts or training out of the eval; split by source document, not by random row.

Unrealistic phrasing

Fluent, complete, polite questions with no typos, abbreviations or context from earlier turns.

Mix with real traffic; add a paraphrase pass that injects typos, jargon and underspecified follow-ups.

Narrow difficulty

Almost all items are single-fact lookups; multi-page or comparison questions are rare.

Request an explicit type mix (single hop, multi hop, comparison, unanswerable) and count the result.

Wrong or ambiguous references

The "answer" misreads the page, or the page admits two readings.

Human review of every item that enters the gate set; drop questions the reviewers disagree on.

Duplicates

Near-identical questions from neighbouring chunks inflate one topic’s weight.

Deduplicate by embedding similarity above a threshold chosen by inspecting examples.

Duplicates are cheap to find. Embed each question, then flag pairs whose cosine similarity exceeds a threshold, and let a reviewer keep one of each pair. Pick the threshold by reading pairs just above and below it rather than by quoting a universal value; it depends on the embedding model.

Coverage: intents, categories and edge cases

A set that is 80 percent "how do I install" questions will show a strong score while the troubleshooting answers are broken. Coverage is a design decision made before sampling. For the DocsAssistant, a minimum taxonomy:

Dimension Values to cover

Intent

How-to, concept explanation, troubleshooting an error, API reference lookup, comparison between options, migration, "where is X documented".

Reasoning shape

Single page, multiple pages, needs a code snippet, needs a version-specific answer, needs a calculation.

Answerability

Answerable from the docs, partially answerable, not answerable (the correct behaviour is to say so), out of scope (the correct behaviour is to decline or redirect).

Conversation position

First turn, follow-up that depends on an earlier turn (see Conversational RAG and Memory), correction by the user.

Language and style

Languages you support, typos, abbreviations, very short queries, very long pasted logs.

Risk

Prompt-injection attempts and requests to reveal instructions or other users' data (see Guardrails and Prompt Injection), questions touching restricted documents (see Access Control and Privacy).

Two items in the table deserve emphasis. Unanswerable and out-of-scope questions are the cheapest way to detect a system that always produces a confident answer; the expected output for them is a refusal or a "not documented" message, which is a legitimate reference. And edge cases should be collected on purpose: every production incident, every ticket and every feedback item turns into a new row with the tag of the failure it exposed, so the set grows where the system has actually broken.

When the total budget is fixed, allocate items to strata (intent and answerability, for instance) rather than sampling uniformly. With strata and a total of items, a simple allocation gives each stratum a floor and shares the rest by traffic weight:

where is the share of production traffic in stratum and is the smallest count you can accept for a per-stratum score (the next section suggests how to choose it). Report the score per stratum as well as overall, and when you need the overall figure to mirror production, re-weight each stratum by instead of averaging the items.

How many items are enough

The answer depends on the difference you need to detect, but a rule of thumb comes from the binomial confidence interval. If is the observed pass rate on independent items, the normal-approximation half-width at 95 percent confidence is

Solving for the size needed to reach a half-width in the worst case gives

so about 100 items per reported group give a half-width near plus or minus 10 percentage points and about 400 give plus or minus 5. The consequence for gates: a set of 100 items cannot reliably reveal a 3-point regression; it only catches large ones. Hence the layering used in the rest of this sub-section: a small fast set for every pull request that guards against collapses, and a larger set for model selection and nightly runs that can resolve small differences. When both systems are graded on the same items, the comparison of two systems is paired and needs fewer items than two independent samples; the treatment of paired comparisons and the binomial interval for preference data is on Comparative Evaluation and Arenas.

Two other results are useful at the small end. The normal approximation is poor near 0 or 1 and for small ; use a Wilson interval from a statistics library instead. And when a rare, serious failure (a leaked secret, a harmful instruction) never occurs in independent trials, the "rule of three" gives an approximate 95 percent upper bound on its rate:

so 300 clean trials still allow a failure rate of roughly one percent. For safety-style checks that is the honest way to phrase a result: not "zero failures" but "an upper bound of about ". The dependence assumption matters too: questions generated from the same page are not independent, so the effective sample size is smaller than the row count.

If an LLM judge scores the items, the judge’s own error adds to the noise; calibrate it against human labels first, as described in LLM-as-a-Judge.

Labelling workflows

Every item needs a reference that a grader (human or judge) can trust. For the DocsAssistant a record usually carries a reference answer written as facts that must appear rather than one exact sentence, the supporting page URLs, and flags for answerability. Writing the reference as key points makes both human and model grading less brittle than demanding word-for-word equality.

A workable workflow:

  1. Draft. A model proposes questions and reference key points from the page, or a person writes them from a real trace.

  2. Independent review. Two reviewers label each item without seeing each other’s answer: is the question clear, is it answerable from the docs, are the key points correct.

  3. Measure agreement. Compute chance-corrected agreement on a shared slice, using Cohen’s kappa for two raters:

    where is the observed proportion of items on which they agree and is the proportion expected by chance from each rater’s label frequencies. A low kappa usually means the guidelines are vague, not that the raters are careless; rewrite the guidelines, relabel, and recompute. The same statistic is used to calibrate a judge against humans.

  4. Adjudicate. A third person resolves disagreements; items that remain contested are dropped from the gate set or tagged ambiguous, since they only add noise.

  5. Freeze and sign off. Record who reviewed which version and when.

Reviewers need a short, written labelling guide with examples of accepted and rejected items, the answerability definitions above, and a rule for version-specific answers (always state the version the reference is valid for). Sampling a slice of already-accepted items each quarter and relabelling it detects reviewers (and guidelines) drifting. Hosted tools such as LangSmith annotation queues or Langfuse annotation can host the review screens; the principle stays the same whichever you use.

Storing the set

The same records can live in three places. Keep a plain file as the source of truth and mirror it into whichever platform runs your experiments.

A JSONL and CSV convention

One JSON object per line is easy to diff, stream, review in a pull request and load in any language. Use a fixed schema:

{"id": "docs-0042", "question": "How do I set the Spring AI chat memory window?", "reference_key_points": ["Configure MessageWindowChatMemory with a maximum number of messages", "Memory is passed to the chat client through an advisor"], "reference_urls": ["ai/spring-ai/chat-memory.adoc"], "intent": "how-to", "answerability": "answerable", "turn": 1, "source": "docs", "tags": ["spring-ai", "memory"], "docs_version": "2.0", "created": "2026-10-01", "reviewers": ["ana", "ben"]}

Source: this schema is a convention of this guide, not an official format; the line-per-record layout follows JSON Lines and the syntax is defined by RFC 8259.

Rules that keep it dependable: a stable id that is never reused; every field present in every row (use null, not a missing key); lists for multi-valued fields; UTF-8 and one record per line; and the provenance fields source, docs_version and created. For spreadsheet-based review, export the same fields to CSV with list fields joined by a delimiter you document, and import the reviewed CSV back with a script that validates every row, so the spreadsheet is a view and not a second source of truth. Validate on every pull request:

import json
from collections import Counter

REQUIRED = {"id", "question", "reference_key_points", "intent", "answerability", "source", "docs_version"}
ANSWERABILITY = {"answerable", "partial", "unanswerable", "out_of_scope"}


def load_validated(path: str) -> list[dict]:
    rows, seen = [], set()
    with open(path, encoding="utf-8") as fh:
        for n, line in enumerate(fh, start=1):
            row = json.loads(line)
            missing = REQUIRED - row.keys()
            if missing:
                raise ValueError(f"{path}:{n} missing {sorted(missing)}")
            if row["answerability"] not in ANSWERABILITY:
                raise ValueError(f"{path}:{n} bad answerability {row['answerability']!r}")
            if row["id"] in seen:
                raise ValueError(f"{path}:{n} duplicate id {row['id']}")
            seen.add(row["id"])
            rows.append(row)
    return rows


if __name__ == "__main__":
    rows = load_validated("evals/docsassistant/golden.jsonl")
    print(Counter(r["intent"] for r in rows))

This is plain Python (the json module) and uses no external library; the file layout is a convention of this guide, not of any tool. The standard library reference for the format handling is json.

In Java, the same file can drive a JUnit parameterized test, which is how a gate runs inside a Spring AI build (see Spring AI Evaluation). A @MethodSource factory reads the lines and returns one argument per question:

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import java.util.stream.Stream;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;

class GoldenSetTest {

    static Stream<JsonNode> goldenQuestions() throws IOException {
        ObjectMapper mapper = new ObjectMapper();
        List<String> lines = Files.readAllLines(Path.of("evals/docsassistant/golden.jsonl"));
        return lines.stream().map(line -> {
            try {
                return mapper.readTree(line);
            } catch (IOException e) {
                throw new IllegalStateException(e);
            }
        });
    }

    @ParameterizedTest(name = "{0}")
    @MethodSource("goldenQuestions")
    void answersGoldenQuestion(JsonNode item) {
        String question = item.get("question").asText();
        // call the DocsAssistant and apply the evaluators here (see the Spring AI evaluation page)
    }
}

The annotations derive from the JUnit 5 user guide section Parameterized Tests. The method body is deliberately empty: the evaluators and thresholds are covered on Evaluation in CI. Jackson is used here as the common JSON library in Spring Boot projects; the Jackson package names changed in its 3.x line, so check the version your Boot release manages.

LangSmith datasets

LangSmith stores a dataset as examples with inputs, outputs and metadata. The documented Python workflow creates the dataset and adds examples in bulk:

import json
from langsmith import Client

client = Client()

dataset = client.create_dataset(
    dataset_name="docsassistant-golden",
    description="DocsAssistant golden set, mirrored from evals/docsassistant/golden.jsonl",
)

examples = []
with open("evals/docsassistant/golden.jsonl", encoding="utf-8") as fh:
    for line in fh:
        row = json.loads(line)
        examples.append(
            {
                "inputs": {"question": row["question"]},
                "outputs": {"key_points": row["reference_key_points"]},
                "metadata": {
                    "id": row["id"],
                    "intent": row["intent"],
                    "answerability": row["answerability"],
                    "docs_version": row["docs_version"],
                },
            }
        )

client.create_examples(dataset_id=dataset.id, examples=examples)

The calls derive from the LangSmith page Manage datasets programmatically. Metadata can later be used to select a subset with client.list_examples(dataset_name=…​, metadata={…​}), and update_examples accepts a splits argument to assign examples to named splits; both are on that page. To run an experiment the documented call passes the dataset name or id:

results = client.evaluate(
    target_function,          # your function: inputs dict -> outputs dict
    data="docsassistant-golden",
    evaluators=[key_points_evaluator],
    experiment_prefix="baseline",
    max_concurrency=4,
)

This derives from Evaluate an LLM application. That page documents data as a dataset name or UUID. LangSmith also keeps dataset versions (a new version is recorded on each change to the examples) and supports evaluating on a split or a version; the pages consulted for this write-up did not show the exact parameters for those two, so look them up in the LangSmith documentation for your SDK version before relying on them. The evaluators and the pytest integration are covered in LangChain Evaluation.

MLflow evaluation datasets

MLflow 3 has first-class evaluation datasets in its GenAI module. A dataset is created within an experiment and populated with records that have inputs and expectations:

import json
import mlflow
from mlflow.genai.datasets import create_dataset

dataset = create_dataset(
    name="docsassistant-golden",
    experiment_id=["0"],
    tags={"team": "docs", "stage": "golden"},
)

records = []
with open("evals/docsassistant/golden.jsonl", encoding="utf-8") as fh:
    for line in fh:
        row = json.loads(line)
        records.append(
            {
                "inputs": {"question": row["question"]},
                "expectations": {"expected_facts": row["reference_key_points"]},
            }
        )

dataset.merge_records(records)

The calls derive from the MLflow page Evaluation Datasets. merge_records accepts a list of dictionaries or a DataFrame and updates existing records instead of duplicating them. The experiment_id value shown is the documented default experiment placeholder; use your own. The dataset is then passed to mlflow.genai.evaluate(data=…​, predict_fn=…​, scorers=[…​]), as in the MLflow evaluation quickstart, whose built-in scorers read the expectations field. Which expectation keys a given built-in scorer reads is defined on its own documentation page, so align the key names (expected_response, expected_facts) with the scorer you choose.

Which to use

Store Strengths Limits

JSONL in Git

Reviewable in a pull request, works offline, tool-neutral, versioned with the code that it tests.

No UI for annotation; large sets and attachments need extra tooling.

LangSmith dataset

Annotation queues, experiment comparison UI, filtering by metadata and splits.

Hosted service; the dataset is a mirror, so keep the file as the source of truth.

MLflow evaluation dataset

Sits beside traces, runs and the prompt registry in a self-hostable platform.

Record shape is tied to MLflow’s inputs and expectations; export if you leave.

Mirroring from the file means any platform can be replaced without losing the set, and the CI job can build the dataset in whichever store the experiment runner uses.

Dataset versioning

Evaluation scores are only comparable across runs if the dataset did not change, and any change should be traceable. Treat the golden set as code:

  • Files in Git. The JSONL lives next to the application; changes arrive as pull requests with reviewers, and the diff shows exactly which questions changed.

  • Semantic version in a manifest. Increment the minor version when items are added and the major version when items are removed or their references change, since the latter breaks comparison with earlier scores.

  • A content hash. Store a digest of the file in the manifest and in every experiment’s metadata, so a run states precisely which bytes it was graded against.

  • Append, do not edit silently. Mark a retired item as deprecated with the reason instead of deleting its row; the history shows why a score moved.

  • Hold-out discipline. Keep a split that nobody tunes prompts against. Prompt authors iterate on a dev split; the held-out split is only run at release time. Without this the set drifts from an evaluation into a training signal.

A manifest and a hash take a few lines:

import hashlib
import json
from pathlib import Path


def dataset_manifest(path: str, version: str) -> dict:
    data = Path(path).read_bytes()
    rows = [json.loads(line) for line in data.decode("utf-8").splitlines()]
    return {
        "name": "docsassistant-golden",
        "version": version,
        "sha256": hashlib.sha256(data).hexdigest(),
        "items": len(rows),
        "docs_version": sorted({r["docs_version"] for r in rows}),
    }


manifest = dataset_manifest("evals/docsassistant/golden.jsonl", "1.3.0")
Path("evals/docsassistant/manifest.json").write_text(json.dumps(manifest, indent=2))

This uses only the Python standard library; see hashlib. Pass the manifest’s version and sha256 as metadata when you start an experiment (an experiment prefix or metadata field in LangSmith, a tag or parameter in an MLflow run), and the dashboards on Metrics, Dashboards and Alerting can annotate score changes that coincide with a dataset change.

Documents change too. When the documentation is revised, the reference for a version-specific question can become wrong without anyone touching the eval set. Re-run reference checks when a referenced page changes (the reference_urls field exists for this purpose) and tag items with the docs_version they were validated against.

Keeping the set alive

A golden set decays. A short maintenance routine prevents it:

  1. Monthly. Sample new production questions, label the failures and the novel intents, add them to the next minor version.

  2. On each documentation release. Revalidate items referencing changed pages; add questions for new pages.

  3. On each incident. Add the triggering question with a tag naming the failure.

  4. Quarterly. Compare the intent distribution of the set with the intent distribution of recent traffic. A large gap means the set no longer represents users; this is the offline counterpart of the checks on Drift and Regression Monitoring.

  5. Yearly. Relabel a random slice with fresh reviewers and recompute kappa against the previous labels.

For RAG-specific retrieval metrics computed over the reference_urls labels, see Evaluating RAG Systems and Evaluation at Scale; for trajectory and tool-call references used with agents see Agent Evaluation.

Checklist

  1. Taxonomy written down; every stratum has at least the minimum item count you chose.

  2. Real-traffic items scrubbed of personal data, with trace provenance recorded.

  3. Synthetic items generated by a different model family from the one under test, reviewed by people, deduplicated.

  4. Unanswerable, out-of-scope, multi-turn and adversarial items present.

  5. Two independent labellers on a shared slice, kappa recorded, ambiguous items removed.

  6. A fast gate set and a larger selection set, with the confidence interval of each stated.

  7. File in Git as source of truth, manifest with version and hash, held-out split untouched by prompt tuning.

  8. Maintenance routine scheduled; failures from production flow back into the set.

References