Evaluation in CI

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against langsmith 0.14.2 with openevals 0.2.0, Spring AI 2.0.1 on Spring Boot 4.1.x with Java 21 and JUnit Jupiter, Python 3.12, and the GitHub Actions workflow syntax and first-party actions current on that date (actions/checkout v7, actions/setup-python v7, actions/setup-java v6, actions/cache v6, actions/upload-artifact v7, actions/github-script v9); the full list is the version baseline on the landing page. Every snippet here was written from the official documentation linked after it and was not run: no workflow was executed on GitHub, no judge model was called and the Java test was not compiled by this repository. The names of the LangSmith pytest plugin API were read from its documentation; the openevals call shapes were read from its repository README. Where a detail could not be confirmed it is said so in the text.

An evaluation that someone has to remember to run is not a gate, it is a hobby. The previous pages built the pieces: a metric (Evaluation Methodology), a judge (LLM-as-a-Judge), a versioned golden set (Building Eval Sets) and the RAG and agent metrics (RAG and Agent Evaluation). This page wires them into the pull request so that a prompt edit, a model bump, a new chunk size or a rebuilt index cannot reach production without a number attached to it. The general idea of running quality checks in the pipeline is the one described by Aryan in LLMOps and by Huyen in AI Engineering; the workflows, statistics and tool calls below are written from current documentation, because the book pipelines pre-date the tools used here.

What changes, and what must trigger an evaluation

In a traditional service, code is the only thing that changes behaviour. In the DocsAssistant, at least five artefacts do, and only the first lives in the application source tree in the usual sense:

Artefact Typical location Why it needs a gate

Artefact	Typical location	Why it needs a gate
Prompt template	`prompts/*.txt` or a prompt registry id pinned in config	One reworded sentence can change refusal behaviour and citation format.
Model or its parameters	`config/model.yaml` (model id, temperature, max tokens), or the gateway route	A provider alias moving to a newer snapshot changes every answer without a code change.
Chunking and retrieval settings	`ingest/chunking.yaml`, top-k, reranker	Chunk size alters what the model sees; see Ingestion Pipelines and Freshness.
Index content	The vector store, rebuilt from the documentation sources	New or removed pages change recall without a single line of code changing.
Evaluation assets	`evals/docsassistant/golden.jsonl`, judge prompts, thresholds	A gate that can be edited in the same pull request it guards needs review of its own.

Prompt template

prompts/*.txt or a prompt registry id pinned in config

One reworded sentence can change refusal behaviour and citation format.

Model or its parameters

config/model.yaml (model id, temperature, max tokens), or the gateway route

A provider alias moving to a newer snapshot changes every answer without a code change.

Chunking and retrieval settings

ingest/chunking.yaml, top-k, reranker

Chunk size alters what the model sees; see Ingestion Pipelines and Freshness.

Index content

The vector store, rebuilt from the documentation sources

New or removed pages change recall without a single line of code changing.

Evaluation assets

evals/docsassistant/golden.jsonl, judge prompts, thresholds

A gate that can be edited in the same pull request it guards needs review of its own.

The first four are inputs to the system under test; the last is the test itself. Both kinds should trigger a run, but they do not trigger it in the same way. Source, prompt and config changes are visible in the pull request diff, so a path filter on pull_request is enough. An index rebuild happens in a pipeline, not in a pull request, so it calls the same evaluation workflow through workflow_dispatch or workflow_call with the index snapshot as an input.

The shape of the pipeline

flowchart TD PR[Pull request opened or updated] --> F{Changed paths match
prompts, model, chunking, evals?} F -- no --> S[Skip eval job, required check reports success] F -- yes --> B[Build and unit tests] B --> R[Run DocsAssistant on the golden set] R --> C[Score: code checks, then cached judge calls] C --> D[Compare with baseline from main] D --> G{Gate: lower bound above floor
and no significant drop?} G -- yes --> P[Post report comment, check passes] G -- no --> X[Post failing cases, check fails] X --> H{Reviewer accepts the trade-off?} H -- yes --> U[Update baseline in a reviewed commit] H -- no --> PR U --> P IDX[Index rebuild pipeline] -. workflow_dispatch .-> R

Three design decisions are visible in the diagram. First, a skipped evaluation must still report a successful required check, otherwise a pull request that touches only the README can never merge; the usual trick is to have the workflow always run and let a first step decide whether to continue, rather than relying on a paths filter together with a required status check. Second, the baseline comes from the main branch, not from a number typed into a file, so the gate measures regression rather than absolute quality. Third, accepting a regression is an explicit, reviewed commit that rewrites the baseline, which keeps the decision in the history.

Deciding what to run and what to gate on

A full evaluation is slow and costs model calls, so the pipeline is layered, cheapest and most deterministic first. Each layer only runs if the previous one passed:

Deterministic checks (seconds, no model call): the answer parses, every citation id exists in the retrieved set, the refusal phrase appears for the out-of-scope questions, the length is inside bounds. These are ordinary test assertions.
Retrieval metrics (seconds, embeddings only): hit rate and rank of the expected source page for each golden question, as defined on the Evaluating RAG Systems page.
Judge metrics (minutes, model calls): faithfulness and answer correctness scored by the judge, with the cache described below so unchanged answers are not re-judged.
Nightly or pre-release only: the large set, pairwise comparison against the previous release (Comparative Evaluation and Arenas) and agent trajectory evaluation.

The golden set is also split so that a pull request runs a smoke subset (for example one question per intent cluster, a few dozen items) while the nightly job runs everything. The subset is chosen by a stable tag in the JSONL, never at random, so two runs of the same commit evaluate the same questions. The examples on this page use the file, schema and field names defined on Building Eval Sets: the full set lives in evals/docsassistant/golden.jsonl, and the smoke subset evals/docsassistant/golden.smoke.jsonl is generated from it by keeping the rows tagged smoke, so it has exactly the same schema (id, question, reference_key_points, reference_urls, tags and the rest).

Thresholds that survive noise

A pass-rate gate such as "at least 90 percent correct" is easy to write and easy to get wrong. With questions and passes the measured rate is an estimate with sampling error, and on the small sets a pull request can afford that error is large. Gate on a confidence bound, not on the point estimate. The lower end of the Wilson score interval, the same bound used on the Spring AI evaluation page, is:

where for a 95 percent interval. Two gates then express two different questions:

Absolute floor. Fail if : even optimistically-but-honestly, this change is not good enough to ship. This protects against a baseline that has slowly decayed.
Regression against the baseline. Fail if the candidate is significantly worse than main on the same questions. Because the same items are scored on both sides, use a paired comparison: let be the number of items the baseline passes and the candidate fails, and the number the candidate passes and the baseline fails. Under the null hypothesis that nothing changed, out of discordant items is a coin flip, so an exact one-sided sign test (McNemar’s exact form) gives the p-value:

Reject the change when (for example 0.05) and the candidate’s net drop exceeds a minimum effect size you actually care about. The paired test is far more sensitive than comparing two Wilson intervals, because it ignores the many questions that both versions get right or both get wrong.

Neither gate is magic. With a smoke subset of 30 questions the floor can only catch large failures; the nightly run on the full set is what detects slow erosion. State this in the pull-request comment so reviewers do not read a green check as proof of quality.

Flakiness and repeat runs

An LLM pipeline is non-deterministic at several points: sampling in the generator (even at temperature zero, batching and hardware can change results), the judge, and sometimes retrieval ties. A gate that fails one run in ten for no reason trains the team to click "re-run", which destroys its value. There are three complementary defences:

Remove the avoidable randomness. Pin temperature to zero or a low value and a seed where the provider supports one, pin the model snapshot rather than an alias, and keep the judge on a different, pinned model. Treat any remaining variation as noise to be measured, not hidden.
Measure it. Run each smoke question times. If a single run of an item fails with probability independently, the chance that a good system trips a rule of the form "fail if any run fails" is:

With and that is already about 14 percent, which is why "any failure" is a poor rule.
Aggregate with a majority or a mean. Score an item as passed when most of its runs pass, or average the scores, then apply the Wilson floor and the paired test to the aggregated results. Items that disagree with themselves are reported separately as unstable so someone can fix the question, the rubric or the prompt instead of ignoring them.

Repeat runs multiply the cost, so a common compromise is to run each item once, then re-run only the failures up to twice before counting them. This keeps the typical green run cheap while still absorbing a one-off judge slip. Never use retry-until-green on the gate itself: a retry that is allowed to loop until it passes is a gate that cannot fail.

The evaluation harness in Python

The DocsAssistant core is an HTTP service (see Serving RAG as an API), so the harness treats it as a black box, calls it for each golden item, scores the result and writes a JSON report. The judge is built with create_llm_as_judge from openevals; its documentation shows inputs, outputs and reference_outputs keyword arguments and a result dictionary with key, score and comment. The judge model is read from an environment variable in the provider:model format that openevals uses, so the workflow, not the code, decides which model judges.

evals/run_eval.py: run the golden set, score it and write a report

import hashlib
import json
import os
import pathlib
import sys
from concurrent.futures import ThreadPoolExecutor

import httpx
from openevals.llm import create_llm_as_judge
from openevals.prompts import CORRECTNESS_PROMPT

ASSISTANT_URL = os.environ.get("ASSISTANT_URL", "http://localhost:8080")
JUDGE_MODEL = os.environ["JUDGE_MODEL"]            # "provider:model", pinned in the workflow
CACHE_DIR = pathlib.Path(os.environ.get("JUDGE_CACHE_DIR", ".judge-cache"))
GOLDEN = pathlib.Path(sys.argv[1])                 # evals/docsassistant/golden.jsonl (or its smoke subset)
REPORT = pathlib.Path(sys.argv[2])                 # report.json
RUNS = int(os.environ.get("EVAL_RUNS", "1"))

judge = create_llm_as_judge(
    prompt=CORRECTNESS_PROMPT,
    model=JUDGE_MODEL,
    feedback_key="correctness",
)


def ask(question: str) -> str:
    response = httpx.post(f"{ASSISTANT_URL}/ask", json={"text": question}, timeout=60)
    response.raise_for_status()
    return response.json()["answer"]


def cached_judge(question: str, answer: str, reference: str) -> dict:
    """Judge calls are keyed by everything that can change the verdict."""
    key = hashlib.sha256(
        json.dumps([JUDGE_MODEL, CORRECTNESS_PROMPT, question, answer, reference]).encode()
    ).hexdigest()
    path = CACHE_DIR / f"{key}.json"
    if path.exists():
        return json.loads(path.read_text())
    verdict = judge(inputs=question, outputs=answer, reference_outputs=reference)
    CACHE_DIR.mkdir(parents=True, exist_ok=True)
    path.write_text(json.dumps(verdict))
    return verdict


def evaluate(item: dict) -> dict:
    # Schema from Building Eval Sets: the reference is a list of key points, the sources a list of page paths.
    reference = "\n".join(f"- {point}" for point in item["reference_key_points"])
    must_cite = item.get("reference_urls") or []
    passes = 0
    for _ in range(RUNS):
        answer = ask(item["question"])
        if all(url in answer for url in must_cite):
            verdict = cached_judge(item["question"], answer, reference)
            passes += 1 if verdict["score"] else 0
    return {"id": item["id"], "tags": item.get("tags", []), "passes": passes, "runs": RUNS}


if __name__ == "__main__":
    items = [json.loads(line) for line in GOLDEN.read_text().splitlines() if line.strip()]
    with ThreadPoolExecutor(max_workers=8) as pool:
        results = list(pool.map(evaluate, items))
    REPORT.write_text(json.dumps({"judge": JUDGE_MODEL, "results": results}, indent=2))

Source: openevals README, with the HTTP call shape being the DocsAssistant’s own.

Two details deserve a comment. The cache key includes the judge model id, the full judge prompt, the question, the answer and the reference (the key points joined into one string); leave any of them out and an edit to the rubric silently keeps returning stale verdicts. And the deterministic citation check runs before the judge call, so a malformed answer costs nothing. Whether a verdict’s score is a boolean or a number depends on how the judge is configured (continuous, choices); the harness above assumes the default boolean, and a configuration that returns a float would need a threshold instead of the truthiness test.

The gate script

The gate is a separate, deterministic step with no model calls, so it can be tested on its own and re-run for free. It reads the candidate and baseline reports, applies the floor and the paired test and exits non-zero on failure.

evals/gate.py: floor on the Wilson bound plus an exact paired sign test

import json
import math
import sys

FLOOR = 0.80          # tau_floor: reviewed like code, lives in the repository
ALPHA = 0.05
MIN_DROP = 0.03       # smallest net drop worth blocking a change for
Z = 1.96


def wilson_lower(k: int, n: int) -> float:
    p = k / n
    centre = p + Z * Z / (2 * n)
    margin = Z * math.sqrt(p * (1 - p) / n + Z * Z / (4 * n * n))
    return (centre - margin) / (1 + Z * Z / n)


def passed(entry: dict) -> bool:
    return entry["passes"] * 2 > entry["runs"]    # majority of the repeat runs


def load(path: str) -> dict:
    return {r["id"]: passed(r) for r in json.load(open(path))["results"]}


def sign_test_p(b: int, c: int) -> float:
    """One-sided exact p-value that the candidate lost b of b+c discordant items by chance."""
    n = b + c
    return sum(math.comb(n, i) for i in range(b, n + 1)) / 2**n if n else 1.0


candidate = load(sys.argv[1])
baseline = load(sys.argv[2])
shared = candidate.keys() & baseline.keys()

k, n = sum(candidate.values()), len(candidate)
if n == 0:  # wrong dataset path, or every item errored: fail loudly instead of dividing by zero
    print("GATE FAILED: no results")
    sys.exit(1)
lower = wilson_lower(k, n)
b = sum(1 for i in shared if baseline[i] and not candidate[i])
c = sum(1 for i in shared if candidate[i] and not baseline[i])
drop = (b - c) / max(len(shared), 1)
p = sign_test_p(b, c)

print(f"pass rate {k}/{n}, Wilson lower bound {lower:.3f} (floor {FLOOR})")
print(f"regressions {b}, improvements {c}, net drop {drop:.3f}, sign test p={p:.3f}")

failures = []
if lower < FLOOR:
    failures.append("lower bound below the floor")
if p < ALPHA and drop > MIN_DROP:
    failures.append("significant regression against the baseline")
for item_id in sorted(i for i in shared if baseline[i] and not candidate[i]):
    print(f"  regressed: {item_id}")
if failures:
    print("GATE FAILED: " + "; ".join(failures))
    sys.exit(1)
print("GATE PASSED")

Source: Wilson score interval and McNemar’s test (exact form), the formulas implemented above.

An empty results list (a wrong dataset path, or every item erroring) fails the gate explicitly with GATE FAILED: no results rather than crashing in the Wilson bound or, worse, passing. Because the script is plain Python it needs its own unit tests (a candidate identical to the baseline must pass; a candidate that loses ten items must fail; an empty report must fail), which run in the ordinary test job in milliseconds.

The GitHub Actions workflow

The workflow below runs the smoke subset on pull requests and accepts an index snapshot for rebuild-triggered runs. It uses the pull_request trigger, not pull_request_target, so it runs the code of the pull request without handing repository secrets to untrusted forks; the security hardening guide warns specifically against combining pull_request_target with checkout of untrusted code. Consequently the judge step has no API key on a fork pull request, and the job reports that the judged part was skipped rather than failing opaquely.

.github/workflows/eval.yml

name: eval

on:
  pull_request:
  workflow_dispatch:
    inputs:
      index_snapshot:
        description: Index snapshot id to evaluate (set by the index rebuild pipeline)
        required: false

concurrency:
  group: eval-${{ github.ref }}
  cancel-in-progress: true

permissions:
  contents: read
  pull-requests: write

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    env:
      JUDGE_MODEL: ${{ vars.JUDGE_MODEL }}            # pinned in repository variables
      EVAL_RUNS: "1"
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0

      - name: Decide whether the change can affect answers
        id: scope
        run: |
          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
            echo "run=true" >> "$GITHUB_OUTPUT"; exit 0
          fi
          base="${{ github.event.pull_request.base.sha }}"
          if git diff --name-only "$base"...HEAD | grep -E '^(prompts/|config/model|ingest/|evals/|src/main/)' >/dev/null; then
            echo "run=true" >> "$GITHUB_OUTPUT"
          else
            echo "run=false" >> "$GITHUB_OUTPUT"
          fi

      - uses: actions/setup-python@v7
        if: steps.scope.outputs.run == 'true'
        with:
          python-version: "3.12"
          cache: pip

      - name: Install evaluation dependencies
        if: steps.scope.outputs.run == 'true'
        run: pip install -r evals/requirements.txt

      - name: Restore judge cache
        if: steps.scope.outputs.run == 'true'
        uses: actions/cache@v6
        with:
          path: .judge-cache
          key: judge-${{ vars.JUDGE_MODEL }}-${{ hashFiles('evals/judge_prompt.txt', 'evals/docsassistant/golden.jsonl') }}-${{ github.run_id }}
          restore-keys: |
            judge-${{ vars.JUDGE_MODEL }}-${{ hashFiles('evals/judge_prompt.txt', 'evals/docsassistant/golden.jsonl') }}-
            judge-${{ vars.JUDGE_MODEL }}-

      - name: Start the DocsAssistant
        if: steps.scope.outputs.run == 'true'
        run: docker compose -f docker-compose.eval.yml up -d --wait

      - name: Run the smoke subset
        if: steps.scope.outputs.run == 'true'
        env:
          JUDGE_API_KEY: ${{ secrets.JUDGE_API_KEY }}
        run: python evals/run_eval.py evals/docsassistant/golden.smoke.jsonl report.json

      - name: Fetch the baseline from main
        if: steps.scope.outputs.run == 'true'
        run: git show origin/main:evals/baseline.json > baseline.json

      - name: Apply the gate
        if: steps.scope.outputs.run == 'true'
        run: python evals/gate.py report.json baseline.json | tee gate.txt

      - name: Keep the report
        if: always() && steps.scope.outputs.run == 'true'
        uses: actions/upload-artifact@v7
        with:
          name: eval-report
          path: |
            report.json
            gate.txt

Source: Workflow syntax for GitHub Actions, Dependency caching reference and Security hardening for GitHub Actions.

Points worth knowing about this file:

The scope step implements the "always report, decide inside" pattern from the flowchart. Every later step is guarded with the step output, so the job always finishes green when nothing relevant changed and a required check on eval stays satisfiable. A paths: filter on the trigger would instead leave the check absent.
The environment variable name for the key (JUDGE_API_KEY) is a placeholder: the real name is whatever your judge provider’s SDK reads. Keep the key in an environment-scoped secret.
concurrency with cancel-in-progress cancels the previous evaluation of the same branch when a new commit arrives, which saves judge calls on a busy pull request.
The cache key ends with the run id, so every run saves a new cache while restore-keys falls back to the most recent one with the same judge and rubric. The dependency caching reference documents that a cache is only restored from the current branch, its base or the default branch, so a first run on a new branch starts from the cache of main.
fetch-depth: 0 is needed because the baseline is read from origin/main with git show.
A comment on the pull request with the gate output is usually worth adding with actions/github-script, which needs the pull-requests: write permission declared above; the script call is omitted here because it is routine and varies with how you format the report.

The rebuild pipeline for the index triggers the same job by calling the workflow dispatch endpoint (for example gh workflow run eval.yml -f index_snapshot=<id>), and the compose file reads the snapshot id so the assistant under test serves that index. Which compose services start, and how health checks gate --wait, is general container work covered on the Docker CI/CD with GitHub Actions page. The same structure (a gating job, cached inputs, an uploaded report) is applied to mobile builds on the Android CI/CD with GitHub Actions page, and load-testing the serving tier as a separate pipeline stage is the subject of Performance Testing with JMeter. Quality gates and performance gates should be separate jobs: they fail for different reasons, cost different amounts and want different runners.

Caching judge calls

Judge calls dominate the bill of an evaluation run, and most of them are repeats: the same question, the same answer from an unchanged code path, the same rubric. A content-addressed cache removes them. The cache key is a hash of everything that can change the verdict:

where is a cryptographic hash such as SHA-256. The harness above implements exactly this. Properties to rely on:

Correctness by construction. Changing the rubric, the judge model or the answer changes the key, so a stale verdict can never be served; there is no invalidation logic to get wrong.
Only judges are cached, not the system under test. Caching the DocsAssistant’s own answers would hide exactly the regressions the gate exists to find. If a change touches nothing that affects an item, its answer is identical and the judge call is saved naturally.
Cache hits require identical text. If sampling makes the answer differ on every run, the hit rate drops; this is another argument for pinned temperature and for scoring with deterministic checks first.
Storage is a convenience, not a source of truth. actions/cache entries can be evicted (the reference documents removal of entries not accessed for seven days and a per-repository size limit), so the pipeline must work with an empty cache, only slower and more expensive.
Cached verdicts are data. They contain user-visible questions and answers; treat the cache directory like the golden set from a privacy point of view and keep real user text out of it where the data operations page advises (Data Operations).

The LangSmith plugin (below) offers an HTTP-level cache with a different trade-off: it records responses from model hosts by request, so the key is the exact request body, which is just as strict but does not understand your rubric versioning.

What an evaluation costs

Estimate the cost of a run before it surprises the budget. Let be the number of golden items evaluated, the repeat runs per item, the judge cache hit rate, and and the mean cost of one system call and one judge call, each computed from the token counts of a sample run and the current price page of your provider (this page quotes no prices):

Monthly spend is then times the number of runs. Three levers follow from the formula:

Reduce for pull requests. Smoke subset on every push, full set nightly, repeat runs only for failures.
Raise . Deterministic checks first, a cache restored from main, and stable answers.
Reduce . A smaller pinned judge for the pull-request smoke run and the strong judge for the nightly run, as long as the cheap judge has been calibrated against human labels (see the Cohen’s kappa procedure on LLM-as-a-Judge).

Also count the cost of a wasted run: a pull request pushed ten times costs ten runs unless concurrency cancels the superseded ones. The wider picture, per-tenant attribution and a monthly report, is on Cost Management. A local model as the judge removes the per-call fee but puts the run on slower hardware; see Local LLMs for that trade-off.

JUnit gates in a Spring AI project

If the evaluation is part of the Java build, the same ideas apply with JUnit as the runner. The gate is a test that iterates the golden file, asks the service, judges with Spring AI’s RelevancyEvaluator (which takes a ChatClient.Builder and implements Evaluator, as shown on the Spring AI evaluation page) and asserts on the Wilson bound. The test is tagged so that the ordinary unit-test run skips it and a dedicated CI job selects it.

GoldenSetGateTest.java: an aggregate gate, not one test per question

package com.example.docsassistant.eval;

import static org.assertj.core.api.Assertions.assertThat;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;

import com.example.docsassistant.DocsAssistant;
import com.example.docsassistant.WilsonInterval;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.evaluation.RelevancyEvaluator;
import org.springframework.ai.document.Document;
import org.springframework.ai.evaluation.EvaluationRequest;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

@Tag("eval")
@SpringBootTest
class GoldenSetGateTest {

    // Fields of the Building Eval Sets schema that this gate reads; the others are ignored.
    @JsonIgnoreProperties(ignoreUnknown = true)
    record GoldenItem(
            String id,
            String question,
            @JsonProperty("reference_key_points") List<String> referenceKeyPoints,
            @JsonProperty("reference_urls") List<String> referenceUrls,
            List<String> tags) { }

    private static final double FLOOR = 0.80;

    @Autowired
    DocsAssistant assistant;

    @Autowired
    RelevancyEvaluator relevancy;

    @Test
    void goldenSetStaysAboveTheFloor() throws IOException {
        ObjectMapper mapper = new ObjectMapper();
        List<GoldenItem> items = Files.readAllLines(Path.of("evals/docsassistant/golden.smoke.jsonl")).stream()
                .filter(line -> !line.isBlank())
                .map(line -> {
                    try {
                        return mapper.readValue(line, GoldenItem.class);
                    } catch (IOException e) {
                        throw new IllegalStateException(e);
                    }
                })
                .toList();

        int passes = 0;
        for (GoldenItem item : items) {
            DocsAssistant.Answer answer = assistant.ask(item.question());
            List<Document> context = answer.retrievedDocuments();
            var result = relevancy.evaluate(new EvaluationRequest(item.question(), context, answer.text()));
            if (result.isPass()) {
                passes++;
            }
        }

        double lower = WilsonInterval.lowerBound(passes, items.size());
        assertThat(lower)
                .as("Wilson lower bound for %d/%d passes", passes, items.size())
                .isGreaterThanOrEqualTo(FLOOR);
    }
}

Source: Spring AI: Model Evaluation and JUnit: Tagging and filtering; the record binding uses Jackson Annotations (@JsonProperty, @JsonIgnoreProperties). RelevancyEvaluator judges the answer against the retrieved context, so this gate does not read reference_key_points; a correctness judge would. DocsAssistant, Answer and WilsonInterval are the application’s own classes from the Spring AI sub-section, not a library API.

Design notes for this test:

One aggregate assertion. A parameterized test with one invocation per question reports a red line per failing question, which is helpful in the report but wrong as a gate: a 90 percent system is expected to fail some individual questions. Gate on the aggregate and use the per-item output only for diagnosis.
Repeat for flakiness at the right level. JUnit’s @RepeatedTest takes the number of repetitions and a failureThreshold that skips the remaining repetitions after that many failures. It is suited to a single behaviour you expect to be stable, such as "the out-of-scope question is refused"; for the aggregate gate, repeat inside the test and average, as in the Python harness, rather than repeating a statistical assertion.
Select it explicitly. With Maven Surefire, run mvn test -Dgroups=eval in the evaluation job and exclude the tag (excludedGroups) in the default build so that unit tests never call a model. Both parameters were checked on 2026-10-01 against the Surefire surefire:test parameter reference: groups and excludedGroups are String parameters with user properties of the same name, and they apply to JUnit 5 tags through the JUnit Platform provider since Surefire 2.22.0.
Run it in the same workflow. In eval.yml, replace the Python steps with actions/setup-java@v6 and the Maven command, or run both gates side by side; the result is the same exit code and uploaded report either way.

The Micrometer metrics the test could also emit, such as a gauge for the lower bound, belong to the monitoring story rather than the gate; they appear on Metrics, Dashboards and Alerting.

The LangSmith pytest plugin

If the team already uses LangSmith, its pytest integration turns ordinary tests into tracked evaluation runs without a separate harness. According to the documentation it needs langsmith 0.3.4 or later (this page did not check the plugin against 0.14.2); pip install -U "langsmith[pytest]" adds rich output and caching. A test is marked with @pytest.mark.langsmith; helper functions in langsmith.testing log the inputs, the outputs, the reference outputs and feedback scores; and expect offers fuzzy assertions.

tests/test_docsassistant_eval.py: golden questions as a LangSmith-tracked pytest suite

import json
import pathlib

import pytest
from langsmith import testing as t

from docsassistant_client import ask      # the application's own client

GOLDEN = [
    json.loads(line)
    for line in pathlib.Path("evals/docsassistant/golden.smoke.jsonl").read_text().splitlines()
    if line.strip()
]


@pytest.mark.langsmith(output_keys=["reference"])
@pytest.mark.parametrize(
    "question, reference, reference_urls",
    [(g["question"], "\n".join(g["reference_key_points"]), g["reference_urls"]) for g in GOLDEN],
)
def test_answer_matches_reference(question, reference, reference_urls):
    t.log_inputs({"question": question})
    t.log_reference_outputs({"reference": reference})

    answer = ask(question)
    t.log_outputs({"answer": answer})

    cited = all(url in answer for url in reference_urls)
    t.log_feedback(key="cites_reference_pages", score=1.0 if cited else 0.0)
    t.expect.embedding_distance(prediction=answer, expectation=reference).to_be_less_than(0.4)

Source: LangSmith: Test LLM applications with pytest.

The decorator’s output_keys parameter names the arguments to log as reference outputs. The expect utility is documented with embedding_distance, edit_distance, to_contain and to_be_less_than; the exact call chain above is assembled from that list, so check it against the page before relying on it. Run it with:

LANGSMITH_TEST_SUITE="docsassistant-golden" \
LANGSMITH_EXPERIMENT="pr-${GITHUB_EVENT_NUMBER}" \
LANGSMITH_TEST_CACHE="tests/cassettes" \
pytest tests/test_docsassistant_eval.py --langsmith-output

Source: LangSmith: Test LLM applications with pytest. GITHUB_EVENT_NUMBER is not a GitHub-provided variable; export it in the workflow from github.event.number.

The documented environment variables are LANGSMITH_TEST_SUITE (group the runs into one suite), LANGSMITH_EXPERIMENT (name the experiment) and LANGSMITH_TEST_CACHE (a directory in which outgoing HTTP requests are saved and replayed on later runs), and LANGSMITH_TEST_TRACKING=false runs the suite without syncing to LangSmith. The decorator also accepts cached_hosts to restrict caching to selected hosts. What the plugin gives you is tracking and a comparison UI across experiments; what it does not give you is a statistical gate. The pass or fail of each test is still a plain pytest assertion, so the aggregate floor and the paired comparison from earlier remain your job, either in the same suite (a final test that reads the logged scores) or in the gate.py step. openevals evaluators can be called inside these tests, and the langsmith pytest page lists how they log their feedback. Keeping the cassette directory in the repository means a pull request replays recorded calls; refreshing the cassettes is then a deliberate, reviewed action, which brings back the idea that an accepted regression is an explicit commit. See Evaluation with LangChain and Observability with LangSmith for the rest of the LangSmith tooling.

Updating the baseline deliberately

A baseline that updates itself on every merge is a ratchet that only moves down. Better:

The baseline is evals/baseline.json, committed in the repository, produced by the same harness on the full set.
A pull request that intentionally changes behaviour (a new model, a stricter prompt) runs the gate, which fails, and the author regenerates the baseline in a commit that is reviewed like any other change, with the diff of per-item results in the description.
CODEOWNERS or an equivalent rule requires an owner of the evaluation to approve changes to evals/ and to the threshold constants, since they are the rules of the gate.
The nightly run on main compares against the baseline and opens an issue when a gap appears from changes outside the pull-request path, such as a provider silently updating a model alias; that is the bridge to Drift and Regression Monitoring.

The reverse risk is also real: a team that always updates the baseline to turn the check green has turned the gate into a formality. Track how often baselines are rewritten and what the pass rate did across the rewrites.

Common pitfalls

Gating on an average without a bound. A single mean over a few dozen items moves by chance; use the Wilson floor and the paired test.
Letting the system under test grade itself. Use a different, pinned judge model, and calibrate it.
Fork pull requests. The pull_request trigger withholds secrets from forks, which is the safe default; do not switch to pull_request_target to get them back unless the workflow never checks out untrusted code.
Cache poisoning by identical keys. Include judge model and rubric in the key, as above.
Unbounded cost. Always set timeout-minutes and concurrency, and put a hard limit on the judge provider account.
Tests that depend on live external services. The gate should start a known assistant build and a known index snapshot, not call a shared staging service that others are changing.
Confusing the PR gate with production quality. The gate samples a fixed set; live behaviour is the subject of Online Evaluation and Feedback. Prompt versioning across environments is covered on Prompt and Model Management, and background on prompt practices is on Managing Prompts.

Adversarial and abuse-focused test cases, such as prompt injection and data exfiltration attempts, are part of AI Red Teaming in the AI Security & Responsible AI sub-section; they can be added to the same golden file under their own tags, but the threat models and tooling are out of scope here.