Evaluation

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written against Spring AI 2.0.1, Spring Boot 4.1.x and Java 21 (verified 2026-09-29). A model can produce a fluent, confident and wrong answer, and no compiler or unit test notices. Evaluation is the practice of checking answers with something other than the model that wrote them. Spring AI gives you a small Evaluator interface, two ready-made model-based evaluators and the hooks (advisors and Spring Framework’s @Retryable) to run a check at runtime. This page evaluates DocsAssistant: it checks a retrieved-context answer for relevance and factual support, adds a rule-based citation check and a score-based judge, retries an answer the checker rejects, and turns a golden set of questions into a JUnit suite. All classes were compiled against the 2.0.1 jars and their tests were run. The judge in the unit tests is a WireMock server that returns scripted verdicts, so those tests prove the wiring and not the quality of any real judge. One smoke run used a real (very small) model in a Testcontainers Ollama and is reported where it matters.

What to evaluate, and when

Three questions are different and need different checks:

Table 1. Three kinds of check
Question Checker When it runs

Did the answer use the retrieved context and stay on topic?

RelevancyEvaluator

In tests and, optionally, at runtime

Is every claim in the answer supported by the context?

FactCheckingEvaluator, a stronger judge

In tests and for high-stakes answers at runtime

Does the answer follow our rules (cites a source, right format, no forbidden content)?

A custom Evaluator in plain Java

Everywhere, because it is free and deterministic

Retrieval quality (was the right passage found?) is a separate measurement with its own metrics, such as recall at k, and is covered on Evaluating RAG Systems. This page starts after retrieval, with a question, the passages that were retrieved and the answer that was produced. The evaluation of whole agents, with trajectories and tool use, is on Agent Evaluation.

The Evaluator API

The interface is one method, so a lambda is an evaluator:

Table 2. The evaluation types (org.springframework.ai.evaluation)
Type Role

Evaluator

EvaluationResponse evaluate(EvaluationRequest), plus the helper doGetSupportingData, which joins the text of the request’s documents with line separators

EvaluationRequest

Three parts: getUserText() (the question), getDataList() (a List<Document> of context, possibly empty) and getResponseContent() (the answer being judged)

EvaluationResponse

isPass(), a getScore(), getFeedback() and a getMetadata() map. The constructor without a score sets the score to 0

Source: Spring AI: Model Evaluation. The two ready-made evaluators live in org.springframework.ai.chat.evaluation (module spring-ai-client-chat) and both take a ChatClient.Builder, so the judge is just another model behind a ChatClient.

RelevancyEvaluator

RelevancyEvaluator asks the judge whether the answer is in line with the context for the query and expects one word, YES or NO. Its default template is:

Your task is to evaluate if the response for the query
is in line with the context information provided.

You have two options to answer. Either YES or NO.

Answer YES, if the response for the query
is in line with context information otherwise NO.

Query:
{query}

Response:
{response}

Context:
{context}

Answer:

Source: Spring AI: Model Evaluation.

The placeholders query, response and context are required if you replace the template with promptTemplate(…​) on the builder. The result is pass = true and score = 1 (otherwise 0) only when the judge’s whole reply, stripped, equals "yes" ignoring case. That strictness matters, as the third test below shows.

Two beans build the evaluators once. The judge uses its own model settings (temperature zero, a fixed seed, a few output tokens), because a judge that varies from run to run makes the suite flaky. The builder is created inside the configuration rather than exposed as a ChatClient.Builder bean, so it cannot be confused with the auto-configured builder that the application injects.

JudgeConfig.java: a deterministic judge for both evaluators
package com.example.docsassistant;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.evaluation.FactCheckingEvaluator;
import org.springframework.ai.chat.evaluation.RelevancyEvaluator;
import org.springframework.ai.ollama.OllamaChatModel;
import org.springframework.ai.ollama.api.OllamaApi;
import org.springframework.ai.ollama.api.OllamaChatOptions;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * A separate, deterministic model for the judge. The builder is not a bean, so it cannot be confused with the
 * auto-configured ChatClient.Builder that the application itself injects.
 */
@Configuration(proxyBeanMethods = false)
class JudgeConfig {

    @Bean
    RelevancyEvaluator relevancyEvaluator(OllamaApi ollamaApi,
            @Value("${docsassistant.judge.model:llama3.1:8b}") String judgeModel) {
        return RelevancyEvaluator.builder().chatClientBuilder(judge(ollamaApi, judgeModel)).build();
    }

    @Bean
    FactCheckingEvaluator factCheckingEvaluator(OllamaApi ollamaApi,
            @Value("${docsassistant.judge.model:llama3.1:8b}") String judgeModel) {
        return FactCheckingEvaluator.builder(judge(ollamaApi, judgeModel)).build();
    }

    private static ChatClient.Builder judge(OllamaApi ollamaApi, String judgeModel) {
        OllamaChatModel model = OllamaChatModel.builder()
                .ollamaApi(ollamaApi)
                .options(OllamaChatOptions.builder().model(judgeModel).temperature(0.0).seed(42).numPredict(8).build())
                .build();
        return ChatClient.builder(model);
    }
}

Source: Spring AI: Model Evaluation.

JudgeSupport.java: test helper that points a chat client at WireMock
package com.example.docsassistant;

import com.github.tomakehurst.wiremock.WireMockServer;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.ollama.OllamaChatModel;
import org.springframework.ai.ollama.api.OllamaApi;
import org.springframework.ai.ollama.api.OllamaChatOptions;

/** Builds Ollama-backed chat clients that talk to a WireMock server instead of a real model. */
final class JudgeSupport {

    private JudgeSupport() {
    }

    static ChatClient.Builder clientFor(WireMockServer server, String model) {
        OllamaChatModel chatModel = OllamaChatModel.builder()
                .ollamaApi(OllamaApi.builder().baseUrl(server.baseUrl()).build())
                .options(OllamaChatOptions.builder().model(model).temperature(0.0).seed(42).build())
                .build();
        return ChatClient.builder(chatModel);
    }
}

Source: Spring AI: Model Evaluation.

EvaluatorsTest.java: relevancy, fact checking, a chatty judge, and two custom evaluators
package com.example.docsassistant;

import static org.assertj.core.api.Assertions.assertThat;

import java.util.List;
import java.util.Map;

import com.github.tomakehurst.wiremock.WireMockServer;
import com.github.tomakehurst.wiremock.core.WireMockConfiguration;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.evaluation.FactCheckingEvaluator;
import org.springframework.ai.chat.evaluation.RelevancyEvaluator;
import org.springframework.ai.document.Document;
import org.springframework.ai.evaluation.EvaluationRequest;

class EvaluatorsTest {

    private final List<Document> context = List.of(new Document(
            "MatrixInverse computes the inverse of a square matrix using LU decomposition.",
            Map.of("source_path", "irurueta-algebra/matrices.adoc")));

    private WireMockServer ollama;

    @BeforeEach
    void start() {
        this.ollama = new WireMockServer(WireMockConfiguration.options().dynamicPort());
        this.ollama.start();
    }

    @AfterEach
    void stop() {
        this.ollama.stop();
    }

    private RelevancyEvaluator relevancy() {
        return RelevancyEvaluator.builder().chatClientBuilder(JudgeSupport.clientFor(this.ollama, "judge")).build();
    }

    @Test
    void relevancyEvaluatorPassesOnYes() {
        OllamaStubs.chatReply(this.ollama, "judge", "YES", 80, 1);

        var result = relevancy().evaluate(new EvaluationRequest("How do I invert a matrix?", this.context,
                "Use MatrixInverse."));

        assertThat(result.isPass()).isTrue();
        assertThat(result.getScore()).isEqualTo(1.0f);
    }

    @Test
    void relevancyEvaluatorFailsOnNo() {
        OllamaStubs.chatReply(this.ollama, "judge", "NO", 80, 1);

        var result = relevancy().evaluate(new EvaluationRequest("How do I invert a matrix?", this.context,
                "Bake at 180 degrees for 20 minutes."));

        assertThat(result.isPass()).isFalse();
    }

    @Test
    void aChattyJudgeFailsEveryAnswer() {
        // the evaluators compare the whole reply with "yes"; "Yes, it is." is not a pass
        OllamaStubs.chatReply(this.ollama, "judge", "Yes, the response is in line with the context.", 80, 12);

        var result = relevancy().evaluate(new EvaluationRequest("How do I invert a matrix?", this.context,
                "Use MatrixInverse."));

        assertThat(result.isPass()).isFalse();
    }

    @Test
    void factCheckerSendsTheContextAsDocumentAndTheAnswerAsClaim() {
        OllamaStubs.chatReply(this.ollama, "judge", "yes", 60, 1);
        FactCheckingEvaluator checker = FactCheckingEvaluator
                .builder(JudgeSupport.clientFor(this.ollama, "judge")).build();

        var result = checker.evaluate(new EvaluationRequest("How do I invert a matrix?", this.context,
                "MatrixInverse uses LU decomposition."));

        assertThat(result.isPass()).isTrue();
        this.ollama.verify(com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor(
                com.github.tomakehurst.wiremock.client.WireMock.urlEqualTo("/api/chat"))
                .withRequestBody(com.github.tomakehurst.wiremock.client.WireMock.containing("Claim:"))
                .withRequestBody(com.github.tomakehurst.wiremock.client.WireMock.containing("LU decomposition")));
    }

    @Test
    void citationEvaluatorIsPlainJava() {
        var evaluator = new CitationEvaluator();

        var cited = evaluator.evaluate(new EvaluationRequest("q", this.context,
                "Use MatrixInverse (irurueta-algebra/matrices.adoc)."));
        var uncited = evaluator.evaluate(new EvaluationRequest("q", this.context, "Use MatrixInverse."));

        assertThat(cited.isPass()).isTrue();
        assertThat(uncited.isPass()).isFalse();
        assertThat(uncited.getFeedback()).contains("cite");
    }

    @Test
    void scoreEvaluatorParsesTheJudgeVerdict() {
        OllamaStubs.chatReply(this.ollama, "judge", "{\"rating\": 3, \"feedback\": \"Correct but terse.\"}", 90, 14);
        var evaluator = new JudgeScoreEvaluator(JudgeSupport.clientFor(this.ollama, "judge"), 3);

        var result = evaluator.evaluate(new EvaluationRequest("How do I invert a matrix?", this.context,
                "Use MatrixInverse."));

        assertThat(result.isPass()).isTrue();
        assertThat(result.getScore()).isEqualTo(0.75f);
        assertThat(result.getFeedback()).isEqualTo("Correct but terse.");
    }
}

Source: Spring AI: Model Evaluation.

All six tests passed. The stub server is OllamaStubs from Observability, which answers only requests addressed to one model name, so an answer model and a judge can share a server.

FactCheckingEvaluator

FactCheckingEvaluator treats the context as a document and the answer as a claim and asks whether the document supports the claim. It is the check for hallucination: a relevant answer can still contain an invented detail. Its default prompt asks for "yes" or "no", and forBespokeMinicheck(builder) selects the shorter prompt that the dedicated fact-checking model Bespoke-Minicheck expects. That model is small and cheap, is available through Ollama, and is a better fit for this one job than a general chat model. It builds its response without a score, so getScore() is 0 even when the claim passes; read isPass(). Build the evaluator with the builder: FactCheckingEvaluator.builder(clientBuilder).build(). The reference page shows new FactCheckingEvaluator(builder), but the constructor in 2.0.1 is protected, so that line does not compile from application code.

The fourth test above sends an answer to the checker and then uses WireMock’s request matching to assert what the judge was asked: the prompt contains Claim: and the passage. Verifying the judge’s request is worth doing once, because a template that silently loses the context turns a strict checker into a rubber stamp.

LLM-as-a-judge

Using a model to grade a model works because judging is easier than generating: the judge does not have to compose an answer, only compare one with a rubric. The reference’s guide names two patterns. Point-wise scoring grades one answer on a scale. Pairwise comparison picks the better of two candidates, which suits A/B tests of prompts or models. The rules from the guide and from the research it cites (Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023) are practical:

  • use a different model for the judge than for the answer, to avoid a model preferring its own style;

  • set the temperature to zero and, where the provider supports it, a seed;

  • ask for an integer scale with a written rubric, and give examples when the scale is subtle;

  • expect biases such as preferring longer answers or the first candidate in a pairwise comparison, and randomise the order when you compare;

  • keep a human in the loop for high-stakes decisions, and check the judge itself against a few hundred human-labelled answers.

The last rule is not optional. A smoke run of both evaluators against a real model in Testcontainers Ollama used smollm2:135m, a 135-million-parameter model chosen to keep the download small, and showed what a weak judge does: for the question "What is the capital of France?" with the context "Paris is the capital of France.", RelevancyEvaluator failed the correct answer, FactCheckingEvaluator passed it and rejected "Berlin is the capital of France.", and a plain "Answer only YES or NO" prompt answered "Yes" to the false claim. Its verdicts are noise. Choose the judge model on a labelled set before you trust any number it produces.

A YES/NO string also throws away information. When you need a score and a reason, ask for a structured verdict and parse it into a record with entity(…​), as described on Structured Output:

JudgeScoreEvaluator.java: a point-wise 1 to 4 rating with feedback
package com.example.docsassistant;

import java.util.Map;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.evaluation.EvaluationRequest;
import org.springframework.ai.evaluation.EvaluationResponse;
import org.springframework.ai.evaluation.Evaluator;

/** Point-wise LLM-as-a-judge: a 1 to 4 rating plus feedback, parsed into a record instead of a YES/NO string. */
public class JudgeScoreEvaluator implements Evaluator {

    record Verdict(int rating, String feedback) { }

    private final ChatClient judge;
    private final int passingRating;

    public JudgeScoreEvaluator(ChatClient.Builder judgeBuilder, int passingRating) {
        this.judge = judgeBuilder.build();
        this.passingRating = passingRating;
    }

    @Override
    public EvaluationResponse evaluate(EvaluationRequest request) {
        Verdict verdict = this.judge.prompt()
                .system("""
                        You grade answers to documentation questions. Rate the answer from 1 (useless) to 4 \
                        (complete and correct given the context). Reply with the rating and one sentence of feedback.""")
                .user(u -> u.text("""
                        Question: {question}
                        Context: {context}
                        Answer: {answer}""")
                        .param("question", request.getUserText())
                        .param("context", doGetSupportingData(request))
                        .param("answer", request.getResponseContent()))
                .call()
                .entity(Verdict.class);
        int rating = verdict == null ? 1 : verdict.rating();
        String feedback = verdict == null ? "no verdict" : verdict.feedback();
        return new EvaluationResponse(rating >= this.passingRating, rating / 4.0f, feedback, Map.of("rating", rating));
    }
}

Source: Spring AI: LLM-as-a-Judge guide.

The last test of EvaluatorsTest feeds this evaluator a scripted verdict of 3 and checks the pass, the score of 0.75 and the feedback. The rubric text is deliberately short. In a real project it lives in a versioned prompt template (see Prompts & Templates) so a change of rubric is reviewed like any other change.

Custom evaluators

Not everything needs a model. A rule-based Evaluator is fast, free, deterministic and explainable, so put it first. DocsAssistant’s rule is that an answer cites the source path of a passage it was given:

CitationEvaluator.java: a rule-based evaluator
package com.example.docsassistant;

import java.util.Map;

import org.springframework.ai.document.Document;
import org.springframework.ai.evaluation.EvaluationRequest;
import org.springframework.ai.evaluation.EvaluationResponse;
import org.springframework.ai.evaluation.Evaluator;

/** A rule-based evaluator: the answer must cite the source path of at least one retrieved passage. */
public class CitationEvaluator implements Evaluator {

    @Override
    public EvaluationResponse evaluate(EvaluationRequest request) {
        long cited = request.getDataList().stream()
                .map(doc -> (String) doc.getMetadata().get("source_path"))
                .filter(path -> path != null && request.getResponseContent().contains(path))
                .count();
        boolean pass = cited > 0;
        float score = request.getDataList().isEmpty() ? 0f : (float) cited / request.getDataList().size();
        String feedback = pass ? "" : "The answer cites none of the retrieved sources; cite at least one source path.";
        return new EvaluationResponse(pass, score, feedback, Map.of("cited", cited));
    }
}

Source: Spring AI: Model Evaluation.

The score is the share of retrieved passages that the answer cites, and the feedback tells the caller what to fix, which the self-evaluation below sends back to the model. Chain cheap evaluators before expensive ones and stop at the first failure. A citation that exists proves nothing about whether it supports the sentence it follows, so the rule check complements the model-based checks and does not replace them.

Runtime self-evaluation with retry

An evaluator can run in production before the user sees an answer. The pattern is generate, check, and on failure regenerate, with a limit. Its cost is one judge call per attempt and the wait for it, so use it for answers where being wrong is expensive.

flowchart LR Q["Question"] --> R["Retrieve passages"] R --> G["Generate answer"] G --> E{"Evaluator passes?"} E -- yes --> A["Return answer"] E -- "no, attempts left" --> W["Wait, then try again"] W --> G E -- "no, out of attempts" --> F["Fail or return a fallback"]

With pass probability for one attempt and at most attempts, the expected number of attempts is

For and that is attempts, and each attempt is a generation plus a judge call, so about 2.8 model calls per answer instead of one, and never more than six. The formula assumes independent attempts. When the failure comes from a bad retrieval or a wrong question, the retry fails the same way and you only paid more.

Spring Framework 7 has resilience annotations built in, so @Retryable needs no extra library. @EnableResilientMethods turns them on and the annotation takes the exceptions to retry, the number of retries, the delay and a multiplier:

LowQualityAnswerException.java
package com.example.docsassistant;

public class LowQualityAnswerException extends RuntimeException {

    private final String feedback;

    public LowQualityAnswerException(String feedback) {
        super("answer rejected by the evaluator: " + feedback);
        this.feedback = feedback;
    }

    public String feedback() {
        return this.feedback;
    }
}

Source: Spring Framework: Resilience features.

The service below takes the retrieval step as a one-method interface, so the pgvector search of DocsTools in production and a fixed list of passages in a test both plug in:

DocsRetriever.java: the retrieval step as a seam
package com.example.docsassistant;

import java.util.List;

import org.springframework.ai.document.Document;

/** The retrieval step of DocsAssistant; the production implementation queries the pgvector store. */
public interface DocsRetriever {

    List<Document> retrieve(String question);
}

Source: Spring AI: LLM-as-a-Judge guide, for the generate, check and refine pattern.

EvaluatedAnswerService.java: answer, check against the context, retry on rejection
package com.example.docsassistant;

import java.util.List;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.evaluation.FactCheckingEvaluator;
import org.springframework.ai.document.Document;
import org.springframework.ai.evaluation.EvaluationRequest;
import org.springframework.ai.evaluation.EvaluationResponse;
import org.springframework.resilience.annotation.Retryable;
import org.springframework.stereotype.Service;

/** Answers from retrieved passages and checks the answer against them before it reaches the user. */
@Service
public class EvaluatedAnswerService {

    private final ChatClient chatClient;
    private final DocsRetriever retriever;
    private final FactCheckingEvaluator factChecker;

    public EvaluatedAnswerService(ChatClient.Builder builder, DocsRetriever retriever, FactCheckingEvaluator factChecker) {
        this.chatClient = builder.build();
        this.retriever = retriever;
        this.factChecker = factChecker;
    }

    /** Each call retries, with a growing pause, when the fact checker rejects the answer. */
    @Retryable(includes = LowQualityAnswerException.class, maxRetries = 2, delay = 200, multiplier = 2.0)
    public String answer(String question) {
        List<Document> context = this.retriever.retrieve(question);
        String answer = this.chatClient.prompt()
                .system("Answer only from the context. Cite the source path of every passage you use.")
                .user(u -> u.text("Question: {question}\n\nContext:\n{context}")
                        .param("question", question)
                        .param("context", String.join("\n---\n", context.stream().map(Document::getText).toList())))
                .call()
                .content();
        EvaluationResponse verdict = this.factChecker.evaluate(new EvaluationRequest(question, context, answer));
        if (!verdict.isPass()) {
            throw new LowQualityAnswerException(verdict.getFeedback());
        }
        return answer;
    }
}

Source: Spring Framework: Resilience features.

maxRetries = 2 means three attempts in total. The next test uses a real Spring context with @EnableResilientMethods, a stubbed answer model and a judge that says "no" twice and then "yes" (WireMock scenarios), then a judge that always says "no":

EvaluatedAnswerServiceTest.java: retry until the judge accepts, then give up
package com.example.docsassistant;

import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
import static com.github.tomakehurst.wiremock.client.WireMock.equalTo;
import static com.github.tomakehurst.wiremock.client.WireMock.matchingJsonPath;
import static com.github.tomakehurst.wiremock.client.WireMock.post;
import static com.github.tomakehurst.wiremock.client.WireMock.urlEqualTo;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;

import java.util.List;
import java.util.Map;

import com.github.tomakehurst.wiremock.WireMockServer;
import com.github.tomakehurst.wiremock.core.WireMockConfiguration;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.evaluation.FactCheckingEvaluator;
import org.springframework.ai.document.Document;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.resilience.annotation.EnableResilientMethods;

class EvaluatedAnswerServiceTest {

    @Configuration
    @EnableResilientMethods
    static class Config {

        static WireMockServer ollama;

        @Bean
        DocsRetriever retriever() {
            return question -> List.of(new Document("MatrixInverse uses LU decomposition.",
                    Map.of("source_path", "irurueta-algebra/matrices.adoc")));
        }

        @Bean
        FactCheckingEvaluator factChecker() {
            return FactCheckingEvaluator.builder(JudgeSupport.clientFor(ollama, "judge")).build();
        }

        @Bean
        EvaluatedAnswerService service(DocsRetriever retriever, FactCheckingEvaluator factChecker) {
            return new EvaluatedAnswerService(JudgeSupport.clientFor(ollama, "llama3.1:8b"), retriever, factChecker);
        }
    }

    private WireMockServer ollama;

    @BeforeEach
    void start() {
        this.ollama = new WireMockServer(WireMockConfiguration.options().dynamicPort());
        this.ollama.start();
        Config.ollama = this.ollama;
        OllamaStubs.chatReply(this.ollama, "llama3.1:8b", "MatrixInverse uses LU decomposition.", 120, 9);
    }

    @AfterEach
    void stop() {
        this.ollama.stop();
    }

    /** The judge says no on the first call and yes afterwards. */
    private void judgeRejectsFirst(String... verdicts) {
        String state = "Started";
        for (int i = 0; i < verdicts.length; i++) {
            String next = "verdict-" + (i + 1);
            this.ollama.stubFor(post(urlEqualTo("/api/chat"))
                    .withRequestBody(matchingJsonPath("$.model", equalTo("judge")))
                    .inScenario("judge").whenScenarioStateIs(state).willSetStateTo(next)
                    .willReturn(aResponse().withHeader("Content-Type", "application/json").withBody("""
                            {"model":"judge","created_at":"2026-09-29T10:00:00Z",
                             "message":{"role":"assistant","content":"%s"},"done":true,"done_reason":"stop"}
                            """.formatted(verdicts[i]))));
            state = next;
        }
    }

    @Test
    void aRejectedAnswerIsRetriedUntilTheJudgeAccepts() {
        judgeRejectsFirst("no", "no", "yes");
        try (var context = new AnnotationConfigApplicationContext(Config.class)) {
            String answer = context.getBean(EvaluatedAnswerService.class).answer("How do I invert a matrix?");

            assertThat(answer).contains("LU decomposition");
            this.ollama.verify(3, com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor(urlEqualTo("/api/chat"))
                    .withRequestBody(matchingJsonPath("$.model", equalTo("llama3.1:8b"))));
        }
    }

    @Test
    void afterTheLastRetryTheFailureSurfaces() {
        judgeRejectsFirst("no", "no", "no", "no");
        try (var context = new AnnotationConfigApplicationContext(Config.class)) {
            var service = context.getBean(EvaluatedAnswerService.class);

            assertThatThrownBy(() -> service.answer("How do I invert a matrix?")).isInstanceOf(LowQualityAnswerException.class);
            // one attempt plus maxRetries = 2
            this.ollama.verify(3, com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor(urlEqualTo("/api/chat"))
                    .withRequestBody(matchingJsonPath("$.model", equalTo("llama3.1:8b"))));
        }
    }
}

Source: Spring Framework: Resilience features.

Both tests passed: the accepted case made three answer-model calls, and after the last retry the caller received the original LowQualityAnswerException, not a wrapper, so the endpoint can map it to a fallback response. Keep spring.ai.retry. and this annotation apart in your head: the former retries transport failures (timeouts, 429, 5xx) inside the model client, this one retries *bad answers. Both are tuned on Production & Deployment.

The alternative is to put the loop in an advisor, so every ChatClient call gets it without a service class. A recursive advisor calls the rest of the chain more than once with chain.copy(this).nextCall(…​), which starts a fresh sub-chain after this advisor. This one adds the evaluator’s feedback to the prompt on each retry, the refinement step of the pattern in the reference’s LLM-as-a-judge guide:

SelfEvaluationAdvisor.java: a recursive advisor with feedback
package com.example.docsassistant;

import java.util.List;

import org.springframework.ai.chat.client.ChatClientRequest;
import org.springframework.ai.chat.client.ChatClientResponse;
import org.springframework.ai.chat.client.advisor.api.CallAdvisor;
import org.springframework.ai.chat.client.advisor.api.CallAdvisorChain;
import org.springframework.ai.document.Document;
import org.springframework.ai.evaluation.EvaluationRequest;
import org.springframework.ai.evaluation.EvaluationResponse;
import org.springframework.ai.evaluation.Evaluator;
import org.springframework.core.Ordered;

/** Calls the rest of the chain again, with the evaluator's feedback, until the answer passes or attempts run out. */
public class SelfEvaluationAdvisor implements CallAdvisor {

    private final Evaluator evaluator;
    private final int maxAttempts;

    public SelfEvaluationAdvisor(Evaluator evaluator, int maxAttempts) {
        this.evaluator = evaluator;
        this.maxAttempts = maxAttempts;
    }

    @Override
    public String getName() {
        return "SelfEvaluationAdvisor";
    }

    @Override
    public int getOrder() {
        return Ordered.HIGHEST_PRECEDENCE + 250; // inside memory (+200), outside tool calling (+300)
    }

    @Override
    public ChatClientResponse adviseCall(ChatClientRequest request, CallAdvisorChain chain) {
        ChatClientRequest current = request;
        ChatClientResponse response = null;
        for (int attempt = 1; attempt <= this.maxAttempts; attempt++) {
            response = chain.copy(this).nextCall(current); // a fresh sub-chain starting after this advisor
            EvaluationResponse verdict = this.evaluator.evaluate(new EvaluationRequest(
                    request.prompt().getUserMessage().getText(), List.<Document>of(),
                    response.chatResponse().getResult().getOutput().getText()));
            if (verdict.isPass()) {
                return response;
            }
            String feedback = verdict.getFeedback();
            current = request.mutate()
                    .prompt(request.prompt().augmentUserMessage(user -> user.mutate()
                            .text(user.getText() + "\n\nYour previous answer was rejected: " + feedback
                                    + "\nAnswer again and fix that.")
                            .build()))
                    .build();
        }
        return response; // out of attempts: return the last answer rather than nothing
    }
}

Source: Spring AI: Recursive Advisors.

SelfEvaluationAdvisorTest.java
package com.example.docsassistant;

import static org.assertj.core.api.Assertions.assertThat;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;

import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.model.Generation;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.evaluation.EvaluationResponse;

class SelfEvaluationAdvisorTest {

    @Test
    void feedbackIsAddedToTheRetriedPrompt() {
        List<String> prompts = new ArrayList<>();
        AtomicInteger calls = new AtomicInteger();
        ChatModel model = prompt -> {
            prompts.add(prompt.getUserMessage().getText());
            String text = calls.incrementAndGet() == 1 ? "I do not know." : "Use MatrixInverse.";
            return new ChatResponse(List.of(new Generation(new AssistantMessage(text))));
        };
        var advisor = new SelfEvaluationAdvisor(request -> request.getResponseContent().contains("MatrixInverse")
                ? new EvaluationResponse(true, "", java.util.Map.of())
                : new EvaluationResponse(false, "Name the class that does it.", java.util.Map.of()), 3);
        ChatClient client = ChatClient.builder(model).defaultAdvisors(advisor).build();

        String answer = client.prompt().user("How do I invert a matrix?").call().content();

        assertThat(answer).isEqualTo("Use MatrixInverse.");
        assertThat(prompts).hasSize(2);
        assertThat(prompts.get(1)).contains("Your previous answer was rejected: Name the class that does it.");
    }
}

Source: Spring AI: Recursive Advisors.

The test passed: the second prompt contained "Your previous answer was rejected: Name the class that does it." The advisor evaluates the answer without context (it has only the question and answer), so a rule such as "names a class" fits it, while a fact check needs the retrieved passages and belongs in the service above. The advisor is CallAdvisor only; the reference states that recursive advisors are non-streaming, cost extra model calls and need care when an inner advisor keeps external state, because that state is touched on every iteration. Advisors run in ascending order, lowest outermost, so HIGHEST_PRECEDENCE + 250 puts it between the memory advisor (+200) and tool calling (+300), the ordering rule of Advisors. Memory wraps the loop and runs once, while each attempt runs a complete tool loop and only its final answer is judged. A higher value such as +500 would put the advisor inside the tool loop, where it would judge, and re-prompt, every intermediate tool-call reply. A memory advisor placed inside the loop would run once per attempt. Always bound the loop: an evaluator that never passes plus no limit is an unbounded bill.

Golden-set tests in JUnit

A golden set is a versioned list of questions with what a good answer must contain or cite. It is the regression suite of a model-backed feature: change the prompt, the model, the chunking or the retriever, run the set, compare. Keep it in the repository as a CSV next to the tests:

src/test/resources/golden-set.csv
question,expectedSource,mustMention
How do I invert a matrix?,irurueta-algebra/matrices.adoc,MatrixInverse
Which class computes the mean of a sample?,irurueta-statistics/descriptive.adoc,mean
How do I sort an array in place?,irurueta-sorting/sorters.adoc,Sorter

Source: Spring AI: Model Evaluation.

WilsonInterval.java: how much a pass rate really proves
package com.example.docsassistant;

/** Lower bound of the 95% Wilson score interval for a pass rate measured on a small golden set. */
public final class WilsonInterval {

    private static final double Z = 1.96;

    private WilsonInterval() {
    }

    public static double lowerBound(int passes, int total) {
        if (total == 0) {
            return 0.0;
        }
        double p = (double) passes / total;
        double z2 = Z * Z;
        double centre = p + z2 / (2.0 * total);
        double margin = Z * Math.sqrt(p * (1 - p) / total + z2 / (4.0 * total * total));
        return (centre - margin) / (1 + z2 / total);
    }
}

Source: Wilson score interval.

GoldenSetTest.java: one test per row, plus the sample-size arithmetic
package com.example.docsassistant;

import static org.assertj.core.api.Assertions.assertThat;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;

import com.github.tomakehurst.wiremock.WireMockServer;
import com.github.tomakehurst.wiremock.core.WireMockConfiguration;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvFileSource;
import org.springframework.ai.document.Document;
import org.springframework.ai.evaluation.EvaluationRequest;

class GoldenSetTest {

    static WireMockServer ollama;
    static EvaluatedAnswerService service;

    @BeforeAll
    static void start() {
        ollama = new WireMockServer(WireMockConfiguration.options().dynamicPort());
        ollama.start();
        // scripted answers stand in for the model here; the same test runs against a live model in the nightly build
        OllamaStubs.chatReply(ollama, "llama3.1:8b",
                "Use MatrixInverse, the mean and Sorter (irurueta-algebra/matrices.adoc, irurueta-statistics/descriptive.adoc, "
                        + "irurueta-sorting/sorters.adoc).", 100, 30);
        OllamaStubs.chatReply(ollama, "judge", "yes", 100, 1);
        DocsRetriever retriever = q -> List.of(new Document("Reference passage for: " + q, Map.of(
                "source_path", "irurueta-algebra/matrices.adoc")));
        service = new EvaluatedAnswerService(JudgeSupport.clientFor(ollama, "llama3.1:8b"), retriever,
                org.springframework.ai.chat.evaluation.FactCheckingEvaluator
                        .builder(JudgeSupport.clientFor(ollama, "judge")).build());
    }

    @AfterAll
    static void stop() {
        ollama.stop();
    }

    @ParameterizedTest(name = "{0}")
    @CsvFileSource(resources = "/golden-set.csv", numLinesToSkip = 1)
    void everyGoldenQuestionIsAnsweredFromItsSource(String question, String expectedSource, String mustMention) {
        String answer = service.answer(question);

        assertThat(answer).contains(mustMention);
        assertThat(new CitationEvaluator().evaluate(new EvaluationRequest(question,
                List.of(new Document("x", Map.of("source_path", expectedSource))), answer)).isPass()).isTrue();
    }

    @Test
    void wilsonBoundShowsHowLittleAFewQuestionsProve() {
        // 19 of 20 passing is 95 percent, but the 95 percent interval starts near 76 percent
        assertThat(WilsonInterval.lowerBound(19, 20)).isBetween(0.75, 0.77);
        // the same 95 percent measured on 200 questions is far tighter
        assertThat(WilsonInterval.lowerBound(190, 200)).isBetween(0.91, 0.92);
        List<Double> rates = new ArrayList<>();
        for (int n : new int[] { 10, 20, 50, 100, 200 }) {
            rates.add(WilsonInterval.lowerBound((int) Math.round(0.95 * n), n));
        }
        System.out.println("WILSON " + rates);
    }
}

Source: Spring AI: Model Evaluation and Wilson score interval.

The suite passed with scripted answers standing in for the model, which proves the harness (CSV parsing, evaluators, citation rule, retry-free path) and nothing about the model. The same test class runs against a live model in a scheduled job: replace the stubs with the Ollama container of Testing and tag the class so the normal build skips it.

Answers vary, so a single run is a sample. Report the pass rate with its uncertainty. For passes out of questions, the lower end of the 95 percent Wilson score interval is

The test asserts two values: 19 passes out of 20 (95 percent) has a lower bound of about 0.76, while 190 out of 200 has about 0.91. Twenty questions cannot show that a system is 95 percent good, only that it is unlikely to be worse than roughly 76 percent, so gate a release on the lower bound, or on a change relative to the previous run using the same questions, and grow the set from every production failure.

Table 3. A practical cadence
Run What it checks

Every commit

Rule-based evaluators and scripted-model tests: seconds, free, deterministic

Before merge, on prompt or retrieval changes

The golden set against a live model, with the lower bound printed in the build log

Nightly

A larger set, the second judge model, and the drift against the last release

In production

Runtime self-evaluation on the answers where an error is costly, with the results counted (see Observability); trends over weeks belong to LLMOps

Common problems

Table 4. Symptoms of a broken evaluation
Symptom Likely cause Fix

Every answer fails, however good

The judge replies with a sentence ("Yes, it is.") and the evaluator compares the whole reply with "yes"

Lower numPredict, tighten the judge prompt, or write a custom evaluator that parses the reply

Everything passes

The template lost the context (empty dataList) or the judge is too weak

Assert the judge’s request as in the fourth test, and measure the judge against labelled data

The suite is flaky

Judge or answer temperature above zero, no seed, or a golden set too small for the threshold

Temperature zero, a seed, and gate on a lower bound

The retry loop never ends

A recursive advisor without an attempt limit

Bound it, and count attempts in a metric

Where older material differs

Table 5. Older evaluation material and the 2.0.1 API
Older text Spring AI 2.0.1

new FactCheckingEvaluator(builder)

FactCheckingEvaluator.builder(builder).build() or forBespokeMinicheck(builder); the constructor is protected

EvaluationRequest with List<Content>

List<Document>

@Retryable from the separate Spring Retry library

Spring Framework 7’s org.springframework.resilience.annotation.Retryable with @EnableResilientMethods; the Spring AI retry now uses org.springframework.core.retry.RetryTemplate

For the retrieval side of a RAG evaluation with the advisors, see the "Testing" section of Integrating with Spring AI.

References