Evaluating RAG Systems

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Examples on this page target Python 3.12 with RAGAS 0.4.3, and Spring Boot 4.1 with Spring AI 2.0.1. A RAG pipeline has two independent failure modes, so it needs two independent kinds of evaluation: the retrieval stage can return the wrong chunks (or the right chunks ranked too low to survive a top-k cut), and the generation stage can produce an answer that is not actually supported by whatever chunks it did receive. Improving one does not fix the other, so every metric below is scoped to one stage or the other.

Retrieval vs. generation

  • Retrieval is evaluated by asking: given a query, did the vector search return the chunks a human (or a reference answer) says are relevant, and how highly are they ranked? This is classic information-retrieval (IR) evaluation, unchanged by the fact that the retriever happens to be an ANN index instead of a keyword index.

  • Generation is evaluated by asking: given the chunks that were actually retrieved, is the LLM’s answer faithful to them (it does not invent claims the context does not support) and relevant to the question (it does not wander off-topic even if every sentence is technically grounded)?

A separate, narrower check belongs to the retrieval side alone: ANN recall, meaning whether the approximate index (HNSW, IVFFlat, …​) finds the same neighbours an exact, brute-force search would. This is not "did we retrieve the right document" — it is "did the ANN index approximate exact search well enough." It is measured by running the same query against the tuned ANN index and against a flat (exact) index built from the same vectors, then comparing the two result sets:

ANN Indexes already introduces this quantity as the axis every ANN parameter (M, ef_search, nlist, nprobe) trades against latency and memory; this page treats it as one input into a larger evaluation, not a new concept. FAISS’s IndexFlatIP is the natural exact baseline, exactly as used there and in Comparing Vector Databases's benchmark harness:

import faiss
import numpy as np

d = 768
xb = np.random.default_rng(0).random((50_000, d), dtype=np.float32)
faiss.normalize_L2(xb)

flat = faiss.IndexFlatIP(d)          # exact ground truth
flat.add(xb)

hnsw = faiss.IndexHNSWFlat(d, 16)    # approximate index under test
hnsw.hnsw.efConstruction = 64
hnsw.add(xb)

xq = xb[:100]
_, exact_ids = flat.search(xq, 10)

hnsw.hnsw.efSearch = 64
_, ann_ids = hnsw.search(xq, 10)

recall_at_10 = np.mean([
    len(set(ann_ids[i]) & set(exact_ids[i])) / 10
    for i in range(len(xq))
])
print(f"recall@10 vs. flat index: {recall_at_10:.3f}")

See the FAISS index wiki for IndexFlatIP and IndexHNSWFlat, and ANN-Benchmarks for how this recall-vs-latency trade-off is measured and compared across ANN libraries at scale.

Information-retrieval metrics

These metrics score a ranked list of retrieved chunks against a set of chunks known to be relevant for a given query (a "qrel" — query relevance judgment). k is the cutoff, typically the same top-k the pipeline actually retrieves (e.g. 5).

Precision@k — the fraction of the top k retrieved items that are actually relevant:

Recall@k — the fraction of all relevant items that were retrieved in the top k:

Mean Reciprocal Rank (MRR) — averaged over queries, the reciprocal of the rank of the first relevant item (1.0 if it is first, 0.5 if it is second, and so on; 0 if none is found):

Normalized Discounted Cumulative Gain (nDCG@k) — like recall, but rewards graded relevance (a chunk can be "highly relevant", "somewhat relevant", or "irrelevant") and discounts items that rank lower, since a relevant chunk buried at position 9 helps a downstream LLM far less than one at position 1:

where is the DCG of the ideal ordering (all relevant items ranked first), so falls between 0 and 1.

Hit rate@k — the fraction of queries for which at least one relevant item appears in the top k (a coarser, query-level version of recall, useful when a single good chunk is enough to answer the question):

A minimal Python implementation of precision@k and recall@k over a set of queries, each with a set of relevant chunk ids and a ranked list the retriever returned:

def precision_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    top_k = retrieved[:k]
    return len(set(top_k) & relevant) / k

def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    top_k = retrieved[:k]
    return len(set(top_k) & relevant) / len(relevant)

These are the standard IR-evaluation definitions, the same ones RAGAS’s own retrieval metrics below build on; see the Wikipedia: Evaluation measures (information retrieval) overview for the formal treatment this page summarizes, and Pinecone’s Evaluation Measures in Information Retrieval (vendor content) for a worked walk-through of the same metrics.

A map of RAG evaluation metrics: retrieval metrics (precision@k, recall@k, MRR, nDCG, hit rate, ANN recall, context precision/recall) feeding into generation metrics (faithfulness, answer relevancy, LLM-as-a-judge), plus online signals (click-through rate, thumbs up/down, follow-up rate) collected once the system is live

A golden question set

Every metric above needs ground truth: a fixed set of questions, each with a reference answer and, ideally, the set of chunks a correct retrieval should surface. DocsAssistant's golden set is small on purpose — a handful of unambiguous questions whose answers are single facts, so a mismatch is unambiguous too:

Question Reference answer

What does ef_search control in an HNSW index?

the size of the candidate list during search

Which pgvector operator computes cosine distance?

<⇒

Which Spring AI advisor adds retrieved documents to the prompt?

QuestionAnswerAdvisor

These questions and answers are defined once and reused verbatim across this documentation set; the second one is answered in detail in PostgreSQL & pgvector, and the third in Integrating with Spring AI. A real golden set is larger — typically 30-100 questions, sampled from real user queries where possible, and tagged by the component the answer should come from (vector-rag or spring-ai) so a regression can be localized to one part of the corpus.

The spot-check method runs each golden question through the whole pipeline and inspects three things by hand (or with an LLM judge, below), rather than relying on a single aggregate score:

  • Retrieval depth — how far down the ranked list the chunk containing the reference answer appears (ideally rank 1, but rank 2-3 with a generous top-k is often acceptable).

  • Context relevance — whether the retrieved chunks are actually about the question, independent of whether the final answer is correct (a chunk can be topically relevant but still fail to contain the specific fact asked for).

  • Faithfulness — whether the generated answer’s claims are traceable to the retrieved context, or whether the model filled a gap with something it already knew (or invented).

golden_set = [
    {"question": "What does ef_search control in an HNSW index?",
     "reference_answer": "the size of the candidate list during search"},
    {"question": "Which pgvector operator computes cosine distance?",
     "reference_answer": "<=>"},
    {"question": "Which Spring AI advisor adds retrieved documents to the prompt?",
     "reference_answer": "QuestionAnswerAdvisor"},
]

for item in golden_set:
    hits = search_docs(item["question"], k=5)              # DocsAssistant retriever
    answer = rag_answer(item["question"], hits)             # DocsAssistant generator
    print(item["question"])
    print("  retrieval depth:", next(
        (i for i, h in enumerate(hits, start=1)
         if item["reference_answer"].lower() in h["content"].lower()), None))
    print("  answer:", answer)

RAGAS metrics

RAGAS is a Python library purpose-built for RAG evaluation. It scores a dataset of samples — each one a question (user_input), the chunks retrieved for it (retrieved_contexts), the generated answer (response) and the reference answer (reference) — against several metrics, using an LLM as the scoring mechanism (below):

  • faithfulness — decomposes the generated answer into individual claims and checks how many are entailed by the retrieved context, scored 0-1.

  • answer relevancy — generates candidate questions the answer could be responding to and compares their embedding similarity to the actual question, penalizing answers that are grounded but off-topic or incomplete.

  • context precision — among the retrieved chunks, what fraction are actually useful for answering the question (a precision-like metric, but scored by an LLM rather than a fixed relevance set).

  • context recall — whether the retrieved chunks, together, cover everything the reference answer needs (an LLM checks the reference answer sentence by sentence against the retrieved context).

A runnable example over the DocsAssistant golden set, using RAGAS 0.4.3 against the local Ollama models already used everywhere else on this page:

from langchain_ollama import ChatOllama, OllamaEmbeddings
from ragas import EvaluationDataset, SingleTurnSample, evaluate
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.llms import LangchainLLMWrapper
from ragas.metrics import (
    Faithfulness,
    LLMContextPrecisionWithReference,
    LLMContextRecall,
    ResponseRelevancy,
)

judge_llm = LangchainLLMWrapper(ChatOllama(model="llama3.1:8b", temperature=0.1))
judge_embeddings = LangchainEmbeddingsWrapper(OllamaEmbeddings(model="nomic-embed-text"))

samples = []
for item in golden_set:
    hits = search_docs(item["question"], k=5)
    samples.append(SingleTurnSample(
        user_input=item["question"],
        retrieved_contexts=[h["content"] for h in hits],
        response=rag_answer(item["question"], hits),
        reference=item["reference_answer"],
    ))

result = evaluate(
    dataset=EvaluationDataset(samples=samples),
    metrics=[Faithfulness(), ResponseRelevancy(), LLMContextPrecisionWithReference(), LLMContextRecall()],
    llm=judge_llm,
    embeddings=judge_embeddings,
)
print(result.to_pandas())

SingleTurnSample and EvaluationDataset replace the older Hugging Face Dataset with question/contexts/answer/ground_truth columns. RAGAS 0.4 already emits deprecation warnings for evaluate(), the metric classes imported from ragas.metrics (newer versions live in ragas.metrics.collections) and the LangChain wrappers, but this is still the path its own RAG-evaluation guide follows. See the RAGAS evaluation dataset reference for SingleTurnSample / EvaluationDataset, the RAGAS available metrics reference for the exact definitions and prompts behind each score, and the RAGAS: Evaluate a simple RAG system guide for the evaluate() flow this example follows.

LLM-as-a-judge and its biases

Every metric above that is not a plain set-overlap calculation (faithfulness, answer relevancy, context precision, context recall, and the spot-check’s faithfulness column) is scored by asking another LLM to judge the output — there is no cheaper way to check "is this claim entailed by this paragraph" at scale than running a model over it. That convenience comes with known biases the results have to be read through:

  • Self-preference bias — a judge model tends to score answers more favorably when they are stylistically similar to how it would have written them itself, independent of correctness.

  • Position bias — when a judge compares two candidate answers, the one presented first is scored slightly higher on average, regardless of content.

  • Verbosity bias — longer, more elaborate answers tend to score higher even when they are no more correct than a terse one.

  • Consistency — the same judge model can give different scores for the same input, particularly at temperature above 0, so evaluation runs should pin temperature low (as here, 0.1) and, where the budget allows, average over a few runs.

None of this means LLM-as-a-judge scores are unusable — it means they are best treated as a relative signal (did faithfulness go up or down after a chunking change?) rather than an absolute one, and that the golden question set’s exact-match checks (like the pgvector operator question above) remain the more trustworthy floor underneath the judged metrics.

Spring AI evaluators and regression testing

Spring AI ships two evaluators built for exactly this purpose, designed to run inside ordinary JUnit tests so a RAG regression is caught the same way a broken unit test is caught — in CI, before a change merges:

  • RelevancyEvaluator — checks whether the retrieved documents (and the generated response) are relevant to the user’s question, using an LLM as the judge.

  • FactCheckingEvaluator — checks whether a generated response is factually consistent with a given set of supporting documents (a faithfulness check, in RAGAS’s terminology).

@SpringBootTest
class DocsAssistantRagEvaluationTest {

    @Autowired
    private ChatClient.Builder chatClientBuilder;

    @Autowired
    private VectorStore vectorStore;

    @Test
    void answersAreRelevantToTheQuestion() {
        var chatClient = chatClientBuilder.clone()
                .defaultAdvisors(QuestionAnswerAdvisor.builder(vectorStore)
                        .searchRequest(SearchRequest.builder().topK(5).build())
                        .build())
                .build();
        String question = "Which pgvector operator computes cosine distance?";

        ChatResponse chatResponse = chatClient.prompt()
                .user(question)
                .call()
                .chatResponse();

        List<Document> documents = chatResponse.getMetadata().get(QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS);
        String response = chatResponse.getResult().getOutput().getText();

        var evaluationRequest = new EvaluationRequest(question, documents, response);
        EvaluationResponse evaluation = new RelevancyEvaluator(chatClientBuilder)
                .evaluate(evaluationRequest);

        assertThat(evaluation.isPass()).isTrue();
    }

    @Test
    void answersAreFactuallyConsistentWithRetrievedDocuments() {
        var documents = vectorStore.similaritySearch(
                SearchRequest.builder().query("ef_search").topK(5).build());
        String response = "ef_search controls the size of the candidate list during HNSW search.";

        EvaluationResponse evaluation = new FactCheckingEvaluator(chatClientBuilder)
                .evaluate(new EvaluationRequest(documents, response));

        assertThat(evaluation.isPass()).isTrue();
    }
}

The first test runs the real RAG pipeline: QuestionAnswerAdvisor is registered on the client, so it performs the retrieval, augments the prompt and then stores the documents it retrieved in the response metadata under QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS, which is exactly the context the evaluator should judge the answer against. clone() keeps the injected, shared ChatClient.Builder free of the advisor, since the evaluators build their own judge client from it. See the Spring AI evaluation testing reference for RelevancyEvaluator, FactCheckingEvaluator and reading the retrieved context from the response metadata, and Spring AI: QuestionAnswerAdvisor for the advisor itself.

Regression-testing pipeline changes. The golden set and its metrics are not a one-time report — they are the harness that catches a regression whenever the pipeline changes: a new chunking strategy, a swapped embedding model, or a re-tuned ANN index parameter. Run the same golden set through RelevancyEvaluator/FactCheckingEvaluator (or the RAGAS script above) before and after the change, on the same model versions, and compare; a drop in faithfulness or recall@k flags the change before it reaches production, the same way evaluate.py in the DocsAssistant module would be run in CI. Chunking changes are covered in depth here; embedding-model migrations (re-embedding, dual-write, cut-over) are covered in Generating Embeddings in Practice.

Online signals

Offline metrics run against a fixed golden set that never quite covers every real question. Once a RAG system is live, a second, cheaper class of signal complements it, collected passively from real usage:

  • Click-through rate on cited sources — users who trust an answer enough to open the document it cites.

  • Thumbs up / thumbs down on individual answers — direct, if sparse, feedback.

  • Follow-up rate — a user who immediately rephrases or repeats a question is implicitly signaling that the previous answer did not satisfy them.

These signals are noisy and biased by who bothers to click or vote, so they are read as trends over time and across cohorts, not as a per-answer verdict — but they are the only feedback that reflects the actual questions the golden set does not contain.

In LangChain and Spring AI

LangChain

LangChain itself does not ship a bundled RAG-evaluation module; the currently recommended path for evaluating a LangChain (or any) LLM application is LangSmith’s evaluation framework — datasets of example inputs/outputs, evaluator functions (including LLM-as-a-judge evaluators), and experiment tracking across runs — or, for RAG specifically, running the same RAGAS metrics above directly against a LangChain retriever’s output, since RAGAS is retriever-agnostic and takes plain SingleTurnSample rows (user_input, retrieved_contexts, response, reference) regardless of which framework produced them. See the LangSmith evaluation concepts page for the current recommended approach.

Spring AI

Spring AI’s evaluators (RelevancyEvaluator, FactCheckingEvaluator, shown above) are the framework-native answer to the same problem, and are the recommended way to regression-test a QuestionAnswerAdvisor or RetrievalAugmentationAdvisor pipeline inside ordinary Spring Boot tests. See Integrating with Spring AI's testing section for how these evaluators combine with Testcontainers-backed integration tests.

Evaluation beyond the golden set — synthetic test sets, reference-free metrics, CI regression gates and online sampled judging — is covered in RAG Systems in Production: Evaluation at Scale. The LLMOps & Evaluation sub-section generalizes this to any LLM application, including evaluation in CI and observability with OpenTelemetry.

Related pages

References