RAG and Agent Evaluation

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against RAGAS 0.4.3, DeepEval 4.2.7, agentevals 0.0.9 with LangSmith SDK 0.14.2, LangChain 1.4.3 on Python 3.12, and Spring AI 2.0.1 on Spring Boot 4.1.x with Java 21 (see the version baseline). Every snippet on this page was written from the official documentation of the tool it uses and was not run: each metric calls a judge model, and no judge endpoint was available when this page was written. Where the documentation did not state a detail (for example the exact placement of metrics in DeepEval’s tracing loop), the prose says so instead of guessing. No benchmark scores and no vendor prices are quoted; cost is expressed as a formula over tokens.

A RAG pipeline and an agent have the same evaluation problem: the answer is the output of several steps, and a bad answer tells you almost nothing about which step broke. This page is about decomposing the system so that a failing score points at a component. The vocabulary of retrieval metrics (recall at , MRR, nDCG) and the first RAGAS example are on Evaluating RAG Systems; synthetic test sets, UMBRELA and nugget metrics, CI gates and champion/challenger tests for RAG are on RAG Systems in Production: Evaluation at Scale; and the first agent harness, trajectory modes and rubric judges are on Agent Evaluation. This page does not repeat them. It adds the reference-free view (scoring without a written ideal answer, which is what makes sampled online evaluation possible), a failure-attribution method, the three levels of agent evaluation, reliability across repeated runs, and cost per task. The metric theory underneath is in Evaluation Methodology and the judge mechanics in LLM-as-a-Judge. Chapters on evaluating retrieval and generation separately in Mendelevitch and Bao’s Hands-On RAG for Production, and on agent evaluation in Albada’s Building Applications with AI Agents, inform the decomposition below; the text, formulas and code are original paraphrase (see the bibliography).

Why decompose

Consider a DocsAssistant answer that is wrong. At least four different faults produce it:

  • the retriever did not return the right page (a recall fault);

  • it returned the page among noise, and the model followed the noise (a precision or robustness fault);

  • the page was there and the model ignored or contradicted it (a faithfulness fault);

  • the model answered a different question than the one asked (a relevancy fault).

A single end-to-end "correct / incorrect" score cannot tell these apart, and each has a different remedy: re-chunk, rerank, change the prompt, or change the model. So an evaluation suite for RAG scores each stage and the answer separately, and an evaluation suite for an agent scores the outcome, each step and the route. The cost is a larger suite; the benefit is that a regression caught by the CI gate in Evaluation in CI names its own cause.

Reference-free RAG metrics

A reference-based metric needs a written ideal answer for every question, which is expensive and goes stale as the documentation changes (see Building Eval Sets). A reference-free metric needs only the three things any production trace already contains: the question, the retrieved passages and the generated answer. That is why reference-free metrics are the ones used on sampled live traffic in Online Evaluation and Feedback. Four of them cover the whole pipeline:

Metric Inputs it uses What a low score means

Faithfulness (groundedness)

answer, retrieved passages

Claims in the answer are not supported by what was retrieved: the model added or contradicted content.

Answer relevancy

question, answer

The answer is off topic, evasive or incomplete, whether or not it is grounded.

Context relevancy (or context precision without reference)

question, retrieved passages (and, for rank-aware variants, the answer)

The retriever returned passages that do not help: noise, or the right page ranked too low.

Task-specific rubric

whatever the rubric names

The product-specific property you care about, for example "cites a page path that exists".

The first three together are often called the RAG triad: relevance of the context to the question, groundedness of the answer in the context, and relevance of the answer to the question. If all three are high the answer is on topic, supported, and built on useful evidence; a low value points at one edge of the triangle.

Faithfulness as a formula

Faithfulness-style metrics decompose the answer into atomic claims and ask a judge whether each is supported by the retrieved context. With the set of claims extracted from the answer and the supported ones:

This is the definition given in the RAGAS paper (Es et al.) and the RAGAS documentation. Two properties matter in practice. Faithfulness says nothing about whether the answer is correct: an answer faithfully repeating an outdated page scores 1. And it is only as good as the judge’s claim extraction, so calibrate the judge against human labels on a sample before trusting a threshold (the calibration method, Cohen’s , is on the judge page).

Context precision without a reference

Rank-aware context precision averages precision at each position that holds a useful chunk. With the judged usefulness of the chunk at rank among retrieved chunks:

The reference-free variant in RAGAS (LLMContextPrecisionWithoutReference) lets the judge decide usefulness by comparing each chunk with the generated response, instead of with a reference answer. That makes it cheap to run on live traffic, with one caveat: if the response is wrong, a chunk that would have helped can be marked useless, so read this metric together with faithfulness rather than alone.

Scoring a DocsAssistant answer with RAGAS

The snippet scores one logged interaction with two reference-free RAGAS metrics. It follows the metric classes and the SingleTurnSample fields of the RAGAS documentation; as noted in the example on Evaluating RAG Systems, RAGAS 0.4 is moving metrics to ragas.metrics.collections, so check the deprecation warnings of your installed version.

docsassistant/eval/reference_free_rag.py — reference-free scores for one trace (Python, RAGAS 0.4.3, not run)
import asyncio
import os

from openai import AsyncOpenAI
from ragas import SingleTurnSample
from ragas.llms import llm_factory
from ragas.metrics import Faithfulness, LLMContextPrecisionWithoutReference

judge = llm_factory(
    os.environ["JUDGE_MODEL"],
    provider="openai",
    client=AsyncOpenAI(base_url=os.environ["JUDGE_BASE_URL"], api_key=os.environ["JUDGE_API_KEY"]),
)

faithfulness = Faithfulness(llm=judge)
context_precision = LLMContextPrecisionWithoutReference(llm=judge)


async def score_trace(question: str, answer: str, passages: list[str]) -> dict[str, float]:
    sample = SingleTurnSample(user_input=question, response=answer, retrieved_contexts=passages)
    f, p = await asyncio.gather(
        faithfulness.single_turn_ascore(sample),
        context_precision.single_turn_ascore(sample),
    )
    return {"faithfulness": f, "context_precision": p}

Source: RAGAS — Context Precision and RAGAS — Faithfulness. The documentation shows single_turn_ascore on metric instances built with an evaluator LLM; whether Faithfulness imported from ragas.metrics and the one from ragas.metrics.collections share that call signature in 0.4.3 was not verified here, which is why the earlier page uses the collections form with ascore. Pick one form and pin the RAGAS version in the lock file.

The same triad in DeepEval

DeepEval states that ContextualRelevancyMetric does not need expected_output: it grades the retrieval context against the input alone. FaithfulnessMetric compares the actual output with the retrieval context, and AnswerRelevancyMetric compares the actual output with the input. All three therefore work on a trace with no reference.

tests/test_rag_triad.py — the RAG triad as thresholds (Python, DeepEval 4.2.7, not run)
from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric, ContextualRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase


def triad_metrics() -> list:
    return [
        ContextualRelevancyMetric(threshold=0.7, include_reason=True),
        FaithfulnessMetric(threshold=0.8, include_reason=True),
        AnswerRelevancyMetric(threshold=0.7, include_reason=True),
    ]


def score_interactions(interactions: list[dict]):
    cases = [
        LLMTestCase(
            input=i["question"],
            actual_output=i["answer"],
            retrieval_context=i["passages"],
        )
        for i in interactions
    ]
    return evaluate(test_cases=cases, metrics=triad_metrics())

Source: DeepEval — Contextual Relevancy and DeepEval — Faithfulness. The thresholds are placeholders: derive real ones from the score distribution of a known-good build, as described on Evaluation in CI. include_reason=True makes the judge write why a score was low, which is what you read when triaging a failure.

Attributing a failure to a stage

With the triad scores in hand, a short decision table turns a failing trace into an owner:

Context relevancy Faithfulness Likely cause and first action

low

any

Retrieval fault. Inspect the query rewrite, the chunking and the reranker; check whether the page exists in the index at all (see Ingestion Pipelines and Freshness).

high

low

Generation fault. The right evidence was present and the model did not stay inside it; tighten the prompt, lower the temperature, or add the grounding checks from Hallucination Detection and Correction.

high

high, but answer relevancy low

The model answered a different question; check the query handling and the conversation context (see Conversational RAG and Memory).

high

high, answer relevancy high, human says wrong

The corpus is wrong or outdated, or the judge is miscalibrated. Check the source page before touching the pipeline.

The last row is the one teams skip: reference-free metrics measure consistency between question, evidence and answer, not truth. Keep a small reference-based golden set (see Building Eval Sets) alongside the reference-free scores so that "consistently wrong" remains detectable.

Agents: three levels of evaluation

An agent adds a loop of decisions on top of a pipeline. LangSmith’s agent-evaluation tutorial names three complementary levels, which are a useful common vocabulary:

Level What it scores Strength and weakness

Final response

Whether the last message is correct and complete, judged against a reference or a rubric.

Maximum signal about what the user sees; cannot say where a multi-step run went wrong.

Single step

One component in isolation: the first model decision given a fixed state, a classifier, a tool-selection prompt.

Cheap, deterministic enough to debug and iterate on; misses interaction effects between steps.

Trajectory

The sequence of tool calls (and optionally of graph nodes) against a reference route or a rubric.

Gives partial credit and catches right-answer-wrong-route runs; needs a reference route and can penalise valid alternatives.

Source: LangSmith — Evaluate a complex agent. Use all three: the final-response score is the product metric, single-step tests are the fast development loop, and trajectory scores explain regressions. The DocsAssistant agent (the one from AI Agents) has three tools, docs_search, fetch_page and github_open_issue, and the cases below use them.

Final response

The final answer of a RAG-backed agent can be scored exactly as in the previous section, using the retrieved passages the agent saw across all its search calls as the context. Concatenate the tool outputs of the run, deduplicate them, and pass them as retrieval_context. For answers that must match a reference, use a rubric judge from LLM-as-a-Judge.

Single step

A single-step test freezes the state and checks one decision. It is an ordinary pytest test, needs no judge, and runs in seconds:

tests/test_first_step.py — the first decision as a unit test (Python, pytest, not run)
import pytest
from langchain_core.messages import HumanMessage

from docsassistant.agent import build_model_with_tools   # returns a chat model bound to the agent's tools

CASES = [
    ("What does ef_search control in HNSW?", "docs_search"),
    ("The memory page is wrong, please report it.", "github_open_issue"),
]


@pytest.mark.parametrize("question,expected_tool", CASES)
def test_first_tool_choice(question, expected_tool):
    reply = build_model_with_tools().invoke([HumanMessage(question)])
    names = [call["name"] for call in reply.tool_calls]
    assert expected_tool in names


def test_no_tool_for_small_talk():
    reply = build_model_with_tools().invoke([HumanMessage("Thanks, that was helpful.")])
    assert reply.tool_calls == []

Source: LangChain — Models (tool calling on chat models). A model is not deterministic even at temperature 0 on every provider, so run each case several times and assert on a rate, as in the reliability section below.

Trajectory

Trajectory scoring compares the route with a reference. The match modes (strict, unordered, subset, superset) and a worked superset example are on Agent Evaluation; here is the other half, an LLM judge for trajectories that have no single correct route, and a per-tool argument override for the match evaluator. The agentevals README documents both calls.

tests/test_trajectory.py — match with argument overrides, and a judge (Python, agentevals 0.0.9, not run)
from agentevals.trajectory.llm import TRAJECTORY_ACCURACY_PROMPT, create_trajectory_llm_as_judge
from agentevals.trajectory.match import create_trajectory_match_evaluator

# Match the tools in order, but compare the search query case-insensitively.
match = create_trajectory_match_evaluator(
    trajectory_match_mode="strict",
    tool_args_match_mode="exact",
    tool_args_match_overrides={
        "docs_search": lambda got, want: got["query"].lower() == want["query"].lower(),
    },
)

# For open-ended runs, a judge grades the route against a generic accuracy rubric.
judge = create_trajectory_llm_as_judge(
    prompt=TRAJECTORY_ACCURACY_PROMPT,
    model="openai:o3-mini",
)


def grade(outputs: list[dict], reference: list[dict] | None = None) -> dict:
    result = {"judge": judge(outputs=outputs)}
    if reference is not None:
        result["match"] = match(outputs=outputs, reference_outputs=reference)
    return result

Source: agentevals. outputs is the message list of the run in the OpenAI message format (role, content, tool_calls); LangChain message objects are also accepted according to the README’s earlier examples, but only the dictionary form is shown above because it is the one the README documents for these two functions. The model string openai:o3-mini is the README’s example; substitute your own judge using the same provider:model convention. For LangGraph agents the package also has graph-trajectory evaluators (create_graph_trajectory_llm_as_judge, graph_trajectory_strict_match) that compare node steps rather than tool calls.

Tool-call accuracy

The set-based tool metrics (recall, precision, forbidden-call rate) are defined on Agent Evaluation. They ignore arguments and order, which is often what matters: a call to the right tool with the wrong page path is a failure. RAGAS and DeepEval both define a more complete score. In RAGAS the agent metrics are Tool Call Accuracy, Tool Call F1, Agent Goal Accuracy and Topic Adherence; the F1 variant combines precision and recall of the calls:

where a call matches when the tool name and the (possibly rule-compared) arguments agree. DeepEval’s ToolCorrectnessMetric compares tools_called with expected_tools and, according to its documentation, can additionally require matching input parameters or outputs (evaluation_params), a specific order (should_consider_ordering), or an exact list (should_exact_match).

tests/test_tool_correctness.py — tool calls against expectations (Python, DeepEval 4.2.7, not run)
from deepeval import evaluate
from deepeval.metrics import ToolCorrectnessMetric
from deepeval.test_case import LLMTestCase, ToolCall

case = LLMTestCase(
    input="The memory page is wrong, please report it.",
    actual_output="I opened an issue for the memory page.",
    tools_called=[ToolCall(name="docs_search"), ToolCall(name="github_open_issue")],
    expected_tools=[ToolCall(name="docs_search"), ToolCall(name="github_open_issue")],
)

metric = ToolCorrectnessMetric(threshold=0.9, should_consider_ordering=True)
evaluate(test_cases=[case], metrics=[metric])

Source: DeepEval — Tool Correctness. Build tools_called from the run’s messages (the tool_calls of each assistant message). Tool descriptions are prompts, so the same suite is the regression test for any edit to a tool’s name, description or schema; see Tool Design. For a safety-relevant tool such as github_open_issue, additionally assert that it is never called when the question did not ask for it; that is a hard zero-tolerance check, not a score.

Task completion without a reference

For goal-level success there is often no reference route. DeepEval’s TaskCompletionMetric is described as referenceless: it needs neither expected_output nor tools_called, infers the task and the outcome from the agent’s trace, and therefore requires the agent function to be traced with DeepEval’s @observe decorator and the metric to be attached to the evaluation loop. A standalone LLMTestCase does not work with it.

eval/task_completion.py — a traced agent scored by task completion (Python, DeepEval 4.2.7, not run)
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric
from deepeval.tracing import observe

from docsassistant.agent import run_agent   # your agent entry point


@observe()
def docs_assistant(question: str) -> str:
    return run_agent(question)


dataset = EvaluationDataset(goldens=[
    Golden(input="Which page explains HNSW's ef_search parameter?"),
    Golden(input="The memory page is wrong, please report it."),
])

metric = TaskCompletionMetric(threshold=0.7)

for golden in dataset.evals_iterator(metrics=[metric]):
    docs_assistant(golden.input)

Source: DeepEval — Task Completion. The documentation names @observe() and evals_iterator() as the integration points; the exact keyword for passing metrics to the iterator can differ between DeepEval releases, so confirm it against the page for your pinned version. A referenceless judge on a trace is a useful smoke signal on live traffic, but it inherits every judge bias described on the judge page, so keep the reference-based trajectory and tool checks for the CI gate.

Reliability: the same task, repeated

Agents are stochastic, and one passing run is weak evidence. Two numbers describe repeated runs of a task. With independent runs of which succeed, an unbiased estimate of the chance that at least one of attempts succeeds is the same pass@k estimator used for code (Chen et al.):

which is the right metric when a retry is cheap and a human or verifier picks the winner. For an agent that must be right every time a user asks, the relevant quantity is the opposite, the chance that all attempts succeed. With a per-run success probability estimated as and treating runs as independent:

(the exact unbiased estimator replaces the power with ). An agent that succeeds 90% of the time per run succeeds on five consecutive runs only about 59% of the time, which is why a single green run misleads. The distinction between "any of k" and "all of k" follows the reliability argument of the tau-bench paper; use pass@k for tasks with a verifier and pass^k for tasks that act on the user’s behalf.

eval/reliability.py — repeat each case and report both estimators (Python, standard library)
from math import comb


def pass_at_k(n: int, c: int, k: int) -> float:
    """Probability at least one of k draws (without replacement) from n runs is a success."""
    if n - c < k:
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)


def pass_pow_k(n: int, c: int, k: int) -> float:
    """Probability all k draws (without replacement) from n runs are successes."""
    return comb(c, k) / comb(n, k) if c >= k else 0.0


def reliability(results: dict[str, list[bool]], k: int = 3) -> dict[str, tuple[float, float]]:
    """results maps a case id to the pass/fail of each of its n repeated runs."""
    return {
        case_id: (pass_at_k(len(runs), sum(runs), k), pass_pow_k(len(runs), sum(runs), k))
        for case_id, runs in results.items()
    }

Source: Chen et al., Evaluating Large Language Models Trained on Code for the pass@k estimator; the combinatorial pass_pow_k is the direct counterpart written for this page and is not taken from a library. In CI, gate on the lower of the two for actions with side effects, and on repeated-run means for the rest (see Evaluation in CI for repeat runs and flakiness budgets).

Cost per task

An agent loop multiplies the token bill: every step re-sends the growing conversation. A task that needs model calls, where call has input and output tokens, costs

where and are the per-token prices of the model you read from your own contract or the provider’s pricing page (this page quotes none), and are the costs of non-model tool calls such as paid search APIs. Because input tokens accumulate across steps, grows faster than linearly with unless the context is trimmed or prompt caching applies; the full cost model, caching and routing are on Cost Management.

Evaluating cost means reporting it next to quality, per case, so that a change that improves accuracy by buying five extra steps is visible as such. Two derived numbers are worth a column each in the eval report: tokens per task, which is price-independent and survives a vendor price change, and cost per successful task, which divides total spend by the number of successes and so charges failures to the successes:

The model calls and their token counts are in the trace (see Observability for Agents and, for the attribute names, Observability with OpenTelemetry), so the eval harness can sum them without a second instrumentation path.

A cost and step budget as a JUnit test

Spring AI emits a token-usage meter from its chat model observations. Its documentation lists the base name gen_ai.client.token.usage (exported to Prometheus as gen_ai_client_token_usage_total), a counter with a gen_ai.token.type tag whose values are input, output and total. The test below runs the agent on one case with a SimpleMeterRegistry bound to the observation registry and fails if the task used more tokens than its budget. The wiring of the registry to the observations follows Spring Boot’s test support; the snippet shows the assertion, and the exact auto-configuration annotations were not verified against Spring AI 2.0.1.

DocsAssistantBudgetTest.java — a token budget per task (Java 21, JUnit 5, Micrometer, not run)
import static org.assertj.core.api.Assertions.assertThat;

import io.micrometer.core.instrument.Meter;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Statistic;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

@SpringBootTest
class DocsAssistantBudgetTest {

    private static final double MAX_TOTAL_TOKENS_PER_TASK = 20_000;

    @Autowired DocsAssistantAgent agent;      // the application's agent bean
    @Autowired MeterRegistry registry;        // the registry that Spring AI observations report to

    @Test
    void simpleLookupStaysInsideTheTokenBudget() {
        double before = totalTokens();

        agent.answer("Which page explains HNSW's ef_search parameter?");

        double used = totalTokens() - before;
        assertThat(used).isLessThanOrEqualTo(MAX_TOTAL_TOKENS_PER_TASK);
    }

    private double totalTokens() {
        return registry.find("gen_ai.client.token.usage")
                .tag("gen_ai.token.type", "total")
                .meters().stream()
                .mapToDouble(DocsAssistantBudgetTest::value)
                .sum();
    }

    private static double value(Meter meter) {
        for (var measurement : meter.measure()) {
            if (measurement.getStatistic() == Statistic.COUNT || measurement.getStatistic() == Statistic.TOTAL) {
                return measurement.getValue();
            }
        }
        return 0.0;
    }
}

Source: Spring AI — Observability for the meter name and tag, and JUnit User Guide. Set the budget from the 95th percentile of a known-good build plus a margin, not from a guess, and keep one such test per task class (lookup, multi-page synthesis, issue filing). The Spring AI evaluators RelevancyEvaluator and FactCheckingEvaluator for the quality side of the same test are on Spring AI Evaluation.

Putting the suite together for the DocsAssistant

A practical layout, from fastest and cheapest to slowest:

  1. Unit and single-step tests on every commit: tool selection, argument construction, refusal cases. No judge.

  2. Trajectory and tool-call checks on every pull request: reference routes for a small set of canonical tasks, the forbidden-call assertions, and step and token budgets.

  3. Reference-free RAG triad and task completion on the golden set on every pull request that touches a prompt, a model, the chunking or the index, with repeated runs for the stochastic cases (this is the gate built in Evaluation in CI).

  4. Sampled reference-free scoring of live traffic, feeding dashboards and the flywheel into the golden set (see Online Evaluation and Feedback).

  5. Periodic full runs against the pinned and the latest provider model to catch silent vendor changes (see Drift and Regression Monitoring).

Each layer reports quality, cost per task and latency in one table, so a change is accepted only when it does not trade one of them away silently. Security-focused testing of agents (prompt-injection and tool-abuse red teaming) belongs to AI Red Teaming in the AI Security & Responsible AI sub-section and is not covered here. The evaluation sections of the framework pages, such as LangChain Evaluation, show the same ideas inside one framework.

References