Agentic RAG

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page’s examples target LangChain / langchain-core / langgraph 1.4 / 1.6 / 1.2 with Ollama llama3.1:8b, and Spring Boot 4.1 with Spring AI 2.0.1, against the shared DocsAssistant search_docs tool. The MCP sections refer to the Model Context Protocol specification dated 2026-07-28.

From a fixed pipeline to a decision

Every page so far in this section retrieves unconditionally: a question always triggers exactly one vector search, and the retrieved chunks are always stuffed into the prompt before the model ever sees the question. Agentic RAG removes that certainty. The LLM is given retrieval as one of several tools it can call, and it decides, for each turn, whether to call it, how many times, with what query, and whether the result is good enough to answer from — the same tool-calling loop used for any other agent, but with search_docs (or a small set of retrieval tools) on the tool list instead of, or alongside, a calculator or a weather API. The framework-neutral design of such agents — the agent loop, tool design, memory, planning, multi-agent systems and human-in-the-loop approval — is covered in the AI Agents sub-section, with memory on its own Memory page.

Retriever-as-tool

The shift is entirely one of framing: the retriever function does not change, only how it is invoked. Instead of a Python function called unconditionally between "receive question" and "build prompt", search_docs becomes a tool with a name, a description and a typed signature that the model reads and decides to invoke:

def search_docs(query: str, component: str | None = None, k: int = 5) -> list[dict]:
    """Search the DocsAssistant documentation corpus for chunks relevant to `query`.

    Args:
        query: A natural-language question or search phrase.
        component: Optional filter, one of "vector-rag" or "spring-ai".
        k: Number of chunks to return.

    Returns:
        A list of dicts with source_path, title, section, content and score.
    """
    ...

The model reads the docstring (or an equivalent JSON schema) and the current conversation, and decides on its own whether the question needs a documentation lookup at all — "what does ef_search control" clearly does, "what’s 2 + 2" clearly does not. This single change is what lets everything below happen: the model can call the tool more than once, change the query between calls, or call a different tool entirely.

Multi-hop and iterative retrieval

A single retrieval call answers a question that one chunk already contains. A question like "how does the HNSW M parameter affect the memory formula used in the capacity example" may need one search for the HNSW parameter meaning and a second, differently worded search for the memory formula, because no single chunk covers both. An agent with search_docs as a tool can issue that second call itself, using the first result’s content to phrase a better second query — multi-hop retrieval — and can repeat the pattern iteratively until it judges the accumulated context sufficient or a step budget (Safety, below) is reached, rather than a fixed number of retrieval rounds decided in advance by the application.

Routing between tools

Once retrieval is a tool, it can sit next to other tools, and the model routes between them based on the question rather than the application hard-coding which source to use:

  • search_docs — semantic search over the DocsAssistant vector store, for conceptual questions.

  • A SQL tool — a parameterized query function over the same PostgreSQL database, for structured questions ("how many chunks does the vector-rag component have") that a vector search cannot answer.

  • A web-search tool — for questions the indexed corpus does not cover at all, such as a library’s very latest release notes.

The routing decision — which tool, or combination, answers this question — is made by the model from the tools' descriptions and the conversation, the same mechanism `search_docs’s own docstring uses above; no separate router component is required, though Retrieval Strategies describes an explicit query router as an alternative when the choice needs to be deterministic rather than model-driven.

With create_agent (introduced in full below), routing is nothing more than a second tool on the list. count_chunks runs a fixed, parameterized query against the shared documents / chunks schema, so the model only ever supplies a value, never SQL text:

import psycopg
from langchain.agents import create_agent
from langchain.tools import tool
from langchain_ollama import ChatOllama

from docsassistant.search import search_docs

DB_URI = "postgresql://docs_readonly:docs@localhost:5432/docsassistant"


@tool
def count_chunks(component: str) -> int:
    """Count the indexed chunks of one documentation component ("vector-rag" or "spring-ai").

    Use this for questions about how much documentation exists, not for questions about its content.
    """
    with psycopg.connect(DB_URI) as conn:
        return conn.execute(
            "SELECT count(*) FROM chunks c JOIN documents d ON d.id = c.document_id WHERE d.component = %s",
            (component,),
        ).fetchone()[0]


agent = create_agent(
    model=ChatOllama(model="llama3.1:8b", temperature=0.1),
    tools=[search_docs, count_chunks],
    system_prompt=(
        "You are DocsAssistant. Use search_docs for questions about what the documentation says, "
        "and count_chunks for questions about how much documentation exists."
    ),
)

agent.invoke({"messages": [{"role": "user", "content": "How many chunks does the vector-rag component have?"}]})

The two docstrings are the whole router: the model picks count_chunks for the counting question and search_docs for a conceptual one. docs_readonly is a read-only database role, one of the tool-permissioning measures described under Safety. See LangChain: Tools for the @tool decorator and how docstrings become tool descriptions.

Self-correcting RAG: grade, rewrite, retry

A plain retriever-as-tool agent still has one weak point: it does not know whether what it retrieved was any good, and it can hand a weak or off-topic set of chunks to the answer step regardless. Self-correcting RAG (the pattern behind Self-RAG and Corrective RAG) adds an explicit grading step between retrieval and generation:

  1. Retrieve — call search_docs with the current query.

  2. Grade — have the model (or a small classifier) judge each retrieved chunk as relevant or irrelevant to the question, typically with a short structured yes/no prompt per chunk.

  3. Decide — if enough chunks graded relevant, proceed to generation with just those; otherwise:

  4. Rewrite — have the model rephrase the query (more specific terms, a synonym, a decomposed sub-question) and go back to step 1, up to a bounded number of retries.

  5. Generate — answer using the retained, relevant chunks — or state plainly that the corpus has no relevant answer once retries are exhausted, rather than answering from irrelevant context.

This trades one extra model call (the grading step) per retrieval for materially better resistance to a single bad query wording, without ever asking a human to intervene.

Answer verification against sources

A second, complementary check runs after generation instead of before it: ask the model (or a separate verifier call) whether the drafted answer’s claims are actually supported by the retrieved chunks it was given, rather than trusting that grounding held. A verification prompt such as "for each claim in the answer below, does the following context support it — yes, no, or not addressed" catches the case where retrieval and grading both succeeded but generation still drifted or extrapolated beyond the sources, which grading alone cannot detect because grading only looks at retrieval, never at what the model did with it.

from pydantic import BaseModel
from langchain_ollama import ChatOllama

VERIFY_PROMPT = """You are checking an answer against its sources.
For each factual claim in the ANSWER, decide whether the CONTEXT supports it.
Reply with supported=true only if every claim is supported; otherwise list the unsupported claims.

CONTEXT:
{context}

ANSWER:
{answer}"""


class Verification(BaseModel):
    supported: bool
    unsupported_claims: list[str]


verifier = ChatOllama(model="llama3.1:8b", temperature=0).with_structured_output(Verification)


def verify_answer(answer: str, chunks: list[dict]) -> Verification:
    context = "\n\n".join(f"[{c['source_path']}] {c['content']}" for c in chunks)
    return verifier.invoke(VERIFY_PROMPT.format(context=context, answer=answer))

When supported comes back false, the application (or a LangGraph verify node) can regenerate with an instruction to drop the listed claims, retrieve again, or return the answer with an explicit caveat, rather than silently serving ungrounded text. See LangChain: Structured output for with_structured_output.

The DocsAssistant agent, three ways

LangChain 1.x create_agent

LangChain 1.x’s create_agent builds a tool-calling agent from a chat model and a list of tools in one call, letting the model decide when to invoke search_docs:

from langchain.agents import create_agent
from langchain_ollama import ChatOllama

from docsassistant.search import search_docs

model = ChatOllama(model="llama3.1:8b", temperature=0.1, num_ctx=8192)

agent = create_agent(
    model=model,
    tools=[search_docs],
    system_prompt=(
        "You are DocsAssistant. Use the search_docs tool to look up the "
        "documentation before answering questions about vector databases, "
        "RAG or Spring AI. Cite the source_path of every chunk you rely on."
    ),
)

result = agent.invoke(
    {"messages": [{"role": "user", "content": "What does ef_search control in an HNSW index?"}]}
)
print(result["messages"][-1].content)

create_agent runs the tool-calling loop internally: the model decides whether to call search_docs, the tool result is appended to the conversation, and the model is invoked again until it produces a final answer with no further tool calls. See LangChain: Agents for `create_agent’s full configuration surface (structured output, hooks, interrupt points).

A LangGraph agentic-RAG graph

`create_agent’s built-in loop is enough for a simple retriever-as-tool agent, but the grade/rewrite/retry pattern above needs explicit branching that a graph expresses more directly than a single agent loop. LangGraph models the self-correcting flow as a small state graph with one node per step:

  • retrieve — calls search_docs with the current query, stores the chunks in the graph state.

  • grade — scores each chunk’s relevance to the question; the state records how many passed.

  • A conditional edge out of grade: if enough chunks are relevant, go to generate; otherwise, go to rewrite (bounded by a retry counter also held in the graph state, to guarantee termination).

  • rewrite — rephrases the query from the original question and the previous attempt, then loops back to retrieve.

  • generate — produces the final answer from the retained chunks, and can itself route to an answer-verification node before returning, following the pattern above.

from langgraph.graph import StateGraph, START, END
from typing import TypedDict


class RagState(TypedDict):
    question: str
    query: str
    chunks: list[dict]
    graded_chunks: list[dict]
    retries: int
    answer: str


graph = StateGraph(RagState)
graph.add_node("retrieve", retrieve_node)
graph.add_node("grade", grade_node)
graph.add_node("rewrite", rewrite_node)
graph.add_node("generate", generate_node)

graph.add_edge(START, "retrieve")
graph.add_edge("retrieve", "grade")
graph.add_conditional_edges(
    "grade",
    lambda state: "generate" if len(state["graded_chunks"]) >= 2 or state["retries"] >= 3 else "rewrite",
    {"generate": "generate", "rewrite": "rewrite"},
)
graph.add_edge("rewrite", "retrieve")
graph.add_edge("generate", END)

app = graph.compile()

retrieve_node, grade_node, rewrite_node and generate_node are ordinary Python functions that take and return a RagState; retrieve_node wraps search_docs, and grade_node and rewrite_node each make one model call with a small structured prompt. See LangChain: Agentic RAG (LangGraph) for a full worked graph of this exact retrieve/grade/rewrite/generate shape, including the conditional-edge wiring.

Spring AI ChatClient with a documentation tool

Spring AI exposes the same retriever-as-tool pattern by registering a plain Java object whose method is annotated @Tool; ChatClient handles the tool-calling loop the same way create_agent does on the Python side:

package com.example.docsassistant;

import org.springframework.ai.tool.annotation.Tool;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.ai.document.Document;

import java.util.List;

public class DocsTools {

    private final VectorStore vectorStore;

    public DocsTools(VectorStore vectorStore) {
        this.vectorStore = vectorStore;
    }

    @Tool(description = "Search the DocsAssistant documentation for chunks relevant to a query")
    public List<DocHit> searchDocs(String query, String component) {
        SearchRequest.Builder request = SearchRequest.builder().query(query).topK(5);
        if (component != null) {
            request.filterExpression("component == '" + component + "'");
        }
        List<Document> hits = vectorStore.similaritySearch(request.build());
        return hits.stream()
                .map(d -> new DocHit(
                        (String) d.getMetadata().get("source_path"),
                        (String) d.getMetadata().get("section"),
                        d.getText(),
                        d.getScore() != null ? d.getScore() : 0.0))
                .toList();
    }
}

record DocHit(String sourcePath, String section, String content, double score) {}
package com.example.docsassistant;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.vectorstore.VectorStore;

public class DocsAssistantAgent {

    private final ChatClient chatClient;

    public DocsAssistantAgent(ChatClient.Builder builder, VectorStore vectorStore) {
        this.chatClient = builder
                .defaultTools(new DocsTools(vectorStore))
                .build();
    }

    public String ask(String question) {
        return chatClient.prompt()
                .user(question)
                .call()
                .content();
    }
}

ChatClient sends the @Tool-annotated method’s schema to the model alongside the conversation; when the model requests searchDocs, Spring AI invokes the Java method, feeds the returned DocHit list back as a tool result, and re-invokes the model, iterating until a final answer with no further tool calls is produced — the same loop shape as create_agent and the LangGraph graph above. See Spring AI: Tool calling for the @Tool annotation, ToolCallback alternatives, and defaultTools/tools(…​) registration.

GraphRAG and exposing retrieval as an MCP tool

GraphRAG

Every agent above treats search_docs as a single vector lookup. GraphRAG extends that lookup with a graph traversal step: after the vector search finds the most similar chunks or entities, a graph walk pulls in the related entities, cross-references, or provenance those hits connect to, before either is handed to the LLM. This site’s Neo4j section already documents that pattern in depth, including its own worked pipeline and diagram; see Neo4j: GraphRAG — vector retrieval plus graph expansion rather than repeating it here. Microsoft’s GraphRAG project takes the idea further at indexing time, building an LLM-extracted knowledge graph and community summaries from a corpus so that retrieval can answer both specific, entity-level questions and broad, corpus-wide "what are the themes in this dataset" questions that a flat vector index cannot answer well on its own; see Microsoft: GraphRAG for its indexing pipeline and query modes.

Exposing vector search as an MCP server tool

The retriever-as-tool pattern above wires search_docs directly into one agent’s process. The Model Context Protocol (MCP) standardizes the same idea across processes: a server exposes a tool — with a name, a description and a JSON Schema input/output shape, the same information search_docs’s docstring already carries — over a client-server protocol, so any MCP-compatible agent (not just the one that embedded `search_docs in its own code) can call it. A minimal tool definition for search_docs, following the MCP specification dated 2026-07-28, looks like:

{
  "name": "search_docs",
  "description": "Search the DocsAssistant documentation corpus for chunks relevant to a query.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "query": {"type": "string", "description": "Natural-language question or search phrase."},
      "component": {"type": ["string", "null"], "enum": ["vector-rag", "spring-ai", null]},
      "k": {"type": "integer", "default": 5}
    },
    "required": ["query"]
  }
}

This is the tool’s advertised shape only; the server behind it still runs the same search_docs function (or DocsTools.searchDocs) against the pgvector store, and a full server also needs transport wiring, a tools/call handler and lifecycle negotiation, which is out of scope here. See Model Context Protocol for the full specification and message flow, and Spring AI: MCP overview for exposing an @Tool-annotated Spring bean such as DocsTools as an MCP server tool, and for consuming one as a ToolCallback on the client side. The full treatment — a complete docs-assistant MCP server with search and retrieve tools, access control from the caller’s identity and wiring to pgvector and to a Spring AI VectorStore — is Exposing RAG through MCP.

Safety

Letting an LLM decide when and what to retrieve, and then act on what comes back, introduces failure modes a fixed pipeline does not have. A retrieved chunk is untrusted input the moment it reaches the prompt: if the indexed corpus can contain attacker-influenced text — a crafted support ticket, a wiki page anyone can edit, a web page pulled in by a search tool — that text can carry instructions aimed at the model itself, a prompt injection through retrieved documents, and an agent that treats every message in its context as equally authoritative can be talked into ignoring its system prompt, calling tools it should not, or leaking the conversation. Treating retrieved content as data to answer from rather than instructions to follow, stripping or flagging suspicious imperative text before it reaches the prompt, and keeping the system prompt’s own instructions clearly separated from retrieved context, all reduce this risk without eliminating it. Tool permissioning narrows the blast radius further: search_docs is read-only by construction, but an agent whose tool list also includes a SQL tool or a web tool should scope each one as tightly as the task allows — a read-only database role for the SQL tool, a query-only web tool with no ability to submit forms or follow authenticated links — so that even a successfully injected instruction has little to act on. Finally, the same iterative loop that lets an agent retry a bad query (the self-correcting pattern above) can also spin indefinitely on a query it can never satisfy, so every agentic RAG loop needs an explicit step limit (a maximum number of retrieve/grade/rewrite cycles, as the LangGraph retry counter above enforces) and, for a longer-running agent, an overall wall-clock or token budget, so that a stuck loop fails loudly and cheaply instead of retrying forever.

stateDiagram-v2 [*] --> Retrieve Retrieve --> Grade Grade --> Generate: enough relevant chunks Grade --> Rewrite: too few relevant chunks and retries remain Grade --> Generate: retries exhausted (answer with caveat) Rewrite --> Retrieve Generate --> Answer Answer --> [*]

In LangChain and Spring AI

LangChain

create_agent with search_docs on its tools list is LangChain’s minimal retriever-as-tool agent, shown above; the self-correcting retrieve/grade/rewrite/generate loop is expressed as a LangGraph graph instead, also shown above. See database/vector-rag/integrating-with-langchain.adoc#langchain-agentic-rag for the wider agentic-RAG surface, including how the retriever tool composes with LangChain’s other retrievers.

Spring AI

A ChatClient configured with .defaultTools(new DocsTools(vectorStore)), shown above, is Spring AI’s equivalent: the @Tool-annotated searchDocs method is exposed to the model the same way search_docs is exposed to create_agent. See database/vector-rag/integrating-with-spring-ai.adoc#spring-ai-tools for the wider tool-calling surface, including registering multiple tools and function-based `ToolCallback`s.

Related pages

References