RAG Systems in Production

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This sub-section was written on 2026-09-30 against LangChain 1.4.3 (with langchain-core 1.6.6, langgraph 1.2.12 and langchain-classic 1.0.8) and FastAPI 0.142.2 on Python 3.12, Spring AI 2.0.1 on Spring Boot 4.1.1 with Java 21, and LangChain4j 1.20.2; the full list is in the version baseline below. Every version was read from PyPI and Maven Central on that date. Every import in the Python snippets was resolved against the released packages in a scratch virtual environment, and every Java snippet was compiled against the released jars in a scratch Maven project that imports spring-ai-bom 2.0.1 and langchain4j-bom 1.20.2. No live model, database or graph was called while the pages were checked, so the outputs shown are illustrative; each page says what was checked in its lead paragraph.

A retrieval-augmented generation (RAG) prototype takes an afternoon: split some Markdown, embed it, store the vectors, retrieve the five nearest chunks and paste them into a prompt. The Vector Databases & RAG section of this site covers exactly that path, and covers it well. Everything that happens after the prototype works is the subject of this sub-section: real PDFs with tables and scanned pages, documents that change every hour, users who must not see each other’s files, conversations that span days and devices, answers that must cite their sources and admit when they do not know, an HTTP API and an MCP server in front of the same retriever, a latency budget, a cost per question, a knowledge graph for the questions vectors cannot answer, and the evaluation and tracing that tell you whether last week’s change made things better or worse. This is the day-2 problem of RAG.

What Vector Databases & RAG covers and what this sub-section adds

The two sections describe one system. The pages here link the foundations instead of repeating them.

Topic Covered in Vector Databases & RAG Added here

Ingestion

Document Ingestion & Chunking: loaders, splitters, chunk sizes and overlap for clean text.

Document Parsing (the parser ladder, OCR, structure as metadata), Multimodal RAG (tables, images, audio) and Ingestion Pipelines & Freshness (idempotent jobs, CDC, deletes, PII redaction).

Retrieval

Retrieval Strategies (multi-query, HyDE, parent-document, MMR), Hybrid Search & Re-ranking, Metadata Filtering.

Query Understanding & Routing (routing across vector, SQL, graph and tools; text-to-SQL) and Advanced Indexing Patterns (hypothetical questions, propositions, RAPTOR, contextual retrieval, late chunking).

Conversation

Agent Memory & Semantic Cache: memory tables and a semantic cache.

Conversational RAG & Memory: persisting conversations in three stacks, multi-device sessions, retention and deletion.

Answer quality

Prompt Augmentation & Generation and Evaluating RAG Systems (offline metrics).

Citations & Grounding, Hallucination Detection & Correction and Evaluation at Scale (reference-free metrics, CI gates, online judging).

Security

The security checklist in Production Considerations.

Guardrails & Prompt Injection and Access Control & Privacy.

Exposure

The vector-store APIs of LangChain and Spring AI.

Serving RAG as an API (REST, SSE, WebSocket) and RAG Behind MCP.

Graphs

Neo4j — Vector Search and GenAI.

GraphRAG & Knowledge Graph Construction, Text2Cypher & Graph Retrievers and Graph-Powered Recommendations.

Operations

Scale path and capacity in Production Considerations.

Production RAG Architecture, Caching, Latency & Cost, Observability & Tracing and Deployment Patterns.

The running scenario: DocsAssistant as a production service

Every page extends the same service. DocsAssistant answers questions about this site’s documentation. It reuses the corpus, the pgvector schema (documents, chunks, chunk_embeddings) and the local models of PostgreSQL & pgvector, the agent of LangChain and Spring AI, and the docs-assistant MCP server of Exposing RAG through MCP. Here it grows into a service with users:

flowchart LR subgraph Clients W[Web widget] C[Claude Code / Copilot] A[LangGraph agent] end subgraph DocsAssistant API["/v1/ask REST + SSE"] MCP["MCP server search_docs"] O[Orchestrator] R[Retriever with ACL filter] M[(Conversation store)] end subgraph Stores PG[(PostgreSQL + pgvector)] N[(Neo4j graph)] end W --> API C --> MCP A --> MCP API --> O O --> R O --> M MCP --> R R --> PG R --> N

  • Ingestion adds real PDFs with tables and images next to the AsciiDoc sources, keeps the index fresh from change events and deletes what was deleted.

  • POST /v1/ask takes a question, a conversationId and optional filters, and streams the answer as server-sent events with citations.

  • Conversation IDs persist memory per authenticated user, across devices.

  • Per-component ACLs filter retrieval: a caller’s groups come from the verified token and become a metadata filter.

  • The same retriever is exposed through an MCP server, so coding assistants and agents can search the docs.

  • A Neo4j graph of components, pages and concepts answers the questions that need relationships (GraphRAG).

Each build page shows the same capability three times: Python (LangChain 1.x with FastAPI), Java (Spring AI 2.0 on Spring Boot 4.1) and Java (LangChain4j 1.20). Model names are never hard-coded in the snippets: every example reads the chat and embedding model from configuration (CHAT_MODEL, EMBEDDING_MODEL, or the spring.ai. and langchain4j. properties), so the same code runs against a local runtime from Running LLMs Locally or a hosted provider.

Reading path

Read the pages in navigation order on a first pass. Shorter paths:

From prototype to first production release

  1. Production RAG Architecture for the target shape and the latency budget.

  2. Document Parsing and Ingestion Pipelines & Freshness.

  3. Access Control & Privacy before anyone else can log in.

  4. Serving RAG as an API and Conversational RAG & Memory.

  5. Observability & Tracing and Deployment Patterns.

Improving answer quality

  1. Evaluation at Scale first, so every change is measured.

  2. Query Understanding & Routing and Advanced Indexing Patterns.

  3. Citations & Grounding and Hallucination Detection & Correction.

Hardening

  1. Guardrails & Prompt Injection.

  2. Access Control & Privacy.

  3. Caching, Latency & Cost.

Knowledge graphs

  1. Neo4j — Vector Search and GenAI for the basics.

  2. GraphRAG & Knowledge Graph Construction.

  3. Text2Cypher & Graph Retrievers and Graph-Powered Recommendations.

Exposing RAG to other tools

  1. Serving RAG as an API.

  2. RAG Behind MCP, with Exposing RAG through MCP for the protocol detail.

What’s covered

Architecture

  • Production RAG Architecture — the reference microservice architecture, a stateless orchestrator with fan-out and gather, independent scaling tiers, the latency budget, DIY versus RAG platforms, total cost of ownership and a POC-to-production plan.

Ingestion

  • Document Parsing — what is inside a PDF, the parser ladder from pypdf to a vision model, Docling and Unstructured, OCR, structure as metadata, and the Tika and PDF readers on the JVM.

  • Multimodal RAG — tables, images, audio and video, visual citations and multimodal hallucinations.

  • Ingestion Pipelines & Freshness — batch and streaming ingestion, idempotent jobs, content hashes, CDC with Debezium and Kafka, deletes, PII redaction and provenance.

Retrieval

  • Query Understanding & Routing — rewriting, condensation, self-query filters, decomposition, routing across vector, SQL, graph and tools, and safe text-to-SQL.

  • Advanced Indexing Patterns — hypothetical questions, propositions, auto-merging, sentence windows, multi-vector, RAPTOR, contextual retrieval and late chunking.

Conversation and answers

  • Conversational RAG & Memory — condensation, window, summary and long-term memory, persistence in three stacks, multi-device sessions, retention and memory poisoning.

  • Citations & Grounding — chunk IDs end to end, structured answers with citations, citation verification and the UX of sources.

  • Hallucination Detection & Correction — faithfulness, LLM-as-judge and HHEM, blocking and asynchronous checks, regenerate and abstain.

  • Guardrails & Prompt Injection — input and output guardrails, indirect injection through documents, delimiting context and tool allow-lists.

  • Access Control & Privacy — ACLs as filters derived from identity, tenancy models, redaction, encryption, audit and data residency.

Exposing RAG

  • Serving RAG as an API — the /v1/ask contract, OpenAPI, SSE and WebSocket streaming, async ingestion, idempotency keys, OAuth2, rate limits, versioning, CORS and a browser widget.

  • RAG Behind MCP — the same retriever as an MCP tool, resource and prompt, identity mapping, and consumption from Claude Code, Copilot and LangGraph.

  • Caching, Latency & Cost — cache tiers, semantic-cache thresholds, event-driven invalidation, prompt caching, model cascades and cost per request.

Knowledge graphs

Operations

  • Evaluation at Scale — synthetic test sets, reference-free metrics, CI gates, online judging, A/B tests and the feedback flywheel.

  • Observability & Tracing — spans, OpenTelemetry GenAI conventions, RAG metrics and tracing backends.

  • Deployment Patterns — containers, managed platforms, GPU inference, secrets, blue/green rollouts, loads and disaster recovery.

Cheat sheet

Version baseline

Every version below was read from PyPI, Maven Central and Docker Hub on 2026-09-30. Each page repeats the subset it uses in its lead paragraph; if a page and this table ever disagree after an update, this table wins.

Area Versions

LangChain (Python)

langchain 1.4.3, langchain-core 1.6.6, langgraph 1.2.12, langgraph-checkpoint-postgres 3.1.2, langchain-classic 1.0.8 (home of index(), SQLRecordManager, ParentDocumentRetriever, MultiVectorRetriever and SelfQueryRetriever), langchain-text-splitters 1.1.2, langchain-postgres 0.0.18, langchain-neo4j 0.10.0, langchain-mcp-adapters 0.3.2 and langchain-docling 3.0.0, on Python 3.12.

Web layer (Python)

FastAPI 0.142.2, whose fastapi.sse module provides EventSourceResponse and ServerSentEvent.

Parsing

Docling 2.131.0, Unstructured 0.27.10, pypdf 6.19.0 and PyMuPDF 1.28.2; Apache Tika through the Spring AI and LangChain4j readers.

Privacy

Presidio (presidio-analyzer, presidio-anonymizer) 2.2.364.

Neo4j and GraphRAG

Neo4j 2026.09.0 (calendar versioning), neo4j-graphrag 1.21.0 with the neo4j Python driver 6.3.1, the Neo4j Java driver 6.3.0, and Microsoft graphrag 3.2.0.

MCP

The 2026-07-28 revision of the specification and the Python mcp SDK 2.2.0 used by the Model Context Protocol sub-section; Spring AI 2.0.1 speaks the 2025-11-25 revision through the MCP Java SDK 2.0.x, as explained on MCP Server.

Evaluation

RAGAS 0.4.3, DeepEval 4.2.7 and Open RAG Eval (open-rag-eval) 0.3.0; Vectara’s HHEM classifier (vectara/hallucination_evaluation_model on the Hugging Face Hub).

Spring AI

2.0.1 through org.springframework.ai:spring-ai-bom:2.0.1 on Spring Boot 4.1.1 and Java 21. A 2.1 milestone exists; nothing here depends on it.

LangChain4j

1.20.2 through dev.langchain4j:langchain4j-bom:1.20.2 (the beta integration modules, such as langchain4j-pgvector and the Tika parser, are 1.20.2-beta30), and langchain4j-community-neo4j with langchain4j-community-neo4j-retriever 1.20.0-beta30 through langchain4j-community-bom.

Stores

PostgreSQL 17 with pgvector, as in PostgreSQL & pgvector; Redis for caches, as in Redis Use Cases and Patterns.

Where the books are out of date

The requester-provided books below are excellent on concepts, but several of their APIs have moved on. The pages flag each difference where it matters; the most important are collected here.

  • LangChain. Older examples import langchain.text_splitter and build agents with create_react_agent. In LangChain 1.x the splitters live in langchain_text_splitters, agents are built with langchain.agents.create_agent and extended with middleware, and several retrievers used in Learning LangChain (ParentDocumentRetriever, MultiVectorRetriever, SelfQueryRetriever, the indexing API) now live in langchain-classic. LlamaIndex moved its documentation to developers.llamaindex.ai (llama-index 0.14.25 on 2026-09-30).

  • The Neo4j book’s stack. It uses LangChain4j 0.35, a pre-1.0 Spring AI, an earlier Haystack line, an OpenAI embedding model of 2022 and Neo4j 5.21. The current versions are LangChain4j 1.20 (with the community Neo4j embedding store and text-to-Cypher retriever), Spring AI 2.0.1 (with Neo4jVectorStore and Neo4jChatMemoryRepository), haystack-ai 3.2.0 with neo4j-haystack 2.2.1, and the calendar-versioned Neo4j 2026.x releases; the embedding model it uses has been superseded. It predates neo4j-graphrag’s text2cypher retrievers, `SimpleKGPipeline, the LLM Graph Builder and the Neo4j MCP servers, which these pages document from Neo4j’s own documentation.

  • Spring AI. Books written for Spring AI 1.0 or earlier describe PromptChatMemoryAdvisor (gone in 2.0), property paths with an .options segment (dropped), and ChatModel.call() running tools (it no longer does). See Migrating from 1.x to 2.x.

  • The Python cookbook’s UI layer. Its only interface is Streamlit, on the end-of-life python:3.9 image, with the 0.x OpenAI Agents SDK. REST, FastAPI and server-sent events are not covered; Serving RAG as an API fills that gap from the FastAPI documentation.

  • Not covered by any of the books: SSE and WebSocket API design, authoring a RAG-behind-MCP server, and LangChain 1.x middleware for guardrails and memory.

Bibliography

The following sources were consulted while preparing the pages of this sub-section. Every code example was written from the current official documentation and checked against the released packages as described above — never copied from a book. Every documentation page cited on the pages is listed here once. The official documentation hosts could not be fetched from the environment the pages were checked in, so the version numbers come from the package registries and the API shapes from the released packages; the links are the documentation’s canonical addresses.

Requester-provided books

These books are consulted references for concepts only. Their text and code are not reproduced; concepts are paraphrased and credited on the page that uses them.

oreilly.com and packtpub.com answer automated fetches with an HTTP 403 response, but the publisher pages above are live pages when opened in a browser.

Official documentation

Parsing

LangChain and LangGraph

Spring AI

LangChain4j

Neo4j

Privacy, guardrails and hallucination detection

Evaluation

Observability

Web, messaging and deployment

Further documentation pages cited on the pages of this sub-section (links to sections within these pages are not repeated):

Specifications and standards

Papers and engineering articles

The requester-provided books are consulted references, not the primary source of any example on these pages. On any discrepancy between a book and the official documentation of a tool or framework, the official documentation is authoritative.

References