Prompt Augmentation & Generation

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Examples on this page target Python 3.12, the requests library, Ollama 0.34 with llama3.1:8b, langchain / langchain-core 1.4 / 1.6, and Spring Boot 4.1 with Spring AI 2.0.1. Retrieval only produces a ranked list of chunks; prompt augmentation is the step that turns that list into text the model can actually use — formatted, attributed and bounded by explicit instructions — before generation produces the answer. This is a deliberate act of context engineering, and the budget it has to work within is bounded by the model’s context window.

Formatting retrieved context with source attribution

An LLM has no innate notion of where a paragraph of context came from, so if the prompt does not say, the model cannot say either — and a RAG answer with no citations is much harder to verify against the documentation it was supposedly built from. The fix is mechanical: wrap every retrieved chunk in a small, consistent block that carries its provenance, using exactly the metadata DocsAssistant already stores on each chunk (source_path, title, section):

def format_context(hits: list[dict]) -> str:
    blocks = []
    for i, hit in enumerate(hits, start=1):
        blocks.append(
            f"SOURCE {i}: {hit['title']} § {hit['section']}\n"
            f"{hit['content']}"
        )
    return "\n\n".join(blocks)

Each hit is one of the dictionaries search_docs returns, carrying source_path, title, section and content. See Python: formatted string literals for the f-string syntax used to build each block. Run against the DocsAssistant sample corpus from Document Ingestion & Chunking, the resulting text looks like:

SOURCE 1: HNSW indexes § Tuning parameters
`M` bounds how many edges each node keeps per layer, trading memory for connectivity. ... `ef_search` is the
same idea applied at query time: a larger candidate list improves recall at the cost of latency...

SOURCE 2: pgvector basics § Operators and indexes
`<->` computes Euclidean distance, `<#>` negative inner product, and `<=>` cosine distance; a query orders
by whichever operator matches the embedding model's intended metric...

The system prompt then tells the model how to use those labels, both to answer and to cite:

Answer using only the numbered SOURCE blocks below. After every claim, cite the source it came from
in square brackets, e.g. "Increasing ef_search trades latency for recall [SOURCE 1]." If two sources
disagree, say so explicitly and cite both. Do not cite a source for a sentence it does not support.

Numbering the blocks rather than repeating each source_path inline keeps the citation markers short, while the mapping from SOURCE n back to title / section / source_path lets the application render a real link under each [SOURCE n] marker in the rendered answer, without asking the model to reproduce a path or URL verbatim (a place small models routinely get wrong).

Ordering contexts by relevance

Where the most relevant chunk sits in the prompt is not a neutral choice. Transformer attention over a long context is not uniform: models tend to weigh the beginning and the end of the context window more heavily than the middle, a primacy and recency effect together with the "lost in the middle" degradation that RAG Fundamentals discusses for the context window as a whole. Two orderings follow from this directly:

  • Most relevant first puts the top hit where primacy favours it, which matches how a reader skims a results page and is the simpler default — most RAG pipelines, and Spring AI’s default prompt template described below, place retrieved documents in retrieval order, most relevant first.

  • Most relevant last, closest to the question puts the top hit right before the user’s own question, which recency also favours, and can help when the source list is long enough that anything placed first risks falling into the middle of the window by the time the question is reached.

Neither ordering is universally better across models and context lengths, and the difference only matters once the context is long enough for the middle to genuinely degrade — for the DocsAssistant scenario’s five-chunk, few-hundred-token contexts against llama3.1:8b’s 8192-token `num_ctx, either ordering keeps every chunk close enough to an edge that the effect is unlikely to be visible; it becomes worth testing once context lengths approach the low thousands of tokens.

Handling "no relevant context" explicitly

Retrieval sometimes returns nothing above a usable similarity threshold, or returns chunks that do not actually answer the question — and the prompt has to say what to do then, or the model will happily hallucinate an answer instead. The system prompt states this as an explicit branch, and the application checks for it before formatting any context at all:

NO_CONTEXT_INSTRUCTION = (
    "If the SOURCE blocks below do not contain the answer to the question, "
    "reply exactly: \"I don't have enough information in the documentation to answer that.\" "
    "Do not guess, and do not answer from general knowledge."
)

def build_prompt(question: str, hits: list[dict]) -> str:
    if not hits:
        return f"{NO_CONTEXT_INSTRUCTION}\n\nQuestion: {question}"
    return (
        f"{NO_CONTEXT_INSTRUCTION}\n\n"
        f"{format_context(hits)}\n\n"
        f"Question: {question}"
    )

The string build_prompt returns is what the next section sends as the user message’s content; see Ollama API: Chat for that message format.

Stating the exact fallback sentence, rather than "say you don’t know," keeps the refusal recognisable and easy for the application to detect (for example to trigger a "search the web" or "open a support ticket" fallback) instead of parsing free-form hedging out of the model’s answer.

Popularity and score-bias considerations

Similarity search in a high-dimensional embedding space is subject to hubness: a small number of vectors end up unusually close to a disproportionate share of all queries, simply as a property of the geometry, not because they are more relevant to any one of those queries. A generic, broadly-worded chunk — an introduction paragraph, a glossary entry — can behave like a hub, showing up as a top-k hit for many unrelated questions and crowding out the chunk that actually answers each one. This is a different failure from ranking bias introduced deliberately (boosting a document because it was clicked often, or because it is newer), but the symptom looks the same in the final prompt: the same one or two sources keep appearing regardless of the question. Three mitigations, in increasing cost order:

  • raise the similarity-score floor a hit must clear before it is included at all, so a hub vector that is merely somewhat close to everything stops qualifying for queries it is not a strong match for;

  • re-rank the fused candidate list with a cross-encoder (see Hybrid Search & Re-ranking), which scores the query and each candidate jointly instead of by embedding proximity alone, and is far less prone to hubness than bi-encoder similarity;

  • if the application does track usage signals (click-through, thumbs up/down), treat them as a secondary re-ranking signal applied after a relevance floor, never as the primary retrieval score — a purely popularity-driven ranking optimises for what was clicked before, not for what answers the current question.

Domain-specific prompts

The same context-formatting and citation rules support very different tones. Two templates for the DocsAssistant scenario, both consuming format_context() from above, show the range.

Technical documentation assistant

You are a documentation assistant for an internal engineering knowledge base. Answer precisely and
concisely, using terminology exactly as it appears in the sources. Prefer short, direct sentences over
narrative explanation. Cite every factual claim as [SOURCE n]. If the sources give conflicting parameter
values or steps, present both and say which source each came from -- do not silently pick one.

{context}

Question: {question}
Answer:

Conversational assistant

You are a friendly assistant helping a colleague understand the documentation. Explain concepts in plain
language and feel free to paraphrase, but every paraphrase must still be traceable to a source -- add
[SOURCE n] after each claim exactly as in the technical template. If something in the sources is unclear or
incomplete, say so rather than filling the gap yourself.

{context}

Question: {question}
Answer:

The conversational template relaxes register, not evidence: both still require a citation per claim and both still fall back to the "not enough information" sentence above. {context} and {question} above are template placeholders the application substitutes before the request is sent; see PostgreSQL & pgvector for where hits comes from and Retrieval Strategies for how the top-k list is chosen and ordered before it reaches either template.

Confidence and faithfulness instructions

Citations alone do not stop a model from stating an inference as if it were a fact the source stated directly. Two further instructions, added to either template above, target that gap specifically:

Distinguish what the sources state directly from anything you infer. If you are inferring rather than
quoting a stated fact, say "based on the sources, it appears that..." instead of stating it plainly.
Never state a specific number, parameter name, or code identifier that does not appear verbatim in a
SOURCE block -- if you are unsure of the exact value, say the sources do not give one.

This is faithfulness: every claim traces to the retrieved context, not to the model’s parametric knowledge, which is the property RAG is meant to buy over an ungrounded chat model in the first place. It does not make the model’s citations correct by construction — a small local model can still cite the wrong SOURCE n for a true statement — which is why RAG Fundamentals and a later evaluation pass (golden questions against reference answers) remain necessary even once every answer carries citations.

Local LLMs with Ollama

DocsAssistant generates answers with Ollama’s llama3.1:8b at temperature 0.1 (favouring the most likely token over creative variation, which is what a grounded, citation-carrying answer wants) and num_ctx 8192 (the context window in tokens; a prompt plus context that exceeds it is silently truncated from the front, which is why the "lost in the middle" ordering choice above interacts with num_ctx at all). Both are set per request under options on Ollama’s /api/chat endpoint, not as server-wide defaults (the server-wide and Modelfile alternatives, and how to check the context Ollama actually allocated, are on Ollama — context length):

import json
import requests

def ask_docsassistant(question: str, hits: list[dict], stream: bool = False):
    prompt = build_prompt(question, hits)
    response = requests.post(
        "http://localhost:11434/api/chat",
        json={
            "model": "llama3.1:8b",
            "messages": [
                {"role": "system", "content": "You are a documentation assistant. " + NO_CONTEXT_INSTRUCTION},
                {"role": "user", "content": prompt},
            ],
            "options": {"temperature": 0.1, "num_ctx": 8192},
            "stream": stream,
        },
        stream=stream,
    )
    response.raise_for_status()

    if not stream:
        return response.json()["message"]["content"]

    answer = ""
    for line in response.iter_lines():
        if not line:
            continue
        chunk = json.loads(line)
        delta = chunk["message"]["content"]
        print(delta, end="", flush=True)
        answer += delta
        if chunk.get("done"):
            break
    return answer

/api/chat takes a messages list (roles system / user / assistant) rather than a single prompt string, which is what makes it the endpoint to use once DocsAssistant also needs conversation history (see Agent Memory & Semantic Cache) instead of a one-shot question. With stream: true, the response body is newline-delimited JSON objects, one per token (or small token batch); each carries a message.content delta and a final object with "done": true and timing/token-count fields. Streaming token-by-token is what lets a chat UI show the answer as it is generated instead of waiting for the full response, which matters more the larger the answer and the smaller (slower) the local model. See Ollama API: Chat for the full request/response schema, and Ollama: valid parameters and values for the other options fields (top_p, top_k, repeat_penalty, seed) not needed here.

In LangChain and Spring AI

LangChain

ChatPromptTemplate composes a RAG prompt from typed message parts instead of one hand-formatted string, keeping the system instructions, the formatted context and the user question as separate, independently reusable pieces:

from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_ollama import ChatOllama


def format_documents(docs: list[Document]) -> str:
    # adapt LangChain Documents to the dicts format_context expects
    return format_context([{**d.metadata, "content": d.page_content} for d in docs])


SYSTEM_INSTRUCTIONS = (
    "Answer using only the SOURCE blocks below. Cite every claim as [SOURCE n]. "
    + NO_CONTEXT_INSTRUCTION
)

prompt = ChatPromptTemplate.from_messages([
    ("system", SYSTEM_INSTRUCTIONS),
    ("human", "{context}\n\nQuestion: {question}"),
])

llm = ChatOllama(model="llama3.1:8b", temperature=0.1, num_ctx=8192)

rag_chain = (
    {"context": retriever | format_documents, "question": RunnablePassthrough()}
    | prompt
    | llm
)

answer = rag_chain.invoke("what does ef_search control in an HNSW index")

retriever is the PGVector-backed retriever from RAG Fundamentals. The {"context": retriever | format_documents, "question": RunnablePassthrough()} mapping runs the retriever, adapts the list[Document] it returns into the dictionaries the format_context helper above expects (each Document’s `metadata — title, section, source_path — plus its page_content as content), and passes the raw question through unchanged, before both are substituted into the template’s {context} / {question} placeholders — the same composition RAG Fundamentals's hand-rolled/LangChain/ Spring AI pipeline comparison builds on. See LangChain: ChatPromptTemplate reference, part of LangChain’s RAG chain composition.

Spring AI

Spring AI’s QuestionAnswerAdvisor and the newer ContextualQueryAugmenter (used inside a RetrievalAugmentationAdvisor pipeline) both wrap the same idea as ChatPromptTemplate above: a configurable prompt template with one placeholder for the user’s question and one for the retrieved context, which the advisor fills in before the request reaches the model. The two use different placeholder names, though: QuestionAnswerAdvisor expects {query} and {question_answer_context}, while ContextualQueryAugmenter expects {query} and {context} and rejects a template missing either one when it is built. QuestionAnswerAdvisor is the simpler, single-step form:

var questionAnswerTemplate = PromptTemplate.builder()
        .template("""
                Answer using only the context below. Cite the section for every claim.
                If the context does not contain the answer, say so instead of guessing.

                Context:
                {question_answer_context}

                Question:
                {query}
                """)
        .build();

Advisor questionAnswerAdvisor = QuestionAnswerAdvisor.builder(vectorStore)
        .searchRequest(SearchRequest.builder().similarityThreshold(0.50).topK(5).build())
        .promptTemplate(questionAnswerTemplate)
        .build();

See Spring AI: QuestionAnswerAdvisor for its builder options and default template.

ContextualQueryAugmenter plugs the same kind of template into the modular RetrievalAugmentationAdvisor pipeline from Retrieval Strategies, as the stage that turns retrieved Document`s into the final augmented prompt, and its `allowEmptyContext(…​) flag is exactly the "no relevant context" branch from earlier in this page, expressed as a builder option instead of prose in the template. It needs its own template, written against {context} and {query}:

var augmenterTemplate = PromptTemplate.builder()
        .template("""
                Answer using only the context below. Cite the section for every claim.
                If the context does not contain the answer, say so instead of guessing.

                Context:
                {context}

                Question:
                {query}
                """)
        .build();

Advisor retrievalAugmentationAdvisor = RetrievalAugmentationAdvisor.builder()
        .documentRetriever(VectorStoreDocumentRetriever.builder()
                .similarityThreshold(0.50)
                .vectorStore(vectorStore)
                .build())
        .queryAugmenter(ContextualQueryAugmenter.builder()
                .allowEmptyContext(false)
                .promptTemplate(augmenterTemplate)
                .build())
        .build();

With allowEmptyContext(false) (the default), when no document was retrieved the augmenter discards the user’s query and sends the model its empty-context prompt instead — by default an instruction to tell the user politely that the question is outside its knowledge base, replaceable through emptyContextPromptTemplate(…​) (for example with the exact fallback sentence from NO_CONTEXT_INSTRUCTION above). That is a stricter, advisor-level version of the NO_CONTEXT_INSTRUCTION branch: the model never sees the original question, so it cannot answer it from general knowledge. Setting it to true instead passes the original, un-augmented query through unchanged, useful when a question might be answerable from the model’s own instructions alone (a greeting, a meta-question about the assistant) rather than from retrieved documentation. See Spring AI: ContextualQueryAugmenter, part of the Spring AI RAG reference that documents both advisors and of Spring AI’s RAG advisors.

Related pages

References