Context Windows & Memory

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Examples on this page target Python 3.12 with the current anthropic Python SDK against model claude-sonnet-5 ($2 / MTok input, $10 / MTok output, $2.50 / MTok on a 5-minute cache write, $0.20 / MTok on a cache read), and describe OpenAI’s automatic prompt caching as documented for its current GPT-6.x models, using gpt-6-sol as the worked example ($2 / MTok input, $0.20 / MTok cached input). Prices and API shapes were checked against Anthropic’s pricing page, Anthropic’s prompt-caching guide and OpenAI’s prompt-caching guide as of 2026-09-28.

The context window as working memory

An LLM has no persistent memory of its own between calls — everything it can act on for a given response has to fit inside a single input: the context window. Every prior turn, every tool definition, every retrieved document and every instruction the model is meant to follow has to be re-supplied, in full, as text (or tokens) inside that window, up to the model’s maximum size. claude-sonnet-5 accepts a 1M-token context window; that is roughly 555,000 words on its current tokenizer, but a request rarely wants to fill the window — it wants to fit only what actually matters for the current turn.

This makes the context window the closest analogue an LLM has to a computer’s working memory (RAM): fast to use, entirely rebuilt on every call, and finite. Nothing survives from one API call to the next unless the caller puts it back in. What actually competes for space inside it — the system prompt, tool definitions, retrieved documents, conversation history, the current user turn and the tokens reserved for the model’s own output — is sketched below:

flowchart TB subgraph window["Context window (finite token budget)"] direction TB sys["System / developer prompt
(persona, rules, DocsAssistant instructions)"] tools["Tool / function definitions
(schemas the model may call)"] docs["Retrieved documents
(RAG chunks, file contents)"] hist["Conversation history
(prior user + assistant turns)"] turn["Current user turn"] budget["Reserved output budget
(max_tokens for the response)"] sys --> tools --> docs --> hist --> turn --> budget end

If the sum of system prompt, tools, retrieved documents, history and the current turn leaves too little room for budget, the call either fails outright or truncates whichever section the client didn’t protect — which is why context management, not just context size, is the actual engineering problem.

The KV cache

Generating a response token by token would be prohibitively slow if the model re-processed the entire prompt from scratch for every new token. Instead, a transformer’s self-attention layers cache the key and value vectors they compute for every token they have already processed, in what is called the KV cache. When the model needs to produce the next token, it only has to compute a query vector for that one new token and attend it against the keys and values already sitting in the cache — it never recomputes attention over tokens it has already seen in this generation.

The KV cache is what makes autoregressive generation practical at all: without it, producing an -token response over a -token prompt would recompute attention over the whole sequence at every step, costing roughly work, instead of with the cache (one prefill over the prompt, then the -th new token attends once to the positions cached so far). It is also, from the caller’s side, entirely invisible and per-request — it lives only inside the serving infrastructure for the duration of one generation (or, with server-side prompt caching below, a short window afterward) and is never something a client reads, writes or configures directly. It matters here because prompt caching, covered next, is essentially a way for a hosted API to let a repeated prefix reuse previously computed KV-cache state across separate requests, instead of paying to recompute it every time.

Long-context degradation

A larger context window does not mean the model uses every token in it equally well. Two well-documented failure modes show up as a prompt’s relevant content gets buried inside a large amount of irrelevant content:

  • Needle-in-a-haystack — a single fact ("the API key rotation interval is 90 days") is hidden somewhere inside a long document, and the model is asked a question that only that fact answers. Recall on this task degrades as the haystack grows and depends on where the needle sits, not only on how long the haystack is.

  • "Lost in the middle" — Liu et al. (2023) found that models are noticeably more accurate at using information placed at the very start or the very end of a long context than information buried in the middle, producing a U-shaped accuracy curve as a function of position.

Both failure modes are exactly why retrieval-augmented generation’s prompt-assembly step does not just concatenate every retrieved chunk and hope for the best: it retrieves a small, ranked, deliberately ordered set of passages instead of relying on a huge context window to "find" the right one on its own.

Prompt caching

Re-sending the same long system prompt, tool definitions or reference document on every turn of a conversation is wasted work: the tokens are identical to the previous request, yet a naive integration pays full input-token price and full processing latency for them every single time. Prompt caching lets a provider skip that repeated computation by reusing state computed on an earlier, identical prefix.

Anthropic: explicit cache_control breakpoints

Anthropic’s API requires the caller to mark where a reusable prefix ends, by attaching a cache_control block to a content block (most commonly the system prompt, but also usable on tool definitions or early messages). The marked type is "ephemeral", with a default 5-minute time-to-live or an optional 1-hour TTL. The following request caches a long DocsAssistant system prompt — reused unchanged for every question about this site’s own AsciiDoc pages, as introduced on the Vector Databases & RAG landing page — and prints the cache-related usage fields:

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

docs_assistant_system_prompt = """You are DocsAssistant, a help desk for this documentation site.
Answer only from the retrieved context the user supplies. If the context does not contain the
answer, say so plainly instead of guessing. Cite the source page for every claim you make.
... (the rest of a long, mostly-static system prompt: house style, tone, refusal rules,
few-shot examples of well-formed answers, and formatting instructions) ...
"""

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=300,
    system=[
        {
            "type": "text",
            "text": docs_assistant_system_prompt,
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": "How do I speed up similarity search?"}],
)

print(response.usage.cache_creation_input_tokens)  # tokens written to the cache on a cache miss
print(response.usage.cache_read_input_tokens)       # tokens served from the cache on a cache hit
print(response.usage.input_tokens)                  # tokens after the last cache breakpoint (never cached)

See Anthropic — Prompt caching.

OpenAI: automatic caching

OpenAI’s current GPT-6.x models cache eligible prompt prefixes automatically — there is no cache_control parameter to set for the basic behaviour. A request whose prompt reaches the model’s minimum eligible length (1,024 tokens, unchanged since GPT-5.6) is automatically checked against a recent, identical prefix, and any matching prefix is billed and served at the cached-input rate (0.1x the uncached input-token rate, e.g. $0.20 / MTok for gpt-6-sol’s $2 / MTok base input price). The response reports how many tokens were served from cache under `usage.input_tokens_details.cached_tokens in the Responses API, and under usage.prompt_tokens_details.cached_tokens in the Chat Completions API (the one used elsewhere in this section), so a caller can verify the cache is actually being hit rather than assuming it. See OpenAI — Prompt caching.

A dated cost example (2026-09-28, claude-sonnet-5 prices)

Take a DocsAssistant system prompt of 6,000 tokens, reused unchanged across a 10-turn conversation, where each turn also sends a short 50-token question and receives a 300-token answer. Without caching, every one of the 10 requests pays full input price for the whole system prompt; with caching, only the first request pays the (higher) cache-write price for it, and the remaining nine pay the much cheaper cache-read price:

Without caching (10 turns) With caching (10 turns)

System-prompt tokens billed

60,000 tokens @ $2 / MTok (base input)

Turn 1: 6,000 tokens @ $2.50 / MTok (5-minute cache write); turns 2-10: 54,000 tokens @ $0.20 / MTok (cache read)

System-prompt cost

$0.1200

$0.0150 + $0.0108 = $0.0258

Per-turn question + answer cost (unaffected by caching)

10 x (50 tok @ $2 / MTok + 300 tok @ $10 / MTok) = $0.0310

$0.0310

Total

$0.1510

$0.0568

Caching this one system prompt across a 10-turn conversation cuts the total cost by roughly 62% in this illustrative example — and the saving grows with every additional turn, since only the very first turn ever pays the cache-write premium. The same shape of saving applies to OpenAI’s automatic caching, at that provider’s own cached-input discount.

Truncation and compaction

Even a very large context window eventually fills up in a long-running conversation or agent loop, and a caller has to decide what to drop. Two common strategies:

  • Truncation — simply cut old content once a budget is exceeded, most often the oldest conversation turns (a sliding window), since the most recent turns are usually the most relevant to the next response. Truncation is cheap and fast, but it discards information outright: a fact established ten turns ago is gone once it falls outside the window, even if it is still relevant.

  • Compaction — instead of discarding old turns, summarise them into a shorter form (a running summary, a set of extracted facts) and keep that summary in place of the original turns. Compaction preserves more of the conversation’s substance per token than truncation does, at the cost of an extra summarisation step (and the risk that the summary itself drops or distorts something that mattered).

A minimal truncation helper — dropping the oldest turns until the conversation fits a token budget, using a tokenizer to measure it rather than guessing from character counts — looks like this:

import tiktoken

encoding = tiktoken.get_encoding("cl100k_base")


def count_tokens(text: str) -> int:
    return len(encoding.encode(text))


def trim_to_budget(turns: list[dict], max_tokens: int) -> list[dict]:
    """Drop the oldest turns until the remaining ones fit max_tokens."""
    trimmed = list(turns)
    total = sum(count_tokens(turn["content"]) for turn in trimmed)
    while total > max_tokens and len(trimmed) > 1:
        removed = trimmed.pop(0)
        total -= count_tokens(removed["content"])
    return trimmed


history = [
    {"role": "user", "content": "What is an HNSW index?"},
    {"role": "assistant", "content": "HNSW is a graph-based approximate nearest neighbour index..."},
    {"role": "user", "content": "How does pgvector expose it?"},
    {"role": "assistant", "content": "pgvector adds a vector column type and an hnsw index method..."},
    {"role": "user", "content": "How do I speed up similarity search?"},
]

budgeted_history = trim_to_budget(history, max_tokens=50)

See the tiktoken repository for the encoding this helper uses; the Tokens & Tokenization page covers tiktoken and its alternatives in depth.

Stateless APIs vs. conversation state

The Anthropic and OpenAI chat APIs are, by default, stateless: a call to create a message takes exactly the messages it is given and returns a response, with no memory of any earlier call. "Conversation memory" is therefore not a server-side feature the model provides for free — it is the caller’s own responsibility to resend the growing list of prior turns (subject to truncation or compaction, above) on every single request. What looks, from a chat UI, like the model "remembering" the last ten exchanges is really the client re-sending those ten exchanges as part of the input every time.

This distinction matters because it means conversation state and the context window are really the same budget viewed from two angles: every turn a client decides to keep as "memory" is a turn it must pay to re-send, and re-process, on every subsequent call.

Where longer-lived memory lives

Two kinds of memory that outlast a single context window are covered elsewhere on this site rather than here:

  • Persisting facts, preferences or task state across sessions — so an agent can recall something a user told it days earlier, well after that turn has been truncated out of any context window — is the concern of the planned AI Agents sub-section, under agent memory.

  • Retrieving relevant passages from a large external corpus, rather than trying to keep everything "in context" at once, is exactly what the Vector Databases & RAG reference covers in depth, from RAG fundamentals through agent memory and semantic caching.

Related pages

References