Context Engineering for Agents

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page’s examples target Python 3.12 with LangChain 1.4.2 / langchain-core 1.6.5 / LangGraph 1.2.12 and langchain-ollama 1.1.0, Java 21 with LangChain4j 1.20.2 (langchain4j-agentic 1.20.2-beta30), and Spring Boot 4.1 with Spring AI 2.0.1 (plus the community spring-ai-agent-utils 0.12.0), using DocsAssistant's Ollama llama3.1:8b chat model run with an 8,192-token context (num_ctx=8192), as checked on 2026-09-28 (see the AI Agents baseline).

Context Engineering (in LLM Foundations) covers the discipline for a single call: what goes into one prompt, in what order, and how retrieval, examples and instructions compete for the window. This page covers the problem that only agents have: the context grows with every step of the loop, because each tool call and each tool result is appended to it. An agent that works for twenty steps is carrying twenty steps of history into step twenty-one. Anthropic’s Effective context engineering for AI agents frames the goal as finding the smallest set of high-signal tokens that lets the model do the next step well — and treating context as a finite resource with diminishing returns, since models get measurably worse at using information as the window fills (see Context Windows & Memory).

The context budget per step

At every step, the model call has to fit

where is the context window, the system prompt, the tool definitions, the accumulated history after steps, the new tool results, and the tokens reserved for the reply. , and are roughly fixed; grows by roughly one assistant turn and one tool result per step:

For DocsAssistant with , , (three tools), and a concise search result of tokens plus , the budget for history is about tokens — roughly steps before the window is full. With response_format="detailed" results of about 1,500 tokens, it is four. Every technique below either shrinks , shrinks , or moves work into a different window altogether.

flowchart LR subgraph W[One model call: context window W] S[System prompt S] --- T[Tool definitions T] --- H[History H_n] --- R[New tool results R_n] --- O[Reserved output O] end H -->|grows every step| X{Over budget?} X -->|yes| C[Clear old tool results, then summarise] X -->|no| N[Next step]

Shrinking tool results

The cheapest token is the one never added. Most of an agent’s context is tool output, so tool design is context engineering:

  • return concise results by default and detail on request (the response_format argument on Tool Design);

  • paginate and truncate, saying so in the result;

  • return identifiers and summaries rather than whole documents, and let the agent fetch the detail it needs (the just-in-time section below).

Summarisation and compaction

When history approaches the budget, compact it: replace the oldest turns with a model-written summary and keep the most recent turns verbatim. LangChain 1.x ships this as SummarizationMiddleware, and adds a cheaper first line of defence, ContextEditingMiddleware with ClearToolUsesEdit, which replaces old tool results with a placeholder (tool outputs are usually the bulk of the history and the least needed once acted on) before anything has to be summarised.

DocsAssistant with context editing, compaction, a scratchpad and a sub-agent
from langchain.agents import create_agent
from langchain.agents.middleware import (
    ClearToolUsesEdit,
    ContextEditingMiddleware,
    SummarizationMiddleware,
    TodoListMiddleware,
)
from langchain.tools import tool
from langchain_ollama import ChatOllama

llm = ChatOllama(model="llama3.1:8b", temperature=0, num_ctx=8192)

# A sub-agent with its own, isolated context window: it may read dozens of chunks,
# but only its final, condensed answer ever reaches the main agent.
researcher = create_agent(
    llm,
    tools=[docs_search, docs_get_section],
    system_prompt="Research the question in the docs. Reply with at most 5 bullet points, "
                  "each ending with its source page path. Never paste raw chunks.",
)


@tool
def research(question: str) -> str:
    """Delegate a multi-page documentation question to a research sub-agent.

    Returns a short, cited summary -- not the raw search results.
    """
    result = researcher.invoke({"messages": [("user", question)]})
    return result["messages"][-1].content


docs_agent = create_agent(
    llm,
    tools=[docs_search, docs_get_section, research, github_open_issue],
    middleware=[
        TodoListMiddleware(),                              # a structured scratchpad: write_todos
        ContextEditingMiddleware(edits=[
            ClearToolUsesEdit(trigger=5000, keep=3,        # clear old tool results past 5k tokens
                              exclude_tools=("github_open_issue",)),
        ]),
        SummarizationMiddleware(
            model=llm,
            trigger=("tokens", 6000),                      # compact at ~75% of num_ctx
            keep=("messages", 12),                         # keep the most recent turns verbatim
        ),
    ],
)

docs_search and github_open_issue are the tools from Tool Design; docs_get_section is defined in the just-in-time section below. ClearToolUsesEdit keeps the three most recent tool results and never clears the issue-creation result, so the agent can always see what it already filed. SummarizationMiddleware’s `trigger accepts a token count, a message count or a fraction of the model’s window; keep says how much recent history survives verbatim. A cheaper model can do the summarising — the model argument is independent of the agent’s own model. See LangChain — Agents (middleware) and LangChain — Short-term memory.

Summaries lose information, and what they lose is hard to predict. Two habits limit the damage: compact on a semantic boundary when you can (after a sub-task finishes, not in the middle of one), and keep anything the agent must not forget — the user’s goal, decisions already made, actions already taken — in a structured scratchpad that compaction never touches.

In LangChain4j, the history window is the ChatMemory implementation: MessageWindowChatMemory keeps the last messages and TokenWindowChatMemory the last tokens. In the agentic module, an agent can also receive a model-written summary of what the other agents in the system did, instead of their raw transcripts:

LangChain4j: token-bounded memory and summarised cross-agent context
import dev.langchain4j.agentic.AgenticServices;
import dev.langchain4j.memory.chat.TokenWindowChatMemory;

IssueDrafter drafter = AgenticServices.agentBuilder(IssueDrafter.class)
        .chatModel(chatModel)
        .chatMemoryProvider(memoryId -> TokenWindowChatMemory.builder()
                .id(memoryId)
                .maxTokens(6000, tokenCountEstimator)       // evict the oldest messages past 6k tokens
                .build())
        .summarizedContext("researcher")                    // a summary of the researcher's work, not its transcript
        .outputKey("issueDraft")
        .build();

See LangChain4j — Chat memory and LangChain4j — Agents and Agentic AI (memory and context engineering). Spring AI’s equivalent of the window is MessageWindowChatMemory (default 20 messages) behind a MessageChatMemoryAdvisor, shown on Memory; Spring AI has no built-in summarising compaction, so a summarising step is an advisor you write or a periodic ChatClient call that replaces old messages in the ChatMemoryRepository. See Spring AI — Chat Memory.

Structured scratchpads and notes

A scratchpad is state the agent writes deliberately, in a structure it chose, so that the important facts survive compaction and are cheap to re-read. Three forms are common:

  • A todo list — the plan, with each item’s status. TodoListMiddleware above gives the agent a write_todos tool and keeps the list in the agent’s state; the community spring-ai-agent-utils library provides the same as TodoWriteTool for Spring AI.

  • Notes — short, dated facts the agent wants to keep ("the pgvector page and the HNSW page disagree on the default `m`"). They can live in the graph state for one run, or in a long-term store to persist across runs (see Memory).

  • Typed state — in LangGraph, simply extra keys in the graph state (findings, lessons, open_questions in the examples on Agent Loop & Architectures). They are not message history, so no summariser ever rewrites them.

A notes tool that writes into the agent’s own state
import operator
from typing import Annotated

from langchain.agents import AgentState
from langchain.tools import InjectedToolCallId, tool
from langchain_core.messages import ToolMessage
from langgraph.types import Command


class NotesState(AgentState):
    notes: Annotated[list[str], operator.add]              # each note is appended, never overwritten


@tool
def write_note(note: str, tool_call_id: Annotated[str, InjectedToolCallId]) -> Command:
    """Record one short fact you must not forget (a decision, a contradiction, an open question)."""
    return Command(update={
        "notes": [note],
        "messages": [ToolMessage("noted", tool_call_id=tool_call_id)],
    })

# create_agent(llm, tools=[..., write_note], state_schema=NotesState, ...)

A tool that returns a Command updates graph state directly — the operator.add reducer appends each note — and the model sees only the short "noted" result. The InjectedToolCallId annotation is filled in by the runtime and hidden from the model’s schema. See LangChain — Short-term memory (reading and writing state from tools).

Sub-agent isolation

The most effective compaction is never to put the tokens in the main context at all. A sub-agent runs in its own context window, does the token-heavy work (reading twenty chunks, trying three queries), and returns only a condensed result. The research tool above is exactly that: the main agent sees a five-bullet summary; the sub-agent’s window, with all its raw search results, is discarded when it returns. Isolation also improves quality — each sub-agent’s context is focused on one question — at the cost of more total tokens and of losing detail the summary did not carry.

This is the same principle Claude Code uses for subagents (see Subagents & Custom Agents) and the reason orchestrator-worker systems scale; the multi-agent side is on Multi-Agent Systems. In Spring AI, spring-ai-agent-utils’s `TaskTool provides the sub-agent-as-a-tool pattern directly: the main ChatClient gets a Task tool that launches a configured sub-agent with its own ChatClient and returns its result.

Spring AI: sub-agents via spring-ai-agent-utils
import org.springaicommunity.agent.tools.task.TaskTool;
import org.springaicommunity.agent.tools.task.claude.ClaudeSubagentReferences;
import org.springaicommunity.agent.tools.task.claude.ClaudeSubagentType;

ToolCallback task = TaskTool.builder()
        .subagentReferences(ClaudeSubagentReferences.fromRootDirectory("src/main/resources/agents"))
        .subagentTypes(ClaudeSubagentType.builder()
                .chatClientBuilder("default", chatClientBuilder.clone())
                .build())
        .build();

ChatClient docsAgent = chatClientBuilder
        .defaultToolCallbacks(task)
        .defaultTools(new DocsTools())
        .build();

Sub-agents are defined as Markdown files with front matter (name, description, tools) in the given directory, in the same format Claude Code uses. See spring-ai-agent-utils.

Just-in-time tool results

Pre-loading everything that might be relevant ("stuff the top 20 chunks into the prompt") fills the window with material the agent never uses. The just-in-time alternative keeps lightweight references in context — page paths, section titles, IDs — and gives the agent tools to load the detail when it decides it needs it, exactly as a person skims a table of contents before opening a chapter. It trades a few extra tool calls for a much smaller and more relevant context.

Loading one section on demand
from langchain.tools import tool

from docsassistant.pages import read_section, list_sections


@tool
def docs_get_section(page: str, section: str | None = None) -> dict:
    """Load one section of a documentation page, by the page path docs_search returned.

    Without `section`, returns only the page's section titles, so you can pick one.
    """
    if section is None:
        return {"page": page, "sections": list_sections(page)}
    text = read_section(page, section)
    if text is None:
        return {"error": f"no section '{section}' on {page}; call again without section to list them"}
    return {"page": page, "section": section, "content": text[:4000], "truncated": len(text) > 4000}

docs_search returns concise hits; docs_get_section lists or loads exactly one section. Between them the agent navigates the corpus instead of receiving it. See LangChain — Tools. The progressive tool disclosure on Tool Design applies the same idea to the tool definitions themselves.

A checklist

  • Measure the per-step budget; know how many steps fit before compaction.

  • Default every tool to concise, paginated output with human-readable IDs.

  • Clear old tool results first; summarise only when that is not enough.

  • Keep the goal, decisions and actions taken in a scratchpad that is never summarised.

  • Push token-heavy sub-tasks into sub-agents that return condensed results.

  • Load detail just in time, by reference.

In other frameworks

Framework Context management

OpenAI Agents SDK

Sessions hold history; OpenAIResponsesCompactionSession compacts it with the Responses API, and a handoff can filter what history the next agent receives.

Claude Agent SDK

Inherits Claude Code’s automatic compaction (a PreCompact hook fires before it); subagents declared with AgentDefinition run in isolated contexts.

Microsoft Agent Framework

Compaction strategies (for example ContextWindowCompactionStrategy) and history providers manage an agent session’s messages; context providers inject memory per run.

CrewAI

respect_context_window on an agent summarises when the window is exceeded; task outputs, not full transcripts, flow between tasks.

See OpenAI Agents SDK, Claude Agent SDK, Microsoft Agent Framework and CrewAI.

References