Short-Term Memory & Checkpointers
|
This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production. This section’s bibliography lists the reference material consulted while preparing these pages. |
This page was written against langchain 1.4.3, langchain-core 1.6.5, langgraph 1.2.12,
langgraph-checkpoint 4.2.0, langgraph-checkpoint-sqlite 3.1.1, langgraph-checkpoint-postgres 3.1.2 and
langchain-ollama 1.1.0 on Python 3.12, as verified on 2026-09-29. The running example is DocsAssistant, which
answers questions over this site’s documentation with Ollama’s llama3.1:8b chat model over the pgvector store
built in Integrating with LangChain. Ollama was not
available while the snippets were checked, so the conversations shown here were run against small scripted fake
chat models (a RecallModel that answers "What is my name?" from whatever history it is shown, and
GenericFakeChatModel for the summariser) while the checkpointers, the middleware and the graphs were the real
packages. The SQLite and Postgres examples wrote to a real SQLite file and to a real PostgreSQL 17 database
started in a container. No output comes from a live model.
An agent’s short-term memory is the state of one conversation: the message list and any other fields the agent carries from turn to turn. In LangChain 1.x that state lives in a LangGraph graph, and it survives between calls only when the graph is compiled with a checkpointer and every call names a thread. This page covers that mechanism, how to read and edit the saved state, how to keep the history inside the model’s context window, and how the same checkpoints make an agent resumable after a failure. The framework-neutral picture of memory (types, taxonomies, frameworks other than LangChain) is in Agent Memory; memory that outlives a thread and is shared across threads is in Long-Term Memory Stores.
Threads and checkpointers
A checkpointer saves a snapshot of the graph state (a checkpoint) at the end of every super-step, the unit
in which LangGraph runs all currently runnable nodes. A thread is the sequence of checkpoints that share one
thread_id; the checkpointer uses that ID as the key under which it stores and loads them. Compile the agent with
a checkpointer and pass the thread_id in the run’s configurable config:
thread_idfrom langchain.agents import create_agent
from langgraph.checkpoint.memory import InMemorySaver
agent = create_agent(model, tools=[], checkpointer=InMemorySaver()) # model: ChatOllama in production
alice = {"configurable": {"thread_id": "alice-2026-09-29"}}
bob = {"configurable": {"thread_id": "bob-2026-09-29"}}
agent.invoke({"messages": [{"role": "user", "content": "Hi, my name is Alice."}]}, alice)
agent.invoke({"messages": [{"role": "user", "content": "Hi, my name is Bob."}]}, bob)
for label, cfg in [("alice", alice), ("bob", bob)]:
reply = agent.invoke({"messages": [{"role": "user", "content": "What is my name?"}]}, cfg)
print(label, "->", reply["messages"][-1].content, f"({len(reply['messages'])} messages)")
fresh = {"configurable": {"thread_id": "carol-2026-09-29"}}
print("carol ->", agent.invoke({"messages": [{"role": "user", "content": "What is my name?"}]}, fresh)["messages"][-1].content)
agent.invoke({"messages": [{"role": "user", "content": "hi"}]}) # no thread_id
Running it with the fake RecallModel printed:
alice -> Your name is Alice. (4 messages)
bob -> Your name is Bob. (4 messages)
carol -> I do not know your name.
ValueError Checkpointer requires one or more of the following 'configurable' keys: thread_id, checkpoint_ns, checkpoint_id
Each call only sends the new messages; the checkpointer loads the thread’s saved history, the reducer appends the
new input, and the model sees the whole list. A new thread_id starts an empty conversation. Omitting the ID
raises an error, because without it the checkpointer has nowhere to save the state. See
LangChain — Short-term memory and
LangGraph — Checkpointers.
Choosing thread IDs is an application decision with security consequences. The thread_id is the only thing
that selects a conversation, so anyone who can present it can read and continue that thread. Derive it on the
server from the authenticated user (for example <user id>:<random UUID>) and check on every request that the
thread belongs to the caller, instead of accepting an arbitrary ID from the client. The LangGraph documentation
also warns that Postgres stores the ID in a column with a length limit, so keep it under 255 characters.
config has thread_id"] -->|"load newest checkpoint"| C1 subgraph T["thread alice:0b7c..."] direction LR C0["empty, next = __start__"] --> C1["input saved, next = model"] C1 --> C2["reply saved, idle"] C2 --> C3["next turn: input saved"] end C3 -->|"append, run, save"| App
Choosing a checkpointer
All checkpointers implement the same BaseCheckpointSaver interface, so switching between them is a one-line
change where the agent is built. They ship as separate distributions: InMemorySaver comes with
langgraph-checkpoint (installed with LangGraph), SqliteSaver with langgraph-checkpoint-sqlite, and
PostgresSaver with langgraph-checkpoint-postgres.
| Checkpointer | Use it for | Notes |
|---|---|---|
|
Tests, notebooks, demos. |
Checkpoints live in RAM and are lost when the process exits. |
|
Local development, single-process tools, CLIs. |
One file, no server. The LangGraph documentation recommends it for experimentation and local workflows. |
|
Production services with more than one worker. |
Call |
The SQLite version keeps the conversation across a restart, which the in-memory version cannot do. The with block
below is one "process"; the second block builds a new saver and a new agent over the same file, as after a restart:
from langchain.agents import create_agent
from langgraph.checkpoint.sqlite import SqliteSaver
path = "docsassistant-checkpoints.sqlite"
cfg = {"configurable": {"thread_id": "alice-2026-09-29"}}
with SqliteSaver.from_conn_string(path) as saver: # first "process"
agent = create_agent(model, tools=[], checkpointer=saver)
agent.invoke({"messages": [{"role": "user", "content": "Hi, my name is Alice."}]}, cfg)
with SqliteSaver.from_conn_string(path) as saver: # after a "restart": new saver, new agent, same file
agent = create_agent(model, tools=[], checkpointer=saver)
reply = agent.invoke({"messages": [{"role": "user", "content": "What is my name?"}]}, cfg)
print(reply["messages"][-1].content, f"({len(reply['messages'])} messages)")
The output was Your name is Alice. (4 messages), and the file contained the tables checkpoints and writes. See
LangGraph — Checkpointer
libraries.
For production, use Postgres. DocsAssistant already runs PostgreSQL for its vectors (see PostgreSQL & pgvector), so the operational cost is one more database or schema, not one more system. Keep checkpoints in their own database or schema rather than mixed in with the vector tables, so retention jobs and backups can treat them differently. The connection string comes from the environment, never from source code:
import os
from langchain.agents import create_agent
from langgraph.checkpoint.postgres import PostgresSaver
DB_URI = os.environ["DOCSASSISTANT_DB_URL"] # e.g. postgresql://user@host:5432/docsassistant
cfg = {"configurable": {"thread_id": "alice-2026-09-29"}}
with PostgresSaver.from_conn_string(DB_URI) as saver:
saver.setup() # creates the checkpoint tables (idempotent)
agent = create_agent(model, tools=[], checkpointer=saver)
agent.invoke({"messages": [{"role": "user", "content": "Hi, my name is Alice."}]}, cfg)
with PostgresSaver.from_conn_string(DB_URI) as saver: # a different worker, later
agent = create_agent(model, tools=[], checkpointer=saver)
reply = agent.invoke({"messages": [{"role": "user", "content": "What is my name?"}]}, cfg)
print(reply["messages"][-1].content)
Against the PostgreSQL 17 container it printed Your name is Alice., and the database then held the tables
checkpoint_blobs, checkpoint_migrations, checkpoint_writes and checkpoints. The LangGraph documentation says
to call setup() the first time you use the Postgres checkpointer, so run it once at deployment or start-up rather
than on every request. The asynchronous services in Streaming and
Deployment use AsyncPostgresSaver (from langgraph.checkpoint.postgres.aio)
and AsyncSqliteSaver (from langgraph.checkpoint.sqlite.aio), which have the same role and the same setup()
convention. When the agent runs on LangSmith’s Agent Server, persistence is handled by the server and you do not
configure a checkpointer yourself. See LangChain — Short-term memory (the "In production" section).
Checkpoints contain the whole conversation, including anything a user typed, so treat the tables as personal
data. EncryptedSerializer.from_pycryptodome_aes() (from langgraph.checkpoint.serde.encrypted) encrypts all
persisted state; it reads the AES key from the LANGGRAPH_AES_KEY environment variable, and the key belongs in
your secret manager, not in the repository:
import sqlite3
from langgraph.checkpoint.serde.encrypted import EncryptedSerializer
from langgraph.checkpoint.sqlite import SqliteSaver
serde = EncryptedSerializer.from_pycryptodome_aes() # reads LANGGRAPH_AES_KEY from the environment
saver = SqliteSaver(sqlite3.connect("encrypted.sqlite", check_same_thread=False), serde=serde)
With a key in the environment, the agent still answered Your name is Alice., and a byte search of the
checkpoints table found the text "Alice" in the plain-text run and not in the encrypted one. Any checkpointer accepts the serde argument
(SqliteSaver(conn, serde=…) above); PostgresSaver.from_conn_string takes no serde, so for Postgres build the saver
from a psycopg connection with PostgresSaver(conn, serde=serde) (the settings from_conn_string itself uses: autocommit=True
and row_factory=dict_row). See
LangGraph — Checkpointers: Serializer.
Reading and editing state
Because the state is stored, it can be inspected and changed from outside a run. get_state(config) returns a
StateSnapshot for the thread’s newest checkpoint, update_state(config, values) writes a new checkpoint, and
get_state_history(config) lists every checkpoint, newest first. An agent built with create_agent is a compiled
graph, so all three are methods of the agent itself:
agent = create_agent(model, tools=[], checkpointer=InMemorySaver())
cfg = {"configurable": {"thread_id": "state-demo"}}
agent.invoke({"messages": [{"role": "user", "content": "Hi, my name is Alice."}]}, cfg)
snap = agent.get_state(cfg)
print("next:", snap.next, "| step:", snap.metadata["step"], "| source:", snap.metadata["source"])
print("messages:", [(m.type, m.content) for m in snap.values["messages"]])
agent.update_state(cfg, {"messages": [{"role": "user", "content": "Correction: my name is Alicia."}]})
snap2 = agent.get_state(cfg)
print("after update_state:", [(m.type, m.content) for m in snap2.values["messages"]])
print("source:", snap2.metadata["source"])
reply = agent.invoke({"messages": [{"role": "user", "content": "What is my name?"}]}, cfg)
print(reply["messages"][-1].content)
for s in agent.get_state_history(cfg):
print(" step", s.metadata["step"], s.metadata["source"], "next=", s.next, "msgs=", len(s.values.get("messages", [])))
next: () | step: 1 | source: loop
messages: [('human', 'Hi, my name is Alice.'), ('ai', 'Noted (1 messages in view).')]
after update_state: [('human', 'Hi, my name is Alice.'), ('ai', 'Noted (1 messages in view).'), ('human', 'Correction: my name is Alicia.')]
source: update
Your name is Alicia.
step 5 loop next= () msgs= 5
step 4 loop next= ('model',) msgs= 4
step 3 input next= ('__start__',) msgs= 3
step 2 update next= () msgs= 3
step 1 loop next= () msgs= 2
step 0 loop next= ('model',) msgs= 1
step -1 input next= ('__start__',) msgs= 0
Three things are worth reading in that output:
-
update_stateadds a checkpoint (source: update) and never rewrites an old one. Values go through the state’s reducers, so themessageskey, which uses theadd_messagesreducer, appends the correction instead of replacing the list. To remove a message you send aRemoveMessage, shown below. -
snap.nextis empty when the thread is idle and names the nodes still to run when it is not. A non-emptynexton a thread you did not expect to be running is how you spot a run that failed or paused (see the last section and Human-in-the-Loop & Time Travel). -
The history is the raw material for time travel: each entry carries a
configwith acheckpoint_id, and passing it toget_state,invokeorupdate_stateaddresses that moment instead of the newest one. Replaying and forking from such a checkpoint is covered in the time-travel page.
Custom fields go through the same mechanism. Subclass AgentState and pass it as state_schema, and the extra
keys are saved in every checkpoint and can be supplied at invocation time; the guide’s example adds user_id and
preferences this way:
from langchain.agents import create_agent, AgentState
class DocsState(AgentState):
user_id: str
preferences: dict
agent = create_agent(model, tools=[], state_schema=DocsState, checkpointer=InMemorySaver())
agent.invoke(
{"messages": [{"role": "user", "content": "Hello"}], "user_id": "user_123", "preferences": {"theme": "dark"}},
{"configurable": {"thread_id": "1"}},
)
See LangChain — Short-term memory (the
"Customizing agent memory" section) and
LangGraph — Get and update
state. State that should be shared by every thread of a user belongs in a store, not here; see
Long-Term Memory Stores. The graph fundamentals behind reducers and
StateGraph are in LangGraph Fundamentals.
Managing a growing history
The checkpointer keeps everything, and the model can read only what fits in its context window. Left alone, a thread eventually overflows the window, or gets slow and expensive long before that (the effects of a long context are explained in Context Engineering for Agents). There are three levers, and they differ in what happens to the information they drop.
A token budget for the history
Decide the budget first. The model’s context window must hold the system prompt, the tool schemas, the history, and room for the reply. With the window, the system prompt, the tool schemas and the tokens reserved for the reply, the history may use at most
and trimming keeps the longest suffix of the messages that fits, where is the token count of one message:
LangChain’s count_tokens_approximately estimates without a tokenizer as roughly
plus 3 extra tokens per message for the role and message
delimiters (both are parameters), and it can add the tool schemas to a count through its tools argument. It is
an estimate: use the provider’s real tokenizer, or the usage metadata on replies, when you need to stay close to
a hard limit. With Ollama, is the num_ctx the model is run with (an option of ChatOllama), not the maximum
the model architecture supports; see Ollama for how the context length is set.
from langchain.tools import tool
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage
from langchain_core.messages.utils import count_tokens_approximately, trim_messages
@tool
def search_docs(query: str, k: int = 5) -> str:
"""Search this site's documentation and return the best matching sections."""
return ""
W = 8192 # num_ctx the model runs with
R = 1024 # tokens reserved for the reply
system = SystemMessage("You are DocsAssistant. Answer only from the documentation and cite page paths.")
S_tok = count_tokens_approximately([system])
T_tok = count_tokens_approximately([], tools=[search_docs])
H = W - S_tok - T_tok - R
print(f"S={S_tok} T={T_tok} R={R} W={W} -> H_max={H}")
history = []
for i in range(1, 121):
history.append(HumanMessage(f"Question {i}: how do I tune the pgvector index for a corpus of {i * 1000} chunks?"))
history.append(AIMessage("Use HNSW with m=16 and ef_construction=64, then raise hnsw.ef_search until recall stops improving. " * 2))
print("messages:", len(history), "total approx tokens:", count_tokens_approximately(history))
kept = trim_messages(history, strategy="last", token_counter=count_tokens_approximately,
max_tokens=H, start_on="human")
print("kept:", len(kept), "messages,", count_tokens_approximately(kept), "tokens (<= H_max:", count_tokens_approximately(kept) <= H, ")")
print("first kept:", kept[0].content[:12], "| last kept:", kept[-1].type)
S=24 T=72 R=1024 W=8192 -> H_max=7072
messages: 240 total approx tokens: 9381
kept: 180 messages, 7041 tokens (<= H_max: True )
first kept: Question 31: | last kept: ai
Here tokens, the full 240-message thread is estimated at 9,381 tokens, and
trim_messages keeps the newest 180 messages (7,041 tokens), starting on a human message so the model does not
see a reply with no question. start_on="human" is what makes the kept list a valid conversation; end_on does
the same at the other end, and include_system=True keeps a leading system message. See
LangGraph — Add memory: Trim messages.
Trimming and filtering in an agent
In an agent, trimming is a middleware. There are two designs, and the difference matters. A @before_model hook
that returns RemoveMessage(id=REMOVE_ALL_MESSAGES) followed by the kept messages rewrites the stored state:
the checkpoint loses the old messages for good. A @wrap_model_call hook that passes a trimmed list to the model
through request.override(messages=…) changes only what the model sees on that call, and the checkpoint keeps
the full history:
from typing import Any
from langchain.agents import create_agent, AgentState
from langchain.agents.middleware import before_model, wrap_model_call
from langchain.messages import RemoveMessage
from langchain_core.messages.utils import count_tokens_approximately, trim_messages
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.graph.message import REMOVE_ALL_MESSAGES
from langgraph.runtime import Runtime
MAX_HISTORY_TOKENS = 60 # tiny on purpose, so the demo trims after a few turns
def trimmed(messages):
return trim_messages(messages, strategy="last", token_counter=count_tokens_approximately,
max_tokens=MAX_HISTORY_TOKENS, start_on="human", include_system=True)
@before_model
def trim_state(state: AgentState, runtime: Runtime) -> dict[str, Any] | None:
"""Destructive: rewrite the stored history so it stays within the budget."""
messages = state["messages"]
kept = trimmed(messages)
if len(kept) == len(messages):
return None
return {"messages": [RemoveMessage(id=REMOVE_ALL_MESSAGES), *kept]}
@wrap_model_call
def trim_view(request, handler):
"""Non-destructive: the checkpoint keeps everything, only the model call sees the trimmed list."""
return handler(request.override(messages=trimmed(request.messages)))
Driving the same four-turn conversation ("Hi, my name is Alice.", two documentation questions, then "What is my
name?") through an agent with no trimming, with trim_state, and with trim_view gave:
no trimming : reply='Your name is Alice.'; stored=8 messages; approx tokens=101
trim_state (edit): reply='I do not know your name.'; stored=4 messages; approx tokens=50
trim_view (view) : reply='I do not know your name.'; stored=8 messages; approx tokens=103
Both trimmers made the model forget the name, which is the price of trimming: the oldest information is simply not
in the prompt any more. They differ in what is left afterwards. trim_state shrinks the checkpoint (fewer bytes
to store and reload, but the dropped messages are gone from the thread and from any later time travel), while
trim_view keeps a complete audit trail and pays the storage and load cost of the full history. Pick the view
approach when the full transcript has value (support review, evaluation datasets) and the state approach when the
history is large and disposable. The LangChain guide’s own example of the state approach is a @before_model
function that keeps the first message and the last few; name your function something other than trim_messages
so it does not shadow the langchain_core helper. See
LangChain — Short-term memory:
Trim messages and LangChain — Custom
middleware; the middleware hooks themselves are explained in Middleware.
Filtering is the selective version of trimming: instead of cutting by age or size, drop messages by what they
are. filter_messages (from langchain_core.messages) includes or excludes messages by type, name or ID, and its
exclude_tool_calls argument removes tool calls together with their tool results; use it on the list you pass
to request.override(messages=…) to hide bulky tool output from later turns while the checkpoint keeps it.
To delete specific messages from the stored state, return RemoveMessage(id=m.id) for each, or
RemoveMessage(id=REMOVE_ALL_MESSAGES) to clear the thread; this works because the default AgentState keeps
messages under the add_messages reducer. Whatever you drop, keep the remaining history valid for the provider:
some expect the first message to be a user message, and most require every assistant message that made tool calls
to be followed by the matching tool results. See
LangChain — Short-term memory:
Delete messages.
Summarising with SummarizationMiddleware
Trimming loses information. SummarizationMiddleware (from langchain.agents.middleware) instead replaces the
older part of the history with a model-written summary once a threshold is crossed, and keeps the recent messages
verbatim. Its main parameters are:
| Parameter | Meaning |
|---|---|
|
The model that writes the summary: an identifier string or a chat model instance. It can be cheaper than the agent’s own model. |
|
When to summarise: a tuple such as |
|
How much recent context to preserve unsummarised: exactly one of |
|
A custom counter, a custom summary prompt containing a |
With a scripted summariser (a GenericFakeChatModel that always returns the same summary) and thresholds small
enough to trigger after a few turns:
from itertools import repeat
from langchain.agents import create_agent
from langchain.agents.middleware import SummarizationMiddleware
from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from langchain_core.messages import AIMessage
from langgraph.checkpoint.memory import InMemorySaver
summarizer = GenericFakeChatModel(messages=repeat(AIMessage(
"SESSION INTENT: user Alice explores DocsAssistant. SUMMARY: my name is Alice; she asked about HNSW and IVFFlat tuning."
)))
agent = create_agent(
model,
tools=[],
middleware=[SummarizationMiddleware(model=summarizer, trigger=("messages", 6), keep=("messages", 2))],
checkpointer=InMemorySaver(),
)
cfg = {"configurable": {"thread_id": "summary-demo"}}
for text in ["Hi, my name is Alice.", "Which page explains HNSW ef_search?", "And IVFFlat probes?", "What is my name?"]:
out = agent.invoke({"messages": [{"role": "user", "content": text}]}, cfg)
print(len(out["messages"]), "messages after:", text)
for m in agent.get_state(cfg).values["messages"]:
print(f" {m.type:5} {m.content[:95]!r}")
2 messages after: Hi, my name is Alice.
4 messages after: Which page explains HNSW ef_search?
6 messages after: And IVFFlat probes?
4 messages after: What is my name?
human 'Here is a summary of the conversation to date:\n\nSESSION INTENT: user Alice explores DocsAssistant. ...'
ai 'Noted (5 messages in view).'
human 'What is my name?'
ai 'Your name is Alice.'
The four earlier turns collapsed into one leading message that begins "Here is a summary of the conversation to
date", followed by the messages that were kept, and the agent could still answer "What is my name?" because the
summary carried the fact. Two properties are worth remembering. Summaries are lossy and model-written, so a
detail that must never be forgotten (a user’s stated constraint, a ticket number) is better stored explicitly in
state or in a long-term store than hoped for in a summary. And summarisation is text-only compression: recent
messages in keep retain their original image or audio blocks, while older multimodal messages are reduced to
the text of the summary, as the built-in middleware reference explains. See
LangChain — Built-in
middleware: Summarization.
DocsAssistant combines the pieces: a Postgres checkpointer, a per-user thread ID, a summariser that triggers before
the 7,072-token budget computed above is reached, and the local model’s num_ctx set to the used in the
formula:
import os
import uuid
from langchain.agents import create_agent
from langchain.agents.middleware import SummarizationMiddleware
from langchain_ollama import ChatOllama
from langgraph.checkpoint.postgres import PostgresSaver
CHAT = dict(model="llama3.1:8b", temperature=0, num_ctx=8192) # num_ctx is the W in the budget formula
def new_thread_id(user_id: str) -> str:
return f"{user_id}:{uuid.uuid4()}" # well under Postgres' 255-character limit
with PostgresSaver.from_conn_string(os.environ["DOCSASSISTANT_DB_URL"]) as checkpointer:
checkpointer.setup()
agent = create_agent(
ChatOllama(**CHAT),
tools=[], # search_docs and friends: see the Tools page
system_prompt="You are DocsAssistant. Answer only from the documentation and cite page paths.",
middleware=[
SummarizationMiddleware(
model=ChatOllama(model="llama3.1:8b", temperature=0),
trigger=("tokens", 6000), # summarise before the 7,072-token budget is hit
keep=("messages", 12),
)
],
checkpointer=checkpointer,
)
tid = new_thread_id("alice")
print(type(agent).__name__, len(tid), agent.get_state({"configurable": {"thread_id": tid}}).values)
Constructing this agent against the PostgreSQL container printed CompiledStateGraph 42 {}: an agent, a 42-character
thread ID and an empty state for a thread with no runs yet. No message was sent, because there was no Ollama server
to answer. The search_docs tool this agent would carry is defined in Tools, the
loop around it in Agents with create_agent, and long conversations
where the summariser is not enough (agent-controlled context, offloading to files) are discussed in
Context Engineering for Agents. See
LangChain — Short-term memory.
Retention and cleanup
Checkpoints accumulate: every super-step of every run adds one, so a busy service grows its checkpoint tables
without bound. The LangGraph documentation lists this as a common issue and recommends pruning old checkpoints
periodically or setting a retention policy, for example a scheduled job that deletes threads older than N days.
checkpointer.delete_thread(thread_id) deletes every checkpoint and pending write for one thread; it is also the
building block for honouring a user’s request to delete their conversations:
graph.checkpointer.delete_thread("answer-42")
print(graph.get_state(cfg).values, list(graph.get_state_history(cfg))) # -> {} []
After the call the thread had no state and no history. For very long, append-heavy threads, LangGraph 1.2 also
offers an opt-in DeltaChannel that stores incremental deltas instead of the full accumulated value at each
step; its documentation marks it beta, so read its trade-off notes before relying on it. See
LangGraph — Checkpointers: Optimize checkpoint storage. The privacy side of storing conversations (what to keep, for how long,
and who may read it) is discussed in Agent Memory and in
Agent Memory & Semantic Caching.
Durable execution and fault tolerance
The same checkpoints that give a conversation its memory make a run durable: if a run fails, the thread’s last good state is on disk, and the run can be resumed instead of restarted. Two mechanisms work together. Checkpoints are written at super-step boundaries. Inside a super-step, each node’s output is also written as soon as that node finishes, as a pending write linked to the in-progress checkpoint. When one of several nodes that run in parallel fails, the outputs of the ones that succeeded are already saved, and on resume they are not run again.
To watch it, take a small DocsAssistant graph in which draft_answer (standing for an expensive model call) and
check_links run in parallel, and make check_links fail on the first run:
import operator
import os
import tempfile
from collections import Counter
from typing import Annotated
from langgraph.checkpoint.sqlite import SqliteSaver
from langgraph.graph import END, START, StateGraph
from typing_extensions import TypedDict
calls = Counter()
network_up = False # the link checker's upstream is down on the first run
class State(TypedDict):
question: str
notes: Annotated[list[str], operator.add]
def retrieve(state: State):
calls["retrieve"] += 1
return {"notes": ["retrieved 5 chunks"]}
def draft_answer(state: State):
calls["draft_answer"] += 1 # stands in for an expensive LLM call
return {"notes": ["drafted answer"]}
def check_links(state: State):
calls["check_links"] += 1
if not network_up:
raise ConnectionError("link checker unreachable")
return {"notes": ["links ok"]}
def publish(state: State):
calls["publish"] += 1
return {"notes": ["published"]}
builder = StateGraph(State)
for fn in (retrieve, draft_answer, check_links, publish):
builder.add_node(fn)
builder.add_edge(START, "retrieve")
builder.add_edge("retrieve", "draft_answer") # draft_answer and check_links run in the same super-step
builder.add_edge("retrieve", "check_links")
builder.add_edge("draft_answer", "publish")
builder.add_edge("check_links", "publish")
builder.add_edge("publish", END)
path = os.path.join(tempfile.mkdtemp(), "durable.sqlite")
cfg = {"configurable": {"thread_id": "answer-42"}}
with SqliteSaver.from_conn_string(path) as saver:
graph = builder.compile(checkpointer=saver)
try:
graph.invoke({"question": "How do I tune HNSW?", "notes": []}, cfg, durability="sync")
except ConnectionError as err:
print("run 1 failed:", err)
snap = graph.get_state(cfg)
print("next after failure:", sorted(snap.next), "| notes:", snap.values["notes"])
print("calls so far:", dict(calls))
network_up = True # the dependency recovers
result = graph.invoke(None, cfg, durability="sync") # None = resume this thread from its last checkpoint
print("resumed notes:", result["notes"])
print("calls in total:", dict(calls))
run 1 failed: link checker unreachable
next after failure: ['check_links'] | notes: ['retrieved 5 chunks', 'drafted answer']
calls so far: {'retrieve': 1, 'check_links': 1, 'draft_answer': 1}
resumed notes: ['retrieved 5 chunks', 'links ok', 'drafted answer', 'published']
calls in total: {'retrieve': 1, 'check_links': 2, 'draft_answer': 1, 'publish': 1}
After the failure the snapshot shows that only check_links is still to run, and drafted answer is already in
the state. Invoking the graph with None as the input on the same thread_id resumes it: draft_answer and
retrieve ran once in total, check_links ran twice, and publish ran once. Two consequences follow for how you
write nodes. Work is saved per node, so keep expensive steps (model calls, retrievals) in their own nodes instead
of one large node that redoes everything on a retry. And a node that failed runs again from its first line,
so a node that has side effects (sending a message, opening a GitHub issue) should be idempotent or guarded by an
approval step; the pattern for the approval gate is in
Human-in-the-Loop & Time Travel and
Human-in-the-Loop.
How soon a checkpoint reaches the database is set by the durability mode, an argument of every execution
method (invoke, stream and their asynchronous forms):
durability |
Behaviour |
|---|---|
|
Persist only when the run exits, whether successfully, with an error, or at a human-in-the-loop interrupt. Best performance for long-running graphs, but a process crash in the middle of a run leaves nothing to resume from. |
|
Persist asynchronously while the next step executes. Good performance and durability, with a small risk that a checkpoint is not written if the process dies during execution. |
|
Persist synchronously before the next step starts. Every checkpoint is written before execution continues, at the cost of some overhead. |
The in-process failure above cannot tell the modes apart, because all three save the state when an exception
propagates. A killed process can. This script runs a three-node graph, retrieve then draft_answer then
publish, in two invocations of the same program. In the crash phase publish calls os._exit(1), which stops
the process the way a kill or an out-of-memory error would; the resume phase starts a new process on the same
SQLite file:
import operator
import os
import sys
from typing import Annotated
from langgraph.checkpoint.sqlite import SqliteSaver
from langgraph.graph import END, START, StateGraph
from typing_extensions import TypedDict
DB, MODE, PHASE = sys.argv[1], sys.argv[2], sys.argv[3] # PHASE: crash | resume
class State(TypedDict):
notes: Annotated[list[str], operator.add]
def retrieve(state: State):
return {"notes": ["retrieved"]}
def draft_answer(state: State):
print(" draft_answer runs (expensive)")
return {"notes": ["drafted"]}
def publish(state: State):
if PHASE == "crash":
os._exit(1) # simulate the process being killed mid-run
return {"notes": ["published"]}
builder = StateGraph(State)
for fn in (retrieve, draft_answer, publish):
builder.add_node(fn)
builder.add_edge(START, "retrieve")
builder.add_edge("retrieve", "draft_answer")
builder.add_edge("draft_answer", "publish")
builder.add_edge("publish", END)
cfg = {"configurable": {"thread_id": "answer-42"}}
with SqliteSaver.from_conn_string(DB) as saver:
graph = builder.compile(checkpointer=saver)
snap = graph.get_state(cfg)
print(f"[{PHASE}/{MODE}] on start: next={snap.next} notes={snap.values.get('notes')}")
if PHASE == "crash":
graph.invoke({"notes": []}, cfg, durability=MODE)
else:
print(" result:", graph.invoke(None if snap.next else {"notes": []}, cfg, durability=MODE)["notes"])
Running crash then resume for sync and for exit (each with its own database file, python -u so the
output is not lost when the process is killed) printed:
[crash/sync] on start: next=() notes=None
draft_answer runs (expensive)
exit code 1
[resume/sync] on start: next=('publish',) notes=['retrieved', 'drafted']
result: ['retrieved', 'drafted', 'published']
[crash/exit] on start: next=() notes=None
draft_answer runs (expensive)
exit code 1
[resume/exit] on start: next=() notes=None
draft_answer runs (expensive)
result: ['retrieved', 'drafted', 'published']
With "sync", the new process found the thread waiting at publish with the drafted answer already saved, and
only publish ran. With "exit", the crash left nothing behind: the new process saw an empty thread and had to
run draft_answer again from the start. Use "sync" for anything expensive or user-visible, such as an agent
loop that makes many model calls; use "exit" only for short graphs where rerunning from scratch is cheap.
Durability also means a worker is disposable: with PostgresSaver, a supervisor can restart a
crashed worker or hand the thread to another one, and the next invoke(None, config) on that thread_id
continues. See LangGraph — Checkpointers: Durability modes and LangGraph — Persistence. The same resume mechanism is what an interrupt() uses to wait for a person; see
Human-in-the-Loop & Time Travel. Streaming partial output
while a durable run progresses is covered in Streaming.