Learning & Improvement
|
This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production. This section’s bibliography lists the reference material consulted while preparing these pages. |
This page’s examples target Python 3.12 with LangChain 1.4.2 / langchain-core 1.6.5 / LangGraph 1.2.12 and
langchain-ollama 1.1.0, using DocsAssistant's Ollama llama3.1:8b chat model and nomic-embed-text embeddings,
as checked on 2026-09-28 (see the AI Agents baseline).
An agent that is exactly as good in month six as on launch day is wasting its most valuable input: the record of what it did and how people reacted. There are two ways to turn that record into a better agent. Non-parametric learning changes what the agent sees — its prompts, examples, memories and tools — without touching the model’s weights; it is cheap, fast, reversible and where almost all agents should start. Parametric learning changes the weights by fine-tuning; it is slower and costlier, and pays off only once the non-parametric options are exhausted and there is a lot of good data.
Non-parametric learning
Few-shot exemplars
The simplest form of learning is to show the model what "good" looks like. Rather than a fixed set of examples in the system prompt, store every answer a reviewer approved, and at each turn retrieve the few most similar to the current question — dynamic few-shot selection. The agent’s behaviour then improves as the exemplar store grows, with no deployment.
import hashlib
from langchain.agents import create_agent
from langchain.agents.middleware import ModelRequest, dynamic_prompt
from langchain_ollama import ChatOllama
BASE = "You are DocsAssistant. Answer from the documentation and cite page paths."
@dynamic_prompt
def with_exemplars(request: ModelRequest) -> str:
question = next(m.content for m in reversed(request.state["messages"]) if m.type == "human")
hits = request.runtime.store.search(("docsassistant", "exemplars"), query=question, limit=3)
shots = "\n\n".join(f"Q: {h.value['question']}\nGood answer: {h.value['answer']}" for h in hits)
return f"{BASE}\n\nExamples of answers reviewers approved:\n{shots}" if shots else BASE
agent = create_agent(ChatOllama(model="llama3.1:8b", temperature=0), tools=[docs_search],
middleware=[with_exemplars], store=store)
def record_approved(question: str, answer: str) -> None: # called when a reviewer approves an answer
store.put(("docsassistant", "exemplars"), key=hashlib.sha256(question.encode()).hexdigest()[:16],
value={"text": question, "question": question, "answer": answer})@dynamic_prompt rebuilds the system prompt before every model call from the current state and the store; store
is a LangGraph store with semantic search over the text field, as on
Memory. Only reviewer-approved answers go in, so a
bad answer cannot teach the agent to repeat itself. See
LangChain — Agents (dynamic prompts) and
LangChain — Long-term memory.
Reflexion memories
Where exemplars store successes, Reflexion stores lessons from failures: after a failed or criticised attempt,
the agent writes a short verbal lesson ("search the pgvector page, not the HNSW page, for index parameters") and
reads it back on the next attempt. The mechanics — the draft, reflect and retry loop and the lessons state — are
on Agent Loop & Architectures. To make
the lessons outlive one run, write them to long-term memory (namespaced by task type rather than by user, since a
lesson about the docs applies to everyone) and retrieve them the same way as the exemplars above. This is the
experience memory described on Memory.
Prompt and tool refinement
Much of an agent’s improvement is ordinary engineering on its prompts and tools, driven by traces: read the failed runs, find the pattern (a tool called with the wrong argument, a question type always misrouted), change the system prompt, a tool description or a tool’s behaviour, and confirm the fix on the evaluation set before shipping (see Agent Evaluation). Tool descriptions are the most under-used lever: a clearer description or a better error message often fixes what looked like a model limitation (see Tool Design). Models can help here too — given failing traces and the current tool definitions, a strong model can propose rewrites, which then go through the same evaluation gate.
Parametric learning
When prompts, exemplars and tools stop improving the agent, the next step is to change the model:
-
Supervised fine-tuning (SFT) on curated examples of good agent behaviour — whole trajectories with correct tool calls and grounded answers — teaches a smaller model to act like a larger one, or any model to follow house conventions reliably.
-
Direct Preference Optimization (DPO) trains on pairs of a preferred and a rejected response (a reviewer’s approved answer against the rejected draft), without training a separate reward model.
-
Reinforcement learning with verifiable rewards (RLVR) optimises against an automatic check — the tool call parsed, the cited page exists, the test passed — and is how the current reasoning models were trained.
The training methods themselves are explained on How LLMs Are Trained. Fine-tuning an open model for an agent, with Hugging Face’s TRL library, is covered in Fine-Tuning with PEFT & TRL in the Hugging Face & On-Device Models sub-section; see also Hugging Face TRL. The data these methods need comes from the feedback loops below, which is another reason to build them first.
Feedback loops
Every improvement above needs a signal. Collect it deliberately, from several sources, and store it next to the trace it refers to:
-
Explicit user feedback — thumbs up/down and short comments on answers. Sparse and biased toward extremes, but unambiguous.
-
Implicit signals — the user rephrased the question (the answer missed), copied the answer (it helped), abandoned the session, or escalated.
-
Reviewer decisions — every approval, edit and rejection at a human-in-the-loop gate is a labelled example; an edit is the most valuable of all, because it shows the correct action (see Human-in-the-Loop).
-
Automatic checks — judge scores and verifiable checks run on a sample of production traffic.
Then close the loop: route failures into the evaluation dataset, approvals into the exemplar store, lessons into experience memory, and — once there is enough — pairs into a preference dataset. Before any change ships, compare it against the current version, offline on the evaluation set and online with an A/B test or a bandit.
Bayesian bandits for choosing prompts and tools
An A/B test splits traffic evenly between variants for a fixed period, then picks a winner. That is clean but wasteful: half the users keep getting the worse variant until the test ends. A multi-armed bandit instead shifts traffic toward the variant that is doing better while it learns. Thompson sampling is the standard Bayesian version for a success/failure signal such as a thumbs up.
For each variant (arm) , keep a Beta distribution over its unknown success rate , starting from a uniform prior . On every conversation:
-
sample for every arm;
-
play the arm with the largest sample, ;
-
observe a reward and update only that arm:
The posterior mean is the arm’s estimated success rate, and the spread of the Beta distribution shrinks as evidence accumulates — so an uncertain arm still gets tried occasionally (exploration), while a clearly better arm gets most of the traffic (exploitation). Because the Beta distribution is conjugate to the Bernoulli likelihood, the update is just two additions.
import random
from dataclasses import dataclass
@dataclass
class Arm:
name: str # e.g. a system-prompt variant or a tool-description variant
alpha: float = 1.0 # 1 + successes (Beta(1, 1) = uniform prior)
beta: float = 1.0 # 1 + failures
class ThompsonSampler:
def __init__(self, names: list[str]):
self.arms = {n: Arm(n) for n in names}
def choose(self) -> str:
# Sample a plausible success rate for each arm; play the best sample.
return max(self.arms.values(), key=lambda a: random.betavariate(a.alpha, a.beta)).name
def update(self, name: str, reward: int) -> None: # reward: 1 = thumbs up, 0 = thumbs down
arm = self.arms[name]
arm.alpha += reward
arm.beta += 1 - reward
prompts = ThompsonSampler(["concise-v3", "step-by-step-v1", "cite-first-v2"])
variant = prompts.choose() # pick the system prompt for this conversation
# ... run DocsAssistant with SYSTEM_PROMPTS[variant], collect the user's feedback ...
prompts.update(variant, reward=1)In a simulation with true success rates of 0.8, 0.5 and 0.6, this sampler sent about 97% of 2,000 conversations to the best variant, while still testing the others early on. The same sampler chooses among tool-description variants, models or retrieval settings. Two cautions: rewards should arrive reasonably soon after the choice (a bandit on a signal that takes weeks learns slowly), and a bandit optimises exactly the reward it is given — if thumbs up rewards confident-sounding answers, it will learn confidence, not correctness, so pair it with the groundedness checks on Agent Evaluation. See Russo et al. — A Tutorial on Thompson Sampling.
In other frameworks
| Framework | Learning and improvement support |
|---|---|
OpenAI Agents SDK |
Dynamic instructions (a callable |
Claude Agent SDK |
Skills, |
Microsoft Agent Framework |
Context providers and memory integrations inject learned context per run; built-in evaluation types support the offline comparison step. |
CrewAI |
Crew memory accumulates across runs, and |