Chat Models & Messages

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page was written against Python 3.12 with langchain 1.4.3, langchain-core 1.6.5, langchain-ollama 1.1.0, langchain-anthropic 1.7.4, langchain-openai 1.6.6 and langchain-huggingface 1.2.2, all checked on 2026-09-29 (see Ecosystem & Packages for the package map). No Ollama server was available while writing it, so every output shown below was produced with LangChain’s GenericFakeChatModel (a scripted fake) and the code that builds real llama3.1:8b, Claude and OpenAI models was only constructed, never called. The model names are the AI section’s baseline: llama3.1:8b and nomic-embed-text on Ollama for DocsAssistant, and claude-sonnet-5, claude-opus-5-5, claude-haiku-4-5 and gpt-6-sol as hosted models.

A chat model is the component that takes a list of messages and returns a message. Everything else in the framework — prompts, chains, tools, agents, LangGraph nodes — is built on that one contract, so it pays to learn it well: how to create a model without hard-wiring one provider, what the messages it exchanges look like, and how to call it in the ways production code needs (in bulk, incrementally, rate-limited, with a backup, and with token accounting). The model-side concepts behind this — tokens, sampling, context windows — are in What Is an LLM? and Sampling & Decoding.

Creating a model with `init_chat_model`

init_chat_model is the provider-neutral factory. You give it a model name and, optionally, a provider; it imports the matching integration package, builds the right chat-model class and passes the remaining keyword arguments through to it. Because the provider is just a string, it can come from configuration instead of code.

init_chat_model for a local, an Anthropic and an OpenAI model

from langchain.chat_models import init_chat_model

# DocsAssistant's default: a local model served by Ollama
local = init_chat_model("llama3.1:8b", model_provider="ollama", temperature=0)
print(type(local).__name__, local.model)          # ChatOllama llama3.1:8b

# Hosted models; API keys are read from ANTHROPIC_API_KEY / OPENAI_API_KEY
claude = init_chat_model("claude-sonnet-5", model_provider="anthropic")
print(type(claude).__name__)                      # ChatAnthropic

# The "provider:model" shorthand is equivalent
gpt = init_chat_model("openai:gpt-6-sol")
print(type(gpt).__name__)                         # ChatOpenAI

This snippet was executed; it constructs the three clients without contacting any server. See LangChain — Models and the init_chat_model reference.

Any keyword argument the integration class accepts (temperature, max_tokens, timeout, max_retries, base_url, rate_limiter, …) can be passed straight through. Two details save time in practice:

Name the provider explicitly (model_provider="ollama", or the provider:model shorthand) for local models: a tag such as llama3.1:8b says nothing about which integration should serve it.
The result is a BaseChatModel, so anything that accepts a chat model accepts it — including the model= argument of create_agent (see Agents with create_agent).

Chat models also expose a profile dictionary describing what the model supports (tool calling, structured output, context window, reasoning levels, …), which lets an application adapt to the model instead of hard-coding assumptions. The profile data comes from the open-source models.dev project and is a beta feature whose format may change; see the "Model profiles" section of LangChain — Models.

Provider packages

LangChain 1.x keeps the core small: init_chat_model lives in langchain, and each provider ships as its own partner package that is imported only when needed. The langchain package defines extras that install the package for you, for example pip install -U "langchain[anthropic]".

Table 1. Provider packages and their chat-model classes
Provider	Package	Class	Notes
Ollama (local)	`langchain-ollama`	`ChatOllama`	DocsAssistant’s default. Pull the tag first (`ollama pull llama3.1:8b`) and use the same tag in `model=`. See Ollama.
Anthropic	`langchain-anthropic`	`ChatAnthropic`	Reads `ANTHROPIC_API_KEY`.
OpenAI	`langchain-openai`	`ChatOpenAI`	Reads `OPENAI_API_KEY`. `base_url` also targets OpenAI-compatible servers such as vLLM.
Hugging Face	`langchain-huggingface`	`ChatHuggingFace` wrapping `HuggingFaceEndpoint`	Reads `HUGGINGFACEHUB_API_TOKEN`. Hugging Face has its own sub-section, Hugging Face & On-Device Models; here it is only a provider.

Each class can be used directly when you need provider-specific parameters that init_chat_model does not know about. The same call, without the factory:

Direct construction of the local and the Hugging Face models

from langchain_ollama import ChatOllama
from langchain_huggingface import ChatHuggingFace, HuggingFaceEndpoint

local = ChatOllama(model="llama3.1:8b", temperature=0)

# HUGGINGFACEHUB_API_TOKEN is read from the environment
hf = ChatHuggingFace(llm=HuggingFaceEndpoint(
    repo_id="microsoft/Phi-3-mini-4k-instruct",
    temperature=0.7,
    max_new_tokens=512,
))

The Hugging Face construction was executed with a placeholder token in the environment (nothing was sent to the service). See the integration pages for Ollama, Anthropic, OpenAI and Hugging Face. Running the models themselves — hardware, quantization, servers — belongs to the Running LLMs Locally sub-section; LangChain-specific local setup is on Local Models.

Messages

A chat model consumes and produces messages: small objects with a role, some content and metadata. They are imported from langchain.messages.

Class Role

Class	Role
`SystemMessage`	Instructions that frame the whole conversation: persona, rules, output format.
`HumanMessage`	What the user said. Its content can be text or a list of content blocks (text, image, audio, file).
`AIMessage`	What the model returned: `content`, any `tool_calls`, plus `usage_metadata` and `response_metadata`. Streaming yields `AIMessageChunk` objects that add up to an `AIMessage`.
`ToolMessage`	The result of running one tool call, linked to the request by `tool_call_id`.

SystemMessage

Instructions that frame the whole conversation: persona, rules, output format.

HumanMessage

What the user said. Its content can be text or a list of content blocks (text, image, audio, file).

AIMessage

What the model returned: content, any tool_calls, plus usage_metadata and response_metadata. Streaming yields AIMessageChunk objects that add up to an AIMessage.

ToolMessage

The result of running one tool call, linked to the request by tool_call_id.

The four types are exactly what one tool-using turn of DocsAssistant consists of:

A DocsAssistant conversation as message objects

from langchain.messages import SystemMessage, HumanMessage, AIMessage, ToolMessage

history = [
    SystemMessage("You answer questions about this documentation site."),
    HumanMessage("Which index does pgvector use by default?"),
    AIMessage(
        content="",
        tool_calls=[{"name": "search_docs", "args": {"query": "pgvector default index"}, "id": "call_1"}],
    ),
    ToolMessage(
        "modules/ROOT/pages/database/vector-rag/postgresql-pgvector.adoc: no index by default (exact scan)",
        tool_call_id="call_1",
    ),
]
for m in history:
    print(m.type, "|", m.content[:50] or m.tool_calls)

Output (executed):

system | You answer questions about this documentation site
human | Which index does pgvector use by default?
ai | [{'name': 'search_docs', 'args': {'query': 'pgvector default index'}, 'id': 'call_1', 'type': 'tool_call'}]
tool | modules/ROOT/pages/database/vector-rag/postgresql-

A model can be called with a plain string (a single HumanMessage), a list of message objects, or a list of role/content dictionaries in the OpenAI style; all three are equivalent. See LangChain — Messages. The tool-calling loop that consumes tool_calls and produces ToolMessage objects is built in Tools; its design rules are in Tool Design.

Provider-agnostic `content_blocks`

Providers disagree about how to represent anything richer than text. Anthropic returns thinking blocks, OpenAI returns reasoning items, images are image_url in one API and image with a source in another. LangChain 1.x keeps content in whatever shape the provider used, and adds a content_blocks property that lazily parses it into one standard, typed representation (text, reasoning, image, audio, video, file, tool-call and a few more). Code written against content_blocks does not change when you swap the model.

Reading reasoning and text uniformly, and building a multimodal message

from langchain.messages import AIMessage, HumanMessage

ai = AIMessage(
    content=[
        {"type": "thinking", "thinking": "Compare index types.", "signature": "sig"},
        {"type": "text", "text": "HNSW or IVFFlat, both optional."},
    ],
    response_metadata={"model_provider": "anthropic"},
)
print(ai.content_blocks)
print(ai.text)

msg = HumanMessage(content_blocks=[
    {"type": "text", "text": "What does this diagram show?"},
    {"type": "image", "url": "https://example.com/architecture.png"},
])
print(msg.content)

Output (executed):

[{'type': 'reasoning', 'reasoning': 'Compare index types.', 'extras': {'signature': 'sig'}}, {'type': 'text', 'text': 'HNSW or IVFFlat, both optional.'}]
HNSW or IVFFlat, both optional.
[{'type': 'text', 'text': 'What does this diagram show?'}, {'type': 'image', 'url': 'https://example.com/architecture.png'}]

The Anthropic thinking block came out as a standard reasoning block, with the provider-specific signature kept under extras. .text returns just the reply text, which is what most code wants to print. Passing content_blocks= when creating a message also populates content, so the message stays valid for older code. The full list of block types is in the "Content block reference" of LangChain — Messages; images, audio and files in practice are covered in Multimodality & Voice and Multimodal Models.

Older tutorials read message.content as a string and branch on provider-specific dictionaries. That still works for text-only replies, but it is the first thing to replace when a reasoning model or a multimodal input shows up.

`invoke`, `batch` and `stream`

Chat models are Runnables (see LCEL & Runnables), so they share the three call styles every Runnable has, plus async variants (ainvoke, abatch, astream).

The three call styles, against a scripted fake model

from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from langchain_core.messages import AIMessage


def fake(*texts):
    return GenericFakeChatModel(messages=iter([AIMessage(t) for t in texts]))

# invoke: one input, one AIMessage
r = fake("pgvector has no index by default.").invoke("Which index does pgvector use by default?")
print(type(r).__name__, r.content)

# stream: AIMessageChunk objects as they are generated
print([c.text for c in fake("HNSW builds a layered graph").stream("Explain HNSW")])

# chunks add up to a full message
full = None
for c in fake("HNSW builds a layered graph").stream("Explain HNSW"):
    full = c if full is None else full + c
print(type(full).__name__, repr(full.content))

# batch: many inputs, with a cap on parallel calls
out = fake("a", "b", "c").batch(["q1", "q2", "q3"], config={"max_concurrency": 2})
print([m.content for m in out])

Output (executed):

AIMessage pgvector has no index by default.
['HNSW', ' ', 'builds', ' ', 'a', ' ', 'layered', ' ', 'graph']
AIMessageChunk 'HNSW builds a layered graph'
['a', 'b', 'c']

invoke blocks until the whole reply is ready and returns an AIMessage. It is the right default for tool calls, structured output and anything whose result you need in full before continuing.
stream yields AIMessageChunk objects as the model generates, so a UI can render tokens as they arrive. Adding chunks with + reassembles the complete message, including tool calls and usage. A real model emits multi-token chunks; the fake emits words and spaces. Streaming through agents and graphs is a separate mechanism, covered in Streaming. There is also astream_events() for semantic events.
batch runs a list of independent inputs concurrently and returns results in input order. Set max_concurrency in the RunnableConfig to stay within provider limits; batch_as_completed() yields each result as soon as it is ready (with its input index) when order does not matter.

See LangChain — Models (the Invocation section covers invoke, stream and batch).

Rate limiting

Hosted providers cap requests per minute, and a batch over a large corpus reaches the cap quickly. Chat models accept a rate_limiter argument; LangChain ships InMemoryRateLimiter, a thread-safe token bucket shared by every call that uses the same instance in one process.

Throttling requests with InMemoryRateLimiter

import time

from langchain.rate_limiters import InMemoryRateLimiter
from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from langchain_core.messages import AIMessage

limiter = InMemoryRateLimiter(
    requests_per_second=5,        # steady rate
    check_every_n_seconds=0.01,   # how often a waiting call re-checks the bucket
    max_bucket_size=1,            # burst size
)
model = GenericFakeChatModel(messages=iter([AIMessage("x")] * 3), rate_limiter=limiter)

start = time.monotonic()
for _ in range(3):
    model.invoke("hi")
print(f"3 calls took {time.monotonic() - start:.2f}s")

The executed run printed 3 calls took 0.62s: with a burst of one and five requests per second, the second and third calls each wait roughly 0.2 s. With a real model you pass the same object as init_chat_model("…", model_provider="…", rate_limiter=limiter).

The limiter counts requests, not tokens, so it does not help with a tokens-per-minute quota; it is also per process, so several workers each get their own budget. For fleet-wide limits, put a gateway in front of the models (see LLM Gateways). See LangChain — Models ("Rate limiting").

Fallbacks

Providers time out, rate-limit and go down. Every Runnable has with_fallbacks, which returns a wrapper that tries the original first and then each fallback in order until one succeeds. A typical DocsAssistant policy is a local model first and a cheap hosted model second, or the reverse for quality-critical questions.

A primary that fails and a backup that answers

from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from langchain_core.messages import AIMessage


class Down(GenericFakeChatModel):
    def _generate(self, *args, **kwargs):
        raise TimeoutError("primary down")


primary = Down(messages=iter([]))
backup = GenericFakeChatModel(messages=iter([AIMessage("answered by the fallback model")]))

robust = primary.with_fallbacks([backup])
print(robust.invoke("hello").content)

Output (executed): answered by the fallback model. The signature is with_fallbacks(fallbacks, *, exceptions_to_handle=(Exception,), exception_key=None), so restrict exceptions_to_handle to the errors that justify switching (timeouts, rate limits) rather than swallowing every exception, including programming errors.

The wrapper is a plain Runnable, not a chat model: it has no bind_tools. Bind tools to the primary and to each fallback first, then wrap. Inside an agent, use the ModelFallbackMiddleware instead, which takes the fallback models in order:

Fallback models inside an agent

from langchain.agents import create_agent
from langchain.agents.middleware import ModelFallbackMiddleware

agent = create_agent(
    primary,            # any chat model or model string
    tools=[],
    middleware=[ModelFallbackMiddleware(backup)],
)
print(agent.invoke({"messages": [("user", "hi")]})["messages"][-1].content)

Executed with fresh instances of the two fakes above (a scripted fake is used up by one call, so reusing the backup that already answered would fail); the last message was the fallback’s reply. Middleware is covered in Middleware (which also has retry middleware for model and tool calls). See Runnable.with_fallbacks and the built-in middleware page, LangChain — Built-in middleware.

Token usage metadata

Most providers report how many tokens a call consumed, and LangChain normalises it into AIMessage.usage_metadata: input_tokens, output_tokens, total_tokens, and optional input_token_details / output_token_details (cached, audio, reasoning tokens). Provider-specific extras stay in response_metadata. Tokens are the unit of cost and latency (see Cost, Latency & Throughput and Tokens & Tokenization), so it is worth recording them on every call. To total usage across several calls and models, use a callback handler or its context-manager form; the result is keyed by model name.

Reading and aggregating usage metadata

from langchain_core.callbacks import UsageMetadataCallbackHandler, get_usage_metadata_callback
from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from langchain_core.messages import AIMessage

usage = {"input_tokens": 8, "output_tokens": 21, "total_tokens": 29}


def fake(name):
    return GenericFakeChatModel(messages=iter([AIMessage(
        "hi", usage_metadata=usage, response_metadata={"model_name": name})]))


print(fake("llama3.1:8b").invoke("Hello").usage_metadata)

with get_usage_metadata_callback() as cb:            # context manager
    fake("llama3.1:8b").invoke("Hello")
    fake("nomic-fake").invoke("Hello")
    print(cb.usage_metadata)

handler = UsageMetadataCallbackHandler()               # explicit handler
fake("llama3.1:8b").invoke("Hello", config={"callbacks": [handler]})
print(handler.usage_metadata)

Output (executed):

{'input_tokens': 8, 'output_tokens': 21, 'total_tokens': 29}
{'llama3.1:8b': {'input_tokens': 8, 'output_tokens': 21, 'total_tokens': 29}, 'nomic-fake': {'input_tokens': 8, 'output_tokens': 21, 'total_tokens': 29}}
{'llama3.1:8b': {'input_tokens': 8, 'output_tokens': 21, 'total_tokens': 29}}

The fake models carry no usage of their own, so the numbers above were attached by hand; real integrations fill the field in. Some providers only report usage in streaming mode if you opt in, and with the OpenAI integration streamed usage is switched off by default once base_url points at another server (stream_usage=True turns it on); check the integration page. See LangChain — Messages ("Token usage") and the "Token usage" section of LangChain — Models. For per-request cost dashboards and traces, see Observability with LangSmith.

DocsAssistant chat

The pieces above combine into the chat layer of DocsAssistant. It answers from documentation excerpts (the corpus and pgvector store are built in Vector Databases & RAG; wiring retrieval into LangChain is in Retrieval & RAG Bridge). The chat layer itself is just: a system message, the running history, a local primary model with a rate limit, and a hosted fallback.

docs_chat.py: local model first, hosted fallback, rate limited

from langchain.chat_models import init_chat_model
from langchain.messages import HumanMessage, SystemMessage
from langchain.rate_limiters import InMemoryRateLimiter

SYSTEM = SystemMessage(
    "You are DocsAssistant. Answer from the documentation excerpts you are given; "
    "say so when they do not contain the answer."
)


def build_model(local=None):
    """Primary: local Ollama. Fallback: a hosted model, used only if Ollama fails."""
    local = local or init_chat_model(
        "llama3.1:8b",
        model_provider="ollama",
        temperature=0,
        rate_limiter=InMemoryRateLimiter(requests_per_second=2, max_bucket_size=4),
    )
    hosted = init_chat_model("claude-haiku-4-5", model_provider="anthropic", temperature=0)
    return local.with_fallbacks([hosted])


def ask(model, history, question, excerpts):
    history = [*history, HumanMessage(f"Excerpts:\n{excerpts}\n\nQuestion: {question}")]
    reply = model.invoke([SYSTEM, *history])
    return [*history, reply], reply

Executed with build_model(fake), where fake was a GenericFakeChatModel scripted to answer "pgvector performs an exact scan until you create an HNSW or IVFFlat index." with usage attached, the call inside get_usage_metadata_callback() printed that sentence, the usage {'llama3.1:8b': {'input_tokens': 96, 'output_tokens': 18, 'total_tokens': 114}}, and a history of length 2 (the human message and the reply). The Anthropic key came from the ANTHROPIC_API_KEY environment variable; nothing is hard-coded, and the hosted fallback is only contacted if the local call raises.

Keeping the history as a list of message objects is what later pages build on: prompts are assembled from templates in Prompts & Templates, the reply is constrained to a schema in Structured Output, and the list becomes graph state in Short-Term Memory & Checkpointers. See the LangChain — Models and LangChain — Messages pages for the parameters used here.

Chat Models & Messages

Creating a model with init_chat_model

Provider packages

Messages

Provider-agnostic content_blocks

invoke, batch and stream

Rate limiting

Fallbacks

Token usage metadata

DocsAssistant chat

References

Creating a model with `init_chat_model`

Provider-agnostic `content_blocks`

`invoke`, `batch` and `stream`