Local Models

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page was written against Python 3.12 with langchain 1.4.3, langchain-core 1.6.5, langchain-ollama 1.1.0, langchain-openai 1.6.6 and langchain-huggingface 1.2.2 (with the ollama client 0.6.3 and openai 3.20.0 that they install), all checked on 2026-09-29. No Ollama, vLLM or LM Studio server was available while writing it, so every model call below was executed against a small stub HTTP server that speaks Ollama’s /api/chat, /api/embed, /api/show and OpenAI’s /v1/chat/completions, /v1/embeddings on localhost. The stub replies with canned text and a canned tool call: the output shown proves that LangChain builds the right requests and parses the responses, not that any model answers well. The real models are the AI section’s baseline — llama3.1:8b and nomic-embed-text on Ollama for DocsAssistant. The Hugging Face classes were only constructed (with a placeholder token in the environment), never called.

Running a model locally changes three things for a LangChain application, and none of them is the API: the model is smaller and follows tool schemas less reliably, the server enforces limits (context length, tool support) that hosted providers hide, and there is usually no bill. Choosing hardware, quantization and servers is the job of Running LLMs Locally; this page is about the LangChain side of the connection — which class talks to which server, what to configure, and how to find out early that a model cannot do what the agent needs.

flowchart LR DA["DocsAssistant
chat_model() / embeddings()"] -->|LLM_BACKEND=ollama| CO["ChatOllama
OllamaEmbeddings"] DA -->|LLM_BACKEND=openai| CP["ChatOpenAI(base_url=...)
OpenAIEmbeddings(base_url=...)"] CO --> OL["Ollama
:11434 /api/chat"] CP --> VL["vLLM :8000/v1"] CP --> LM["LM Studio :1234/v1"] CP --> DM["Docker Model Runner
:12434/engines/v1"] HF["ChatHuggingFace"] --> EP["HuggingFaceEndpoint
(Inference Providers)"] HF --> PL["HuggingFacePipeline
(in-process transformers)"]

Figure 1. Which LangChain class reaches which local server

ChatOllama and OllamaEmbeddings

Ollama is the default local backend for DocsAssistant. Its LangChain classes live in the langchain-ollama package (pip install -U langchain-ollama), which talks to the server’s native REST API rather than its OpenAI-compatible one — the native API is what exposes Ollama-specific options such as num_ctx and keep_alive. Pull the model first and use the same tag in model=; the server side is covered in Ollama.

ChatOllama and OllamaEmbeddings against a server (here the stub)

from langchain_ollama import ChatOllama, OllamaEmbeddings

llm = ChatOllama(
    model="llama3.1:8b",
    base_url="http://localhost:11434",   # the default; stated here to show where it is set
    num_ctx=16384,                       # context window requested per call
    temperature=0,
)
reply = llm.invoke("What is HNSW?")
print(reply.content)
print(reply.usage_metadata)

embeddings = OllamaEmbeddings(model="nomic-embed-text", base_url="http://localhost:11434")
print(len(embeddings.embed_query("hnsw")))
print(len(embeddings.embed_documents(["a", "b"])))

Output (against the stub; the num_ctx value was checked in the request the stub received)

HNSW builds a layered graph.
{'input_tokens': 12, 'output_tokens': 7, 'total_tokens': 19}
768
2

ChatOllama returns a normal AIMessage with usage_metadata filled from the server’s token counts, so the accounting from Chat Models & Messages works unchanged. OllamaEmbeddings implements the Embeddings interface (embed_query, embed_documents), which is what the pgvector store from PostgreSQL & pgvector and the retrievers on Retrieval & the RAG Bridge take. The two parameters worth setting deliberately:

num_ctx — Ollama loads models with a small default context and silently truncates longer prompts. Agents accumulate tool results quickly, so set it explicitly and see Ollama’s context-length pitfall.
validate_model_on_init=True — checks at construction that the model has been pulled, so a missing model fails at start-up instead of on the first user request.

Failing fast on a model that was never pulled

from langchain_ollama import ChatOllama

try:
    ChatOllama(model="llama3.1:70b", base_url="http://localhost:11434", validate_model_on_init=True)
except Exception as e:
    print(type(e).__name__, str(e)[:110])

ChatOllama(model="llama3.1:8b", base_url="http://localhost:11434", validate_model_on_init=True)
print("ok")

Output (stub server that only knows llama3.1:8b)

ValidationError 1 validation error for ChatOllama
  Value error, Model `llama3.1:70b` not found in Ollama. Please pull the mod
ok

Embedding models are tied to the vectors already stored: nomic-embed-text produces 768 dimensions, the size of the vector(768) column in DocsAssistant’s store. Swapping in another embedding model, locally or hosted, means re-embedding the corpus. The EMBEDDING_DIMENSIONS check in the DocsAssistant module below turns that into a start-up check. The integration pages are ChatOllama and OllamaEmbeddings; init_chat_model reaches the same class with model_provider="ollama".

OpenAI-compatible servers with `base_url`

vLLM, LM Studio and Docker Model Runner all serve the OpenAI Chat Completions shape under a /v1-style prefix, so one class — ChatOpenAI from langchain-openai — reaches all of them by changing base_url. Only the URL and the model name differ:

Table 1. OpenAI-compatible local servers
Server	`base_url`	Notes
vLLM	`http://localhost:8000/v1`	`--api-key` / `VLLM_API_KEY` on the server makes it require a key. Tool calling needs `--enable-auto-tool-choice --tool-call-parser` with the parser for the model family. See vLLM & SGLang.
LM Studio	`http://localhost:1234/v1`	Authentication is off by default; enable Require Authentication before binding beyond loopback. See Desktop Runners.
Docker Model Runner	`http://localhost:12434/engines/v1`	Host TCP access is opt-in on Docker Desktop and the API is unauthenticated by design. See Desktop Runners.

ChatOpenAI pointed at a local server, with a tool bound

import os

from langchain.tools import tool
from langchain_openai import ChatOpenAI


@tool
def docs_search(query: str) -> str:
    """Search the docs."""
    return "modules/ROOT/pages/database/vector-rag/ann-indexes.adoc"


llm = ChatOpenAI(
    model="meta-llama/Llama-3.1-8B-Instruct",       # the name the server reports, not an OpenAI model
    base_url="http://localhost:8000/v1",
    api_key=os.environ.get("LLM_API_KEY", "not-needed"),   # the SDK insists on a non-empty key
)
print(llm.invoke("hi").content)
print(llm.bind_tools([docs_search]).invoke("find hnsw").tool_calls)

Output (against the stub; it also showed the request carried Authorization: Bearer not-needed)

HNSW builds a layered graph.
[{'name': 'docs_search', 'args': {'query': 'hnsw index'}, 'id': 'call_1', 'type': 'tool_call'}]

Without api_key (or OPENAI_API_KEY) the client refuses to start with a "Missing credentials" error even when the server needs none, hence the placeholder. The base URL can also come from the environment: LangChain resolves an explicit base_url first, then OPENAI_API_BASE, then OPENAI_BASE_URL, which the openai SDK reads. init_chat_model("meta-llama/Llama-3.1-8B-Instruct", model_provider="openai", base_url=…, api_key=…) builds the same object.

Two limits of this route come straight from the LangChain documentation. ChatOpenAI targets the official OpenAI specification only: non-standard response fields that servers add, such as reasoning_content, are neither extracted nor preserved, so a reasoning model’s thinking trace disappears when it is served through vLLM and read with ChatOpenAI. And features such as tool calling, multimodal input and token streaming "depend on the hosted model", as the vLLM integration page puts it. Where a dedicated provider package exists — langchain-ollama for Ollama — prefer it over the compatibility endpoint.

Embeddings need one more flag. OpenAIEmbeddings by default tokenizes each text with tiktoken and sends token IDs, which a local embedding server that expects strings will misread:

Why check_embedding_ctx_length=False matters for local servers

from langchain_openai import OpenAIEmbeddings

base = "http://localhost:8000/v1"
OpenAIEmbeddings(model="nomic-embed-text", base_url=base, api_key="x").embed_query("hnsw")
# request body input: [[25105, 2332]]   (token ids)

OpenAIEmbeddings(model="nomic-embed-text", base_url=base, api_key="x",
                 check_embedding_ctx_length=False).embed_query("hnsw")
# request body input: ['hnsw']          (plain text)

The two comments are what the stub recorded as the request body for each call. See the ChatOpenAI, vLLM and Chat Completions API compatibility pages, and LLM Gateways for putting one authenticated endpoint in front of several backends.

One DocsAssistant, several backends

DocsAssistant should not care which server answers. Two functions read the environment and return a chat model and an embedding model; everything else in the application (tools, agent, retriever) receives them as arguments.

docsassistant/models.py: backend chosen by environment variables, no secrets in code

import os

from langchain_core.embeddings import Embeddings
from langchain_core.language_models import BaseChatModel
from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

EMBEDDING_DIMENSIONS = 768  # the vector(768) column of the pgvector store


def chat_model() -> BaseChatModel:
    if os.environ.get("LLM_BACKEND", "ollama") == "ollama":
        return ChatOllama(
            model=os.environ.get("LLM_MODEL", "llama3.1:8b"),
            base_url=os.environ.get("OLLAMA_BASE_URL", "http://localhost:11434"),
            num_ctx=16384,      # Ollama's default context is much smaller
            temperature=0,
            validate_model_on_init=True,
        )
    # vLLM, LM Studio, Docker Model Runner ...: any OpenAI-compatible endpoint
    return ChatOpenAI(
        model=os.environ["LLM_MODEL"],
        base_url=os.environ["LLM_BASE_URL"],
        api_key=os.environ.get("LLM_API_KEY", "not-needed"),
        temperature=0,
    )


def embeddings() -> Embeddings:
    if os.environ.get("LLM_BACKEND", "ollama") == "ollama":
        emb: Embeddings = OllamaEmbeddings(
            model="nomic-embed-text",
            base_url=os.environ.get("OLLAMA_BASE_URL", "http://localhost:11434"),
        )
    else:
        emb = OpenAIEmbeddings(
            model=os.environ.get("EMBEDDING_MODEL", "nomic-embed-text"),
            base_url=os.environ["LLM_BASE_URL"],
            api_key=os.environ.get("LLM_API_KEY", "not-needed"),
            check_embedding_ctx_length=False,   # send text, not tiktoken ids
        )
    size = len(emb.embed_query("dimension probe"))
    if size != EMBEDDING_DIMENSIONS:
        raise RuntimeError(f"embedding model returns {size} dimensions, the store expects {EMBEDDING_DIMENSIONS}")
    return emb

The module was run in both modes against the stub (OLLAMA_BASE_URL set for the first, LLM_BACKEND=openai, LLM_MODEL and LLM_BASE_URL for the second) and returned a ChatOllama with num_ctx=16384 and a ChatOpenAI whose openai_api_base was the given URL; in both cases embeddings() returned 768-dimension vectors and passed the check. The dimension probe costs one embedding call at start-up and turns the silent failure of a mismatched embedding model (the store would reject inserts, or worse, retrieval would quietly degrade) into an immediate error.

Hugging Face models with `ChatHuggingFace`

langchain-huggingface wraps a Hugging Face LLM object in a chat model. ChatHuggingFace takes an llm= that is either a HuggingFaceEndpoint, which calls a hosted model through Hugging Face’s Inference Providers, or a HuggingFacePipeline, which loads a transformers model into the Python process itself. The endpoint variant is not local in the sense of this sub-section — the weights run on Hugging Face’s or a provider’s infrastructure — but it is how open-weight models are reached without installing anything else. The Hugging Face ecosystem itself (the Hub, transformers, Inference Providers) has a planned sub-section of its own; here it is only a LangChain provider.

ChatHuggingFace over an endpoint; the token comes from the environment

import os

# HUGGINGFACEHUB_API_TOKEN must be set in the environment (a placeholder was used to construct this)
from langchain_huggingface import ChatHuggingFace, HuggingFaceEndpoint

llm = HuggingFaceEndpoint(
    repo_id="meta-llama/Llama-3.1-8B-Instruct",
    task="text-generation",
    max_new_tokens=512,
    provider="auto",       # let Hugging Face choose an inference provider
)
chat = ChatHuggingFace(llm=llm)
print(type(chat).__name__, chat.model_id)
print(hasattr(chat, "bind_tools"))

Output (constructed only, nothing was sent)

ChatHuggingFace meta-llama/Llama-3.1-8B-Instruct
True

For in-process inference, HuggingFacePipeline.from_model_id(model_id=…, task="text-generation", pipeline_kwargs=dict(max_new_tokens=512)) builds the wrapped LLM instead, and a bitsandbytes quantization config can be passed through model_kwargs. It needs transformers (plus torch, and bitsandbytes/accelerate for quantization), which were not installed in the environment used here, so that variant is taken from the ChatHuggingFace integration page and was not run. Loading a multi-gigabyte model into the application process ties its memory, start-up time and GPU to the web worker; for anything but experiments, serve the model with a real server from Running LLMs Locally and use one of the classes above. The integration page’s own model table lists ChatHuggingFace without token-level streaming, unlike ChatOllama.

Tool-calling caveats for local models

An agent built with create_agent (see Agents with create_agent) needs the model to emit structured tool calls. That is a property of the model and its chat template, not of LangChain, and local models are where it goes wrong. The concepts are in Structured Output & Tool Calling and Tool Design; these are the local-specific points:

Not every model can call tools. Ollama reports each model’s features; only models whose template supports tools accept a tool list, and the library marks them with a tools tag. Sending tools to one that does not fails with an HTTP 400.
Servers may need to be told how to parse tool calls. vLLM needs --enable-auto-tool-choice and a --tool-call-parser matching the model family; without them the server does not turn the model’s output into the OpenAI-style tool_calls a client expects.
Small models are unreliable callers. An 8B model given ten tools picks the wrong one, or malformed arguments, far more often than a frontier model. Keep tool lists short and schemas simple (see Tool Design), and evaluate on your own tasks (Evaluation) instead of trusting a leaderboard.
Context is small by default. Tool results fill a small window fast; raise num_ctx (Ollama) or the server’s context size.
Non-OpenAI fields are dropped. Reasoning traces and other extensions served on a compatible endpoint do not survive ChatOpenAI, as above.

The failure is easy to see and easy to detect ahead of time. With a model that lacks the tools capability the error arrives as an ollama.ResponseError:

What binding a tool to a model without tool support looks like (stub replying as Ollama does)

from ollama import ResponseError
from langchain_ollama import ChatOllama

small = ChatOllama(model="gemma2:2b", base_url="http://localhost:11434")
try:
    small.bind_tools([docs_search]).invoke("find hnsw")
except ResponseError as e:
    print(type(e).__name__, e.status_code, e.error)

Output

ResponseError 400 registry.ollama.ai/library/gemma2:2b does not support tools

The stub was scripted to return that message for the gemma2 name; the text a real server sends is Ollama’s, and the shape (a 400 with an error string) is what to handle (see Ollama — Tool calling). To find out before the first request, ask the server what the model can do — Ollama’s show-model-details endpoint returns a capabilities list, and tools is the entry for tool support:

Probing tool support at start-up

import ollama


def supports_tools(model: str, host: str = "http://localhost:11434") -> bool:
    """True when Ollama lists `tools` among the model's capabilities."""
    return "tools" in (ollama.Client(host=host).show(model).capabilities or [])

Against the stub, supports_tools("llama3.1:8b", …) returned True and supports_tools("gemma2:2b", …) returned False. The capability names are the ones defined in Ollama’s source (completion, tools, vision, thinking, and others); the API reference documents the capabilities field.

When a model can not call tools but must still produce structured data, use structured output instead of tool calling. ChatOllama.with_structured_output defaults to method="json_schema", which sends the schema in Ollama’s format field (constrained generation) and no tools at all:

Structured output on a model without tool support

from pydantic import BaseModel
from langchain_ollama import ChatOllama


class Route(BaseModel):
    page: str
    confidence: float


small = ChatOllama(model="gemma2:2b", base_url="http://localhost:11434")
print(small.with_structured_output(Route).invoke("Which page covers HNSW?"))

Output

page='vector-rag/ann-indexes' confidence=0.9

The request the stub received had an empty tools list and the schema’s properties (page, confidence) in format. The alternative methods are "function_calling" (which uses the tool API and so needs a capable model) and "json_mode" (valid JSON but no schema enforcement). Strategies for agents that produce structured final answers, including ToolStrategy, are on Structured Output; the ChatOllama integration page lists the structured-output support of the class.

Finally, a capable local model plugs into create_agent exactly like a hosted one. Both backends, run against the stub with a docs_search tool, produced the same message sequence:

The same agent on ChatOllama and on ChatOpenAI(base_url=…)

from langchain.agents import create_agent

for llm in (ChatOllama(model="llama3.1:8b", base_url=OLLAMA_URL),
            ChatOpenAI(model="llama3.1:8b", base_url=OPENAI_URL, api_key="not-needed")):
    agent = create_agent(llm, tools=[docs_search])
    out = agent.invoke({"messages": [("user", "Which page covers HNSW?")]})
    print(type(llm).__name__, [type(m).__name__ for m in out["messages"]])

Output

ChatOllama ['HumanMessage', 'AIMessage', 'ToolMessage', 'AIMessage']
ChatOpenAI ['HumanMessage', 'AIMessage', 'ToolMessage', 'AIMessage']

(OLLAMA_URL and OPENAI_URL stand for the two base URLs; the stub replied with a tool call to the first request and with plain text once a tool message was present. See LangChain — Agents.) A model that cannot do this reliably is a signal to route the tool-using steps to a hosted model with ModelFallbackMiddleware or a model-selection middleware, as described on Middleware, while keeping embeddings and simple chat local. Data-sensitivity rules should decide that routing, not convenience — see Guardrails & Security and Agent Guardrails & Safety.

Where local models fit

Local models earn their place in the DocsAssistant stack for embeddings (cheap, high-volume, private), for development and tests (no network, no bill), and for privacy-bound workloads. The AI section’s other pages cover the surrounding decisions: Open-Weight Models for which weights to run, Hardware Sizing for what they need, and Running Local LLMs in Production for operating them. Tracing a local model with LangSmith works like any other provider (Observability with LangSmith); the same MCP tools attach to it too (MCP Adapters).

Local Models

ChatOllama and OllamaEmbeddings

OpenAI-compatible servers with base_url

One DocsAssistant, several backends

Hugging Face models with ChatHuggingFace

Tool-calling caveats for local models

Where local models fit

References

OpenAI-compatible servers with `base_url`

Hugging Face models with `ChatHuggingFace`