Ollama

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page was written against Ollama 0.34.4, Python 3.12 with openai 3.20.0, anthropic 1.9.0 and langchain-ollama 1.1.0, and Spring Boot 4.1 with Spring AI 2.0.1, as checked on 2026-09-28 (see the Running LLMs Locally baseline). The Ollama documentation at docs.ollama.com was read from its source in the ollama/ollama repository on that date. Examples use DocsAssistant's llama3.1:8b chat model and nomic-embed-text embeddings, the same local stack as Vector Databases & RAG.

Ollama is the easiest way to run open-weight models on a laptop or workstation: one binary (or app) that downloads GGUF models, runs them on llama.cpp-derived backends with automatic GPU offload, and serves them on http://localhost:11434 behind its own REST API plus OpenAI- and Anthropic-compatible endpoints.

Install

# Linux (installs the ollama service)
curl -fsSL https://ollama.com/install.sh | sh

# Docker, CPU only
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

# Docker, NVIDIA GPU (requires the NVIDIA Container Toolkit)
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

On macOS (Sonoma or newer), mount ollama.dmg and drag the app into Applications; the app offers to link the ollama CLI into /usr/local/bin. On Windows, run OllamaSetup.exe. Both run the server in the background. See Linux, macOS, Windows and Docker.

Ollama binds 127.0.0.1:11434 by default and does not validate API keys on the local server. The Docker commands above publish the port on all host interfaces, so anyone who can reach the host can use the model. Publish it on loopback only (-p 127.0.0.1:11434:11434) or put an authenticating reverse proxy in front — see authentication in front of local servers. The same applies to setting OLLAMA_HOST=0.0.0.0:11434 to expose it on the network (Ollama — FAQ).

Pull, run, list and ps

ollama pull llama3.1:8b          # download a model from the Ollama library
ollama pull nomic-embed-text     # DocsAssistant's embedding model
ollama run llama3.1:8b "Summarise what a vector database is in two sentences."
ollama list                      # models on disk
ollama ps                        # models loaded in memory: size, GPU/CPU split, context, unload time
ollama stop llama3.1:8b          # unload now
ollama rm llama3.1:8b            # delete from disk

ollama ps is the command to remember: its PROCESSOR column tells you whether the model fitted on the GPU (100% GPU) or was split with the CPU (48%/52% CPU/GPU), and its CONTEXT column shows the context actually allocated. See Ollama — CLI reference.

Running Hub GGUFs

Any GGUF repository on the Hugging Face Hub runs directly, optionally with a quant tag:

ollama run hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF
ollama run hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M

This replaces the book-era route of downloading a GGUF file by hand into a desktop app. See Use Ollama with any GGUF model on the Hugging Face Hub; the quant tags are explained on Quantization for Inference.

The Modelfile

A Modelfile derives a new model from an existing one: its base (FROM), default parameters (PARAMETER), the chat template (TEMPLATE, Go template syntax) and a system prompt (SYSTEM). A DocsAssistant variant with a larger context and a fixed persona:

FROM llama3.1:8b
PARAMETER num_ctx 16384
PARAMETER temperature 0.2
SYSTEM """You are DocsAssistant. Answer only from the documentation excerpts you are given, and cite the page."""

ollama create docsassistant -f Modelfile
ollama run docsassistant

TEMPLATE is normally inherited from the base model’s GGUF metadata; override it only for a model imported without a correct template. FROM can also point at a local .gguf file or a Safetensors directory. See Ollama — Modelfile.

Context length and the silent-truncation pitfall

Ollama allocates a context window smaller than the model supports, to save memory. Its documentation gives the defaults: 4k tokens on machines with less than 24 GiB of VRAM, 32k with 24-48 GiB and 256k with 48 GiB or more. A prompt longer than the allocated window is truncated without an error — the oldest tokens are dropped, so a RAG prompt quietly loses its retrieved context or its instructions. This is the failure mode documented for DocsAssistant on Prompt Augmentation & Generation. Ollama recommends at least 64000 tokens for agents, coding tools and web search.

There are four ways to raise it, from most to least global:

# 1. server-wide default
OLLAMA_CONTEXT_LENGTH=32768 ollama serve

# 2. per model, baked into a Modelfile
#    PARAMETER num_ctx 32768

# 3. per request on the native API
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{"role": "user", "content": "Hello"}],
  "options": {"num_ctx": 32768}
}'

# 4. interactively in `ollama run`
#    /set parameter num_ctx 32768

The OpenAI-compatible endpoint has no field for the context size, so for OpenAI-style clients use the server-wide variable or a Modelfile-derived model. Verify with ollama ps, whose CONTEXT column shows what was allocated, and remember that the KV cache grows with the context (see Hardware Sizing). See Ollama — Context length.

The native REST API

The native API lives under /api: /api/chat and /api/generate for inference, /api/embed for embeddings, /api/tags (list), /api/show, /api/pull and /api/ps for management. Responses stream as newline-delimited JSON unless "stream": false:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{"role": "user", "content": "What is a KV cache?"}],
  "stream": false
}'

The answer is in message.content. See Ollama — API.

OpenAI compatibility

http://localhost:11434/v1 implements /v1/chat/completions, /v1/completions, /v1/responses, /v1/models and /v1/embeddings, so any OpenAI SDK or tool works by changing its base URL. An API key is required by the SDKs but ignored by Ollama:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.1:8b", "messages": [{"role": "user", "content": "Hello!"}]}'

See Ollama — OpenAI compatibility.

Anthropic compatibility

POST /v1/messages implements a subset of the Anthropic Messages API: messages, streaming, system prompts, multi-turn conversations, base64 images, tools and tool results, and thinking. This is what makes Claude Code and other Anthropic-API tools usable against a local model (see Local Models in Claude Code).

curl http://localhost:11434/v1/messages \
  -H "Content-Type: application/json" \
  -H "x-api-key: ollama" \
  -H "anthropic-version: 2023-06-01" \
  -d '{"model": "llama3.1:8b", "max_tokens": 512,
       "messages": [{"role": "user", "content": "Hello, how are you?"}]}'

import anthropic

client = anthropic.Anthropic(base_url="http://localhost:11434", api_key="ollama")  # key required but ignored
message = client.messages.create(
    model="llama3.1:8b",
    max_tokens=512,
    messages=[{"role": "user", "content": "Hello, how are you?"}],
)
print(message.content[0].text)

Not supported, per Ollama’s compatibility page: the /v1/messages/count_tokens endpoint, tool_choice, metadata, prompt caching (cache_control), the Batches API, citations, PDF document blocks and server-sent error events. Images must be base64 (no URLs), budget_tokens for extended thinking is accepted but not enforced, and token counts are approximations. Tools that rely on default Anthropic model names can be served by copying a model under that name (ollama cp <local-model> <anthropic-style-name>). See Ollama — Anthropic compatibility.

Embeddings

curl http://localhost:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": ["What is HNSW?", "How does pgvector index vectors?"]
}'

/api/embed accepts one string or a list and returns L2-normalised vectors (768 dimensions for nomic-embed-text); /v1/embeddings offers the same through the OpenAI shape. See Ollama — Embeddings and, for how DocsAssistant uses them, Generating Embeddings in Practice.

Structured outputs

format takes either "json" or a JSON Schema, and generation is constrained to it:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{"role": "user", "content": "Classify: the Redis page has a broken link."}],
  "stream": false,
  "format": {
    "type": "object",
    "properties": {"category": {"type": "string", "enum": ["bug", "question", "praise"]},
                   "page": {"type": "string"}},
    "required": ["category", "page"]
  }
}'

See Ollama — Structured outputs.

Tools

Tool definitions use the OpenAI function schema, on the native API or on /v1/chat/completions; the model returns message.tool_calls, you run the tool and send the result back as a tool message:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{"role": "user", "content": "Search the docs for pgvector index types."}],
  "stream": false,
  "tools": [{
    "type": "function",
    "function": {
      "name": "search_docs",
      "description": "Search the documentation site",
      "parameters": {"type": "object", "required": ["query"],
                     "properties": {"query": {"type": "string"}}}
    }
  }]
}'

Only models whose template supports tools can call them; the library marks them with a tools tag. See Ollama — Tool calling.

GPU selection

Ollama detects NVIDIA (CUDA), AMD (ROCm) and, on Apple silicon, Metal automatically. To restrict it to some GPUs, set the backend’s standard variable on the server:

CUDA_VISIBLE_DEVICES=0,1 ollama serve        # NVIDIA
ROCR_VISIBLE_DEVICES=0 ollama serve          # AMD ROCm
GGML_VK_VISIBLE_DEVICES=1 ollama serve       # Vulkan

A model that does not fit one GPU is split across the visible GPUs, then onto the CPU. Flash Attention is used automatically where supported (OLLAMA_FLASH_ATTENTION=1 forces it), and the KV cache can be quantized with OLLAMA_KV_CACHE_TYPE. See Ollama — GPU and FAQ.

Cloud models

A signed-in Ollama server can also run cloud models — larger models hosted by Ollama — under a :cloud tag, through the same local API. This makes Ollama a small gateway between local and hosted models (see LLM Gateways):

ollama signin
ollama run gemma4:cloud

Prompts sent to a :cloud model leave your machine, which defeats the privacy reason for running locally. To guarantee local-only operation, disable the cloud features with OLLAMA_NO_CLOUD=1 (or "disable_ollama_cloud": true in ~/.ollama/server.json). See Ollama — Cloud and FAQ.

Keeping models loaded

Ollama unloads an idle model after five minutes. The first request after that pays the load time again, which looks like a very slow first token. Control it per request with keep_alive or server-wide with OLLAMA_KEEP_ALIVE:

curl http://localhost:11434/api/generate -d '{"model": "llama3.1:8b", "keep_alive": -1}'   # keep loaded
curl http://localhost:11434/api/generate -d '{"model": "llama3.1:8b", "keep_alive": 0}'    # unload now
OLLAMA_KEEP_ALIVE=1h ollama serve

Related server settings: OLLAMA_NUM_PARALLEL (parallel requests per model; memory scales with it times the context) and OLLAMA_MAX_LOADED_MODELS. See Ollama — FAQ.

Editors

Ollama integrates with the coding tools covered in this sub-section. In VS Code, the built-in Ollama provider is deprecated in favour of the official Ollama extension — see Local Models in GitHub Copilot. For Claude Code, see Local Models in Claude Code; for Continue, Cline, Aider and OpenCode, see Open Coding Assistants with Local Models.

Client snippets

The same three clients appear on every engine page in this sub-section; only the base URL and model name change.

curl against the OpenAI-compatible endpoint:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.1:8b",
       "messages": [{"role": "user", "content": "What is retrieval-augmented generation?"}]}'

Python — the openai SDK with base_url, then LangChain, either with the native ChatOllama (which can set num_ctx per request) or with ChatOpenAI pointed at the same endpoint:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # key required by the SDK, ignored
reply = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "What is retrieval-augmented generation?"}],
)
print(reply.choices[0].message.content)

from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain_openai import ChatOpenAI

chat = ChatOllama(model="llama3.1:8b", num_ctx=16384, temperature=0.2)
print(chat.invoke("What is retrieval-augmented generation?").content)

embeddings = OllamaEmbeddings(model="nomic-embed-text")
print(len(embeddings.embed_query("HNSW index")))  # 768

# the same model through the OpenAI-compatible endpoint
chat_openai = ChatOpenAI(base_url="http://localhost:11434/v1", api_key="ollama", model="llama3.1:8b")

Spring AI — the Ollama starter, which uses the native API and so can set num-ctx:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-ollama</artifactId>
</dependency>

spring:
  ai:
    ollama:
      base-url: http://localhost:11434
      chat:
        model: llama3.1:8b
        num-ctx: 16384
      embedding:
        model: nomic-embed-text

package com.example.docsassistant;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.stereotype.Service;

@Service
public class LocalAnswerService {

    private final ChatClient chatClient;

    public LocalAnswerService(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    public String answer(String question) {
        return chatClient.prompt().user(question).call().content();
    }
}

See openai-python, LangChain — ChatOllama and Spring AI — Ollama Chat. A complete Spring AI application on this stack, with Redis as the vector store, is on Spring Boot: Redis as a Vector Database.