llama.cpp

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page was written against llama.cpp v0.5.0 — the project now publishes semantic-version releases alongside its long-running bNNNN build tags (b11239 on the same day) — and llama-cpp-python 0.3.35, with Python 3.12 and Spring AI 2.0.1 for the clients, as checked on 2026-09-28 (see the Running LLMs Locally baseline).

llama.cpp is the C/C++ inference engine underneath much of the local-LLM world: Ollama, LM Studio, Jan, Docker Model Runner and llamafile all build on it or its ggml library. Using it directly gives you every flag the others hide — exact GPU offload, context and slot layout, KV-cache types, speculative decoding — in one small server with a built-in web UI.

Installing, building and backends

The fastest route is a package or a container:

brew install llama.cpp                 # macOS / Linux (Homebrew)
winget install llama.cpp               # Windows
docker run -p 127.0.0.1:8080:8080 -v "$PWD/models:/models" ghcr.io/ggml-org/llama.cpp:server \
  -m /models/model.gguf -c 8192 --host 0.0.0.0 --port 8080

Pre-built binaries for each backend are on the releases page, and there are CUDA (:server-cuda, :server-cuda13), ROCm and Vulkan image variants. To build from source, pick the backend with a CMake option:

Backend Hardware CMake option

Metal

Apple silicon

enabled by default on macOS

CUDA

NVIDIA GPUs

-DGGML_CUDA=ON

HIP

AMD GPUs (ROCm)

-DGGML_HIP=ON -DGPU_TARGETS=gfx1100 (your GPU’s target)

Vulkan

most GPUs, including integrated

-DGGML_VULKAN=ON

SYCL

Intel GPUs

-DGGML_SYCL=ON with the oneAPI compilers

CPU

x86 (AVX / AVX2 / AVX-512 / AMX), ARM (NEON)

default

 
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

See llama.cpp — How to build and Install pre-built version.

llama serve or llama-server?

Both are current, and they are the same server:

  • The project’s README now leads with a unified llama binary and sub-commands: llama cli -hf <repo> for a chat session and llama serve -hf <repo> for the OpenAI-compatible server.

  • The server README, the Docker images (ghcr.io/ggml-org/llama.cpp:server "only includes the llama-server executable") and most tutorials still use the stand-alone llama-server binary, and the build still produces ./build/bin/llama-server.

This page uses llama serve; every flag below works identically with llama-server. If your package predates the unified binary, use llama-server. See the llama.cpp README.

Server flags

A DocsAssistant server: the Llama 3.1 8B GGUF at Q4_K_M, fully on the GPU, a 32k context shared by four parallel slots:

llama serve -hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M \
  -c 32768 -np 4 -ngl all --port 8080 --alias llama3.1-8b
Flag Meaning

-m, --model FNAME

Load a local GGUF file.

-hf, --hf-repo <user>/<model>[:quant]

Download from the Hugging Face Hub and cache it; the quant defaults to Q4_K_M. -hff picks an exact file; HF_TOKEN authenticates gated repositories.

-a, --alias NAME

The model id clients send and /v1/models reports (by default the model path).

-c, --ctx-size N

Context size; 0 (the default) uses the size stored in the model.

-ngl, --n-gpu-layers N

Layers to keep in VRAM: a number, auto (the default) or all. Lower it when the model does not fit.

--jinja / --no-jinja

Use the Jinja chat template embedded in the GGUF. Enabled by default in current builds; required for tool calling.

-np, --parallel N

Number of server slots (concurrent sequences); -1 (default) chooses automatically.

-cb / -nocb

Continuous batching across slots (enabled by default).

--host, --port

Bind address (default 127.0.0.1) and port (default 8080).

--api-key KEY, --api-key-file FNAME

Require a bearer key (comma-separated list allowed).

-fa, --flash-attn on|off|auto

Flash Attention (default auto).

-ctk / -ctv

KV-cache types, for example q8_0 (see KV-cache quantization).

--metrics

Expose a Prometheus /metrics endpoint.

Every flag also has an environment variable (LLAMA_ARG_CTX_SIZE, LLAMA_ARG_N_GPU_LAYERS, LLAMA_API_KEY, …​), which is convenient in containers. See the llama.cpp server README.

How the context is shared between slots

With several slots, the KV cache is a single pool shared by all sequences (the unified KV buffer, enabled by default when the slot count is automatic), or a fixed share per slot. Either way, -c is the budget for all concurrent conversations together, so -c 32768 -np 4 does not give each user 32k tokens:

flowchart LR GGUF["GGUF file
(-m / -hf)"] --> W["Weights
-ngl layers on GPU,
the rest on CPU"] W --> KV["KV cache pool
-c tokens in total
(-ctk / -ctv types)"] KV --> S0["slot 0"] KV --> S1["slot 1"] KV --> S2["slot 2"] KV --> S3["slot 3"] S0 & S1 & S2 & S3 --> CB["continuous batching
one forward pass
per step"]

--kv-unified-per-slot N caps each slot, and GET /slots shows each slot’s n_ctx and state. A request whose prompt plus output exceeds its slot’s context is reported with "truncated": true in the /completion response — check it. See the server README.

OpenAI- and Anthropic-compatible endpoints

The server exposes /v1/models, /v1/chat/completions, /v1/completions, /v1/responses and /v1/embeddings in the OpenAI shape, and POST /v1/messages plus /v1/messages/count_tokens in the Anthropic shape. It also has its own endpoints: /completion, /tokenize, /embedding, /reranking, /infill (code completion), /health, /props and /slots, plus a web UI at the root URL.

curl http://localhost:8080/health
curl http://localhost:8080/v1/models

With a single model loaded, /v1/models reports the model path as its id unless --alias sets a friendlier name, which is why the example above passes --alias llama3.1-8b and the clients below send that name. The server can also host several models and route by the model field (see "Using multiple models" in the README). See the server README.

Grammars and JSON schema

llama.cpp constrains generation with GBNF grammars, and converts JSON Schemas to grammars for you. On the OpenAI endpoint, use response_format:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role": "user", "content": "Extract the page and the problem: the Redis page has a broken link."}],
  "response_format": {
    "type": "json_schema",
    "schema": {"type": "object",
               "properties": {"page": {"type": "string"}, "problem": {"type": "string"}},
               "required": ["page", "problem"]}
  }
}'

On the native /completion endpoint, pass json_schema or a raw grammar:

curl http://localhost:8080/completion -H "Content-Type: application/json" -d '{
  "prompt": "Is pgvector a PostgreSQL extension? Answer yes or no: ",
  "grammar": "root ::= \"yes\" | \"no\"",
  "n_predict": 4
}'

See the GBNF grammars README and the server README.

Speculative decoding

A small draft model (or an n-gram lookup over the context) proposes several tokens, and the main model verifies them in one forward pass; accepted tokens come almost for free. The output distribution is unchanged, so quality is identical and only speed varies (Leviathan et al.). llama.cpp supports classic draft models, EAGLE-3 heads and draft-free n-gram methods, selected with --spec-type:

# classic draft model from the same family (it must share the tokenizer)
llama serve -m Qwen3-4B.gguf -md Qwen3-0.6B.gguf --spec-type draft-simple --spec-draft-n-max 8

# EAGLE-3 draft head
llama serve -m Qwen3-4B.gguf -md Qwen3-4B-eagle3.gguf --spec-type draft-eagle3

# no draft model: reuse n-grams already in the context (good for code editing and RAG answers that quote)
llama serve -m Qwen3-4B.gguf --spec-type ngram-simple --spec-draft-n-max 64

The older --draft / --draft-max flags have been removed in favour of --spec-draft-n-max. How much it helps depends on how predictable your outputs are; measure with the timing script on Inference Optimization. See llama.cpp — Speculative decoding.

llama-cpp-python

llama-cpp-python binds the same engine into Python, either in-process or as its own OpenAI-compatible server. Backend options are passed through CMAKE_ARGS at install time:

CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python
python3 -m llama_cpp.server --model ./Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf --n_gpu_layers -1
from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="bartowski/Meta-Llama-3.1-8B-Instruct-GGUF",
    filename="*Q4_K_M.gguf",
    n_ctx=16384,
    n_gpu_layers=-1,
    verbose=False,
)
out = llm.create_chat_completion(messages=[{"role": "user", "content": "What is a GGUF file?"}])
print(out["choices"][0]["message"]["content"])

The in-process API is convenient for notebooks and batch jobs; for anything that several clients share, run the llama.cpp server itself, which tracks upstream features first. See llama-cpp-python. Some books still show Llama(model_path=…​) with an explicit chat_format; current GGUFs embed their chat template, which the library uses automatically.

Client snippets

curl:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.1-8b",
       "messages": [{"role": "user", "content": "What is retrieval-augmented generation?"}]}'

Python — the openai SDK and LangChain’s ChatOpenAI (llama.cpp is not Ollama, so langchain-ollama does not apply):

from openai import OpenAI
from langchain_openai import ChatOpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")  # use your --api-key if set
reply = client.chat.completions.create(
    model="llama3.1-8b",
    messages=[{"role": "user", "content": "What is retrieval-augmented generation?"}],
)
print(reply.choices[0].message.content)

chat = ChatOpenAI(base_url="http://localhost:8080/v1", api_key="not-needed", model="llama3.1-8b")
print(chat.invoke("What is retrieval-augmented generation?").content)

Spring AI — the OpenAI starter with its base URL pointed at the server:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>
spring:
  ai:
    openai:
      base-url: http://localhost:8080/v1
      api-key: ${LLAMA_API_KEY:not-needed}
      chat:
        model: llama3.1-8b

The base URL includes /v1 because Spring AI 2.0 appends /chat/completions to it as given (see Spring AI: Local Models). The ChatClient code is the same as on Ollama. See openai-python, LangChain — ChatOpenAI and Spring AI — OpenAI Chat.

References