Running LLMs Locally

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Examples across this sub-section were written against the following versions, all read from the projects' release tags, PyPI, npm and Maven Central on 2026-09-28: Ollama 0.34.4; llama.cpp v0.5.0 (the project now tags semantic-version releases alongside its bNNNN build tags — b11239 on that date — and ships both the unified llama serve binary and llama-server); vLLM 0.30.0; SGLang 0.5.20; Text Generation Inference v3.3.7 (maintenance mode); LM Studio 0.4; Jan v0.8.4; Docker Model Runner v1.2.8; LocalAI v4.10.0; mlx-lm 0.31.3; LiteLLM 1.103.0; llm-compressor 0.14.0; llama-cpp-python 0.3.35; Continue 2.1; Cline 4.1; Aider 0.86.2; OpenCode 1.18.33; GitHub Copilot CLI 1.0.89; and Claude Code 2.1.284. Client code uses Python 3.12 with openai 3.20.0, anthropic 1.9.0, langchain-ollama 1.1.0 and langchain-openai 1.6.6, and Java 21 with Spring Boot 4.1 and Spring AI 2.0.1 — the same Spring baseline as LLM Foundations & Prompting. Every running example uses DocsAssistant's own local models — Ollama llama3.1:8b for chat and nomic-embed-text for embeddings, as in Vector Databases & RAG — plus current open-weight models such as qwen3.6 and gpt-oss:20b where a page needs tool calling. Several documentation hosts (docs.ollama.com, docs.vllm.ai, lmstudio.ai, docs.nvidia.com, huggingface.co model cards and others) could not be reached from the checking environment; those projects' docs were read from their public source repositories instead, and each page says where a fact could not be verified. If a page and this baseline disagree after a later update, trust this baseline.

Why run locally

Hosted APIs are the default for good reasons — the strongest models, no hardware, no operations. Running an open-weight model yourself is worth it when one of these matters more:

  • Privacy and data residency — prompts, code and documents never leave your machine or your network. For regulated data, source code under strict policy, or customer data with residency rules, this can be the only acceptable option.

  • Cost at volume — a hosted API charges per token forever; a GPU you already own costs the same whether it serves a thousand requests or a million. At high, steady volume the balance can tip (the method, not prices, is on Running Local LLMs in Production).

  • Offline use — on a plane, in an air-gapped site, or when the provider has an outage.

  • Latency — no network round trip, no shared queue; a small local model can answer faster than a large remote one.

  • Control — a pinned model that never changes under you, no rate limits, no deprecations, and the freedom to quantize, fine-tune or constrain it.

The trade-offs

  • Quality gap — the best open-weight models are strong, but a model that fits a laptop is not a frontier model. Evaluate on your own tasks; pick the task, then the model (see Open-Weight Models).

  • Operations burden — you install, update, secure, monitor and scale the servers. Most local engines are unauthenticated by default, which makes security your job from day one.

  • Hardware — memory decides which models run at all, bandwidth decides how fast (see Hardware Sizing); a GPU server is a capital purchase with a lead time.

A common outcome is a mix: local models for private data, bulk and offline work, hosted models for the hardest tasks — with a gateway routing between them (see LLM Gateways).

Choosing an engine

Situation Engine Why

A laptop or workstation, one user

Ollama, LM Studio or, on Apple silicon, MLX-LM (see Desktop Runners)

One install, automatic GPU offload, a GUI (LM Studio) or CLI, OpenAI- and Anthropic-compatible APIs.

A single GPU, full control

llama.cpp or Ollama

Every flag exposed: offload, context and slots, KV-cache types, speculative decoding; any GGUF.

Inside a Docker workflow

Docker Model Runner (see Desktop Runners)

Models as Compose dependencies.

A multi-user GPU server

vLLM or SGLang

Continuous batching, PagedAttention / RadixAttention, tensor parallelism, FP8 and 4-bit checkpoints.

An enterprise with NVIDIA AI Enterprise

NVIDIA NIM

Pre-built, tuned, supported containers per model.
flowchart TD Start(["Where will the model run?"]) --> Q1{"Shared by several users
or services?"} Q1 -->|"no: one person"| Q2{"Apple silicon?"} Q2 -->|"yes"| MAC["Ollama or LM Studio
(MLX-LM for native MLX)"] Q2 -->|"no"| Q3{"Want a GUI or
Docker-native?"} Q3 -->|"GUI"| LMS["LM Studio / Jan"] Q3 -->|"Docker"| DMR["Docker Model Runner"] Q3 -->|"CLI / API"| Q4{"Need every
engine flag?"} Q4 -->|"no"| OLL["Ollama"] Q4 -->|"yes"| LCPP["llama.cpp
(llama serve)"] Q1 -->|"yes"| Q5{"NVIDIA AI Enterprise
licence and support needed?"} Q5 -->|"yes"| NIM["NVIDIA NIM"] Q5 -->|"no"| Q6{"Data-centre GPU(s)?"} Q6 -->|"yes"| VLLM["vLLM or SGLang"] Q6 -->|"no: one consumer GPU"| LCPP2["llama.cpp with parallel slots
or Ollama with OLLAMA_NUM_PARALLEL"] MAC & LMS & DMR & OLL & LCPP & LCPP2 & VLLM & NIM --> GW["Put a gateway with auth in front
once more than one client uses it"]

DocsAssistant, locally

DocsAssistant, the running example of this site’s AI and vector-database sections, already runs its models locally: Ollama serves llama3.1:8b for chat and nomic-embed-text for embeddings. This sub-section takes that setup further. Every engine page ends with the same three clients — curl against the OpenAI-compatible endpoint, Python (the openai SDK with a base_url, then LangChain), and Spring AI — so moving DocsAssistant from Ollama to llama.cpp, vLLM or NIM is a base-URL change. The integration pages then point GitHub Copilot, Claude Code and the open coding assistants at the same local models.

What’s covered

Foundations

  • Hardware Sizing — weights and KV-cache memory, overhead, MoE total vs. active parameters, bandwidth and MBU, a worked table, unified memory vs. VRAM, CPU-only, and a sizing calculator.

  • Quantization for Inference — numeric formats, the quantization formula, GGUF quant types and importance matrices, GPTQ / AWQ / FP8, quality trade-offs and KV-cache quantization.

Engines

  • Ollama — install, the CLI, Hub GGUFs, the Modelfile, context length and silent truncation, native / OpenAI / Anthropic APIs, embeddings, structured outputs, tools, GPUs, cloud models and keep_alive.

  • llama.cpp — backends, llama serve vs. llama-server, server flags and slots, endpoints, grammars and JSON schema, speculative decoding and llama-cpp-python.

  • vLLM & SGLang — vllm serve, PagedAttention, continuous batching, prefix caching, quantized checkpoints, tensor parallelism, Docker and Kubernetes, RadixAttention, TGI’s maintenance mode and a benchmark harness.

  • Desktop Runners — LM Studio and llmster, Jan, Docker Model Runner, LocalAI and MLX-LM, compared.

  • NVIDIA NIM — NIM containers, the OpenAI API, NVIDIA AI Enterprise licensing and when an enterprise needs it.

  • Open-Weight Models — a dated table of the current families, licence classes and choosing by task.

Gateways and optimisation

  • LLM Gateways — the LiteLLM proxy, virtual keys, budgets and fallbacks, OpenRouter, Ollama cloud models, and authentication in front of local servers.

  • Inference Optimization — prefill vs. decode, TTFT / TPOT / goodput, batching, caching, speculative decoding, FlashAttention, GQA and a measuring script.

Coding assistants

Operations

  • Running Local LLMs in Production — containers and GPU passthrough, health checks, weight caching, autoscaling limits, securing endpoints, monitoring and a cost-comparison method.

Cheat sheet

Related sub-sections: LLM Foundations & Prompting explains the models themselves; AI-Assisted Development and Customizing AI Workflows cover the coding assistants; and AI Agents builds agents that can run on these models. Obtaining and converting models from the Hugging Face Hub and running them inside apps and devices belongs to the Hugging Face & On-Device Models sub-section; serving at scale to LLMOps: Serving at Scale.

Bibliography

The following sources were consulted while preparing this sub-section’s pages. Every command, flag, environment variable, configuration key, endpoint and version was checked against the projects' official documentation (read on the documentation sites where reachable, otherwise from the documentation sources in the projects' own repositories), release tags and package registries on 2026-09-28 — never copied from a book.

Requester-provided books, consulted for concepts only and never the source of an example:

Official documentation

Ollama:

llama.cpp:

vLLM:

SGLang:

TGI (maintenance mode):

Desktop runners:

NVIDIA NIM:

Gateways:

GitHub Copilot:

Claude Code (https://code.claude.com/docs/en/):

Open coding assistants:

Clients:

Model vendors:

Specifications and standards

Papers and engineering articles

These books are consulted references, not the primary source of any example on this sub-section’s pages. On any discrepancy between a book and a tool’s own official documentation, that documentation is authoritative.