Running Local LLMs in Production

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page was written against Ollama 0.34.4, vLLM 0.30.0, llama.cpp v0.5.0 and Docker Compose v2 with the NVIDIA Container Toolkit, as checked on 2026-09-28 (see the Running LLMs Locally baseline). It covers one team’s GPU server or small cluster. Serving many models at scale — autoscaling policies, multi-tenant capacity planning, evaluation in production and SLO dashboards — is the subject of LLMOps: Serving at Scale; multi-node GPU cluster design is out of scope.

Containerising Ollama and vLLM

GPU containers need the host’s NVIDIA driver plus the NVIDIA Container Toolkit, which lets Docker pass GPUs into containers (--gpus all on the CLI, a device reservation in Compose). A DocsAssistant stack with Ollama for the laptop-class models, vLLM for the large one, and an authenticating proxy as the only published port:

services:
  ollama:
    image: ollama/ollama:latest
    volumes:
      - ollama-models:/root/.ollama            # weights survive container restarts
    environment:
      OLLAMA_CONTEXT_LENGTH: "32768"
      OLLAMA_KEEP_ALIVE: "24h"
      OLLAMA_NO_CLOUD: "1"                     # no prompts to Ollama's cloud
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "ollama", "list"]
      interval: 30s
      timeout: 10s
      retries: 5
    # no "ports:" -- reachable only on the Compose network

  vllm:
    image: vllm/vllm-openai:latest
    command: ["meta-llama/Llama-3.1-8B-Instruct", "--max-model-len", "32768"]
    ipc: host
    environment:
      HF_TOKEN: ${HF_TOKEN}
      VLLM_API_KEY: ${VLLM_API_KEY}
    volumes:
      - hf-cache:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["1"]
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
      interval: 30s
      start_period: 600s                       # first start downloads and loads the weights

  gateway:
    image: nginx:stable
    ports:
      - "443:443"
    volumes:
      - ./nginx:/etc/nginx/conf.d:ro
      - ./tls:/etc/nginx/tls:ro
    depends_on:
      ollama: {condition: service_healthy}
      vllm: {condition: service_healthy}

volumes:
  ollama-models:
  hf-cache:

The gateway configuration is the nginx key check on LLM Gateways; a LiteLLM container can take its place. Secrets come from the environment (${HF_TOKEN}), never from the file. Compose itself — how services, networks and volumes work — is covered on Compose for Local Development and Compose in Practice, and Storage: Volumes, Bind Mounts and tmpfs explains the volume choices. See Ollama — Docker, vLLM — Docker and Docker — GPU support in Compose.

Health checks

A model server is not ready when its process starts: it may still be downloading tens of gigabytes and then loading them into GPU memory. Use each engine’s own endpoint and give it a generous start period:

Engine Health / readiness check

Ollama

GET /api/version or ollama list answers once the server is up; models load on first use, so warm them with a request (keep_alive: -1) after start.

llama.cpp

GET /health — HTTP 503 ("Loading model") while loading, 200 once it can serve; public even with --api-key.

vLLM

GET /health, and GET /v1/models lists the served model once loaded.

NVIDIA NIM

GET /v1/health/ready (readiness) and GET /v1/health/live (liveness).

In Kubernetes, map readiness to these endpoints and set a startupProbe long enough for the first load, so the liveness probe does not kill a pod that is still reading weights. See the llama.cpp server README, vLLM — Online serving and Ollama — API.

Model-weight caching and volumes

Weights are the largest artefact in the system — often larger than every container image combined — so treat them like data, not like image layers:

  • Mount a persistent volume for each engine’s cache: /root/.ollama (Ollama), /root/.cache/huggingface (vLLM, SGLang, llama.cpp’s -hf cache), /opt/nim/.cache (NIM).

  • Pre-pull in a controlled step (ollama pull, hf download, NIM’s download-to-cache) rather than on the first user request, and keep a copy on internal storage so a pod reschedule or an air-gapped site does not depend on the public Hub.

  • Pin versions: a model tag such as llama3.1:8b or a Hub repository’s default branch can change underneath you. Pin the digest or a revision for reproducibility.

  • Keep the cache on fast local disk (NVMe): load time from a network share dominates cold starts.

Do not bake large weights into images; it makes every image push, pull and scan slow. See Image Best Practices and Supply-Chain Security and vLLM — Kubernetes.

Autoscaling limits

Autoscaling a GPU model server is much less elastic than scaling a web service:

  • Cold start is minutes, not seconds: schedule a GPU node, pull the image, load tens of gigabytes into VRAM, and warm the caches. Scale on leading signals (queue depth, running requests, KV-cache usage), not on CPU, and keep a warm minimum.

  • GPUs are scarce and indivisible: a replica needs a whole GPU (or several, with tensor parallelism); scaling from one to two doubles the cost in one step, and the capacity may simply not be available.

  • Scale-to-zero trades cost for a long first response; acceptable for batch jobs, rarely for interactive use.

  • Within one replica, continuous batching already absorbs load up to the goodput knee (see goodput); add replicas only past it.

The Kubernetes mechanics are covered on Orchestration: Swarm Mode and Kubernetes.

Securing endpoints

Treat a model server like a database: never on the public internet, always behind authentication.

  • Authentication: most engines are unauthenticated by default, and vLLM’s --api-key covers only its /v1-style routes. Terminate client authentication in a gateway or reverse proxy with per-client keys.

  • TLS: terminate TLS at that proxy; inside a cluster, use the service mesh or mTLS if your platform provides it.

  • Network isolation: only the gateway may reach the engines. In Kubernetes, a NetworkPolicy expresses that:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vllm-only-from-gateway
spec:
  podSelector:
    matchLabels: {app: docsassistant-vllm}
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels: {app: llm-gateway}
      ports:
        - {protocol: TCP, port: 8000}

  • Egress: block outbound traffic from the engines except to the model registry you mirror, so a compromised or misconfigured server cannot send prompts elsewhere (and disable engine cloud features such as Ollama’s).

  • Model-management endpoints: pulling, deleting and loading models (Ollama’s /api/pull and /api/delete, Docker Model Runner’s whole API, llama.cpp’s /models router endpoints) must never be reachable by ordinary clients.

Prompt-injection, data-leakage and tool-abuse risks of the applications built on these models are covered in Agent Guardrails & Safety and the AI Security & Responsible AI sub-section (start here); container hardening in Docker Security. See vLLM — Security.

Monitoring

vLLM serves Prometheus metrics on /metrics; llama.cpp does so when started with --metrics. The metrics that matter map directly onto the definitions on Inference Optimization:

Question vLLM llama.cpp

Is it overloaded?

vllm:num_requests_running

llamacpp:requests_processing, llamacpp:requests_deferred

Time to first token

vllm:time_to_first_token_seconds (histogram)

 — 

Time per output token

vllm:inter_token_latency_seconds (histogram)

llamacpp:predicted_tokens_seconds (tokens/s)

End-to-end latency

vllm:e2e_request_latency_seconds (histogram)

 — 

Throughput

rate(vllm:generation_tokens_total[5m])

rate(llamacpp:tokens_predicted_total[5m])

Memory pressure

vllm:kv_cache_usage_perc

 — 

 
# prometheus.yml
scrape_configs:
  - job_name: vllm
    static_configs:
      - targets: ["vllm:8000"]
  - job_name: llamacpp
    static_configs:
      - targets: ["llamacpp:8080"]

A p95 TTFT alert, for example:

histogram_quantile(0.95, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m]))) > 2

Add GPU metrics (utilisation, memory, temperature) from NVIDIA’s DCGM exporter. Prometheus itself — scraping, PromQL, recording and alerting rules, Kubernetes monitoring — is covered in Prometheus & Monitoring, in particular Containers & Kubernetes monitoring. See the llama.cpp server README (/metrics) and vLLM — Documentation (metrics).

A cost-comparison method

This site quotes no prices — they change monthly and differ per contract. What stays valid is the method. Over a period (say a month), the cost of serving locally is

where is the hardware price amortised over , the average power draw in kW, the hours per month, the electricity price, the people time to run it, and rack or cloud-GPU rental. The hosted cost for the same traffic is

with , the monthly input and output tokens and , the provider’s per-token prices (see Cost, Latency & Throughput). Local serving breaks even when , but only if the hardware can actually carry the volume:

where is the achievable utilisation (traffic is not flat). To use it: measure goodput on your hardware with the harness on vLLM & SGLang, take your real token volumes from the gateway’s logs, plug in your own prices, and compare equal quality — a local model that needs more retries or more human correction is not cheaper. Privacy, residency and offline requirements often decide the question before cost does (see Running LLMs Locally).

For a managed alternative that keeps data in a chosen cloud region without running the servers yourself, see AI, Data and IoT Services Overview for Azure’s AI services.

References