LLM Gateways

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page was written against LiteLLM 1.103.0 (released 2026-09-27) and Ollama 0.34.4, as checked on 2026-09-28 (see the Running LLMs Locally baseline). The LiteLLM documentation was read from its source repository (BerriAI/litellm-docs) on that date; OpenRouter’s and nginx’s sites could not be reached from the checking environment, so the snippets that use them follow the conventions shown in the Continue and Ollama documentation and should be checked against the vendors' pages.

Once more than one person or tool uses local models, a gateway in front of them pays for itself. It gives every client one URL and one credential, hides which engine (or which hosted provider) actually answers, tracks usage per key, enforces budgets and rate limits, and falls back to another model when one is down. It is also the natural place to add the authentication that local engines lack.

flowchart LR subgraph Clients IDE["VS Code / Copilot
Custom Endpoint"] CC["Claude Code
ANTHROPIC_BASE_URL"] APP["DocsAssistant
(Spring AI / Python)"] end subgraph GW["Gateway (LiteLLM)"] AUTH["virtual keys ·
budgets · rate limits"] ROUTE["routing ·
fallbacks · logging"] end subgraph Local["Local models"] OLL["Ollama
:11434"] VLLM["vLLM
:8000"] end subgraph Hosted["Hosted models"] ANT["Anthropic API"] OR["OpenRouter"] end IDE -->|"OpenAI /v1"| AUTH CC -->|"Anthropic /v1/messages"| AUTH APP -->|"OpenAI /v1"| AUTH AUTH --> ROUTE ROUTE --> OLL ROUTE --> VLLM ROUTE -.->|"fallback"| ANT ROUTE -.-> OR

The LiteLLM proxy

LiteLLM is an open-source gateway that translates between provider APIs. Its proxy exposes one OpenAI-compatible endpoint (/v1/chat/completions, /v1/embeddings, …) and a unified Anthropic-format /v1/messages, and routes each request to the model named in a config.yaml. A DocsAssistant configuration that routes between a local Ollama model, a vLLM server and a hosted Claude model:

model_list:
  - model_name: docs-chat                  # the name clients send
    litellm_params:
      model: ollama_chat/llama3.1:8b       # ollama_chat/ uses Ollama's /api/chat (recommended over ollama/)
      api_base: http://ollama:11434
  - model_name: docs-chat-large
    litellm_params:
      model: hosted_vllm/meta-llama/Llama-3.3-70B-Instruct   # any OpenAI-compatible server
      api_base: http://vllm:8000/v1
      api_key: os.environ/VLLM_API_KEY
  - model_name: docs-embed
    litellm_params:
      model: ollama/nomic-embed-text
      api_base: http://ollama:11434
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  fallbacks: [{"docs-chat": ["docs-chat-large"]}, {"docs-chat-large": ["claude-sonnet"]}]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY  # every call must carry this key or a virtual key
  database_url: os.environ/DATABASE_URL      # Postgres: required for virtual keys and budgets

os.environ/NAME makes LiteLLM read the value from the environment, so no secret is written into the file. Start it with the CLI or the official image:

uv tool install 'litellm[proxy]'
litellm --config config.yaml              # listens on http://0.0.0.0:4000

# or
docker run -v "$PWD/config.yaml:/app/config.yaml" \
  -e LITELLM_MASTER_KEY -e DATABASE_URL -e ANTHROPIC_API_KEY -e VLLM_API_KEY \
  -p 127.0.0.1:4000:4000 docker.litellm.ai/berriai/litellm:latest --config /app/config.yaml

See LiteLLM — Proxy, LiteLLM — Config and LiteLLM — Ollama.

Virtual keys and budgets

With a database attached, the master key mints virtual keys — one per developer, team or application — each restricted to some models and optionally capped with a budget and rate limits:

curl http://localhost:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"models": ["docs-chat", "docs-embed"], "max_budget": 10, "budget_duration": "30d",
       "rpm_limit": 60, "metadata": {"team": "docs"}}'

The response contains the new key; clients use it as their API key. Local models cost nothing per token, so a budget matters for the hosted fallback, while rpm_limit protects a shared GPU from one noisy client. LiteLLM’s docs are explicit that budgets are enforced against spend stored in the database and do nothing on a database-less deployment. See LiteLLM — Virtual keys and LiteLLM — Budgets and rate limits.

Fallbacks and routing between local and hosted models

router_settings.fallbacks retries a failed request on the next model in the list, so the configuration above tries the laptop-class model, then the GPU server, then the hosted model. LiteLLM also has context_window_fallbacks (switch to a larger-context model when a prompt does not fit) and several load-balancing strategies across multiple deployments of the same model_name. Route to a hosted model only for data that is allowed to leave your network — a fallback silently changes where a prompt goes. See LiteLLM — Fallbacks and reliability.

One Anthropic-format endpoint

LiteLLM’s /v1/messages accepts Anthropic-format requests and translates them for whichever model the model field names, so Anthropic-API clients such as Claude Code can reach local models through it:

curl http://localhost:4000/v1/messages \
  -H "Authorization: Bearer $LITELLM_VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "docs-chat", "max_tokens": 256, "messages": [{"role": "user", "content": "Hello"}]}'

For Claude Code specifically, see Local Models in Claude Code, including what Anthropic does and does not support. See LiteLLM — /v1/messages.

OpenRouter

OpenRouter is a hosted gateway: one account and one OpenAI-compatible API in front of many providers' models, including hosted versions of the open-weight families on Open-Weight Models. It is useful to try an open model before buying hardware for it, or as a hosted fallback — with the same caveat that prompts then leave your network.

import os

from openai import OpenAI

client = OpenAI(base_url="https://openrouter.ai/api/v1", api_key=os.environ["OPENROUTER_API_KEY"])
reply = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[{"role": "user", "content": "What is retrieval-augmented generation?"}],
)
print(reply.choices[0].message.content)

In LiteLLM, reference OpenRouter models as openrouter/<model>. See OpenRouter — Documentation and LiteLLM — OpenRouter.

Ollama as a gateway to cloud models

A signed-in Ollama server serves :cloud models through the same local API as its local ones, so one base URL reaches both, and tools configured for Ollama need no change:

ollama signin
curl http://localhost:11434/api/chat -d '{"model": "gemma4:cloud",
  "messages": [{"role": "user", "content": "Hello"}], "stream": false}'

This is convenient on a laptop but it is not an access-control layer: there are no per-user keys or budgets. Use OLLAMA_NO_CLOUD=1 where prompts must never leave the machine. See Ollama — Cloud and Ollama — Cloud models.

Authentication in front of local servers

Most local engines are unauthenticated by default: Ollama’s local server ignores API keys, Docker Model Runner is documented as unauthenticated, LM Studio and LocalAI have authentication switched off unless you enable it, and vLLM’s --api-key protects only its /v1-style routes. Binding any of them to 0.0.0.0 or publishing its port exposes the model — and, for some, model management — to everyone on the network.

The fix is always the same: keep the engine on 127.0.0.1 (or a private container network), and put a reverse proxy in front that terminates TLS and checks a key. With nginx, a map turns the Authorization header into a client name, and requests without a known key get 401:

# /etc/nginx/conf.d/llm.conf
map $http_authorization $llm_client {
    default "";
    include /etc/nginx/llm-keys.map;   # lines like:  "Bearer <random-token>"  docsassistant;
}

server {
    listen 443 ssl;
    server_name llm.internal.example.com;
    ssl_certificate     /etc/nginx/tls/llm.crt;
    ssl_certificate_key /etc/nginx/tls/llm.key;

    location / {
        if ($llm_client = "") { return 401; }
        proxy_pass http://127.0.0.1:11434;
        proxy_set_header Host localhost:11434;   # Ollama checks the Host header
        proxy_set_header Authorization "";        # do not forward client keys upstream
        proxy_buffering off;                      # stream tokens as they are generated
        proxy_read_timeout 600s;                  # long generations
    }
}

Generate tokens with a secrets tool (for example openssl rand -hex 32), keep llm-keys.map out of version control, and log $llm_client to attribute usage. For vLLM, add a location allowlist so only /v1/ routes are proxied. A gateway such as LiteLLM gives you the same protection with per-key budgets on top; the reverse proxy is the minimal version. See Ollama — FAQ (proxy servers and the Host header), vLLM — Security and nginx — map module.