vLLM & SGLang

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page was written against vLLM 0.30.0 (released 2026-09-22), SGLang 0.5.20 (2026-09-18) and Text Generation Inference (TGI) v3.3.7, with Python 3.12 (openai 3.20.0) and Spring AI 2.0.1 for the clients, as checked on 2026-09-28 (see the Running LLMs Locally baseline). The vLLM and SGLang documentation was read from the docs/ folders of their repositories on that date.

Ollama and llama.cpp are built for one user on one machine. When a team shares a GPU server — DocsAssistant behind an internal chat UI, or a coding assistant for a whole department — the engine has to keep the GPU busy with many requests at once. vLLM and SGLang are the two open-source engines built for that: they load Hugging Face checkpoints (Safetensors, often FP8 or 4-bit), batch requests continuously and manage the KV cache in pages, and expose OpenAI- and Anthropic-compatible APIs.

`vllm serve`

pip install vllm
vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 32768 --port 8000

The server listens on port 8000 and downloads the model from the Hugging Face Hub (set HF_TOKEN for gated models). Useful flags:

Flag Meaning

Flag	Meaning
`--max-model-len N`	Maximum context per request; lower it to leave more room for concurrent sequences.
`--gpu-memory-utilization F`	Fraction of GPU memory vLLM pre-allocates for weights plus KV cache.
`--served-model-name NAME`	The name clients send in `model` (the Hub id by default).
`--tensor-parallel-size N`	Split the model across N GPUs (see below).
`--kv-cache-dtype fp8`	FP8 KV cache.
`--enable-auto-tool-choice --tool-call-parser P`	Tool calling, with the parser for the model family.
`--api-key KEY` / `VLLM_API_KEY`	Require a bearer key on the `/v1` routes (see the security note below).

--max-model-len N

Maximum context per request; lower it to leave more room for concurrent sequences.

--gpu-memory-utilization F

Fraction of GPU memory vLLM pre-allocates for weights plus KV cache.

--served-model-name NAME

The name clients send in model (the Hub id by default).

--tensor-parallel-size N

Split the model across N GPUs (see below).

--kv-cache-dtype fp8

FP8 KV cache.

--enable-auto-tool-choice --tool-call-parser P

Tool calling, with the parser for the model family.

--api-key KEY / VLLM_API_KEY

Require a bearer key on the /v1 routes (see the security note below).

See vLLM — Online serving and the vLLM documentation.

PagedAttention, continuous batching, prefix caching

PagedAttention stores the KV cache in fixed-size blocks, like virtual-memory pages, instead of one contiguous buffer per request sized for the maximum length. Almost no memory is wasted on reserved-but-unused context, so many more sequences fit, and requests with a common prefix can share blocks (Kwon et al.).
Continuous batching schedules at the level of a single decoding iteration: a finished sequence leaves the batch immediately and a waiting one joins at the next step, instead of the whole batch waiting for its longest member.
Prefix caching keeps the KV blocks of previous prompts and reuses them when a new request starts with the same tokens — the system prompt, tool definitions, or the retrieved documents of a follow-up question. vLLM calls it automatic prefix caching (--enable-prefix-caching; the default depends on the version, so check vllm serve --help).

flowchart LR subgraph T1["iteration t"] A1["req A · token 7"] B1["req B · token 3"] C1["req C · token 12 (last)"] end subgraph T2["iteration t+1"] A2["req A · token 8"] B2["req B · token 4"] D2["req D · prefill (joins)"] end subgraph T3["iteration t+2"] A3["req A · token 9"] B3["req B · token 5"] D3["req D · token 1"] end T1 -->|"C finishes and leaves,
D takes its slot"| T2 --> T3

Static batching would have left C’s slot idle until A and B finished; continuous batching refills it at the next step. See vLLM — Automatic prefix caching.

Speculative decoding

vLLM takes a JSON --speculative-config: a draft model, an EAGLE/EAGLE-3 or MTP head, or n-gram lookup.

vllm serve Qwen/Qwen3.6-27B \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 4}'

The output distribution is unchanged; the acceptance rate (reported in the vllm:spec_decode_* metrics) decides whether it pays off for your traffic. See vLLM — Speculative decoding and Leviathan et al..

Quantized checkpoints

vLLM reads the quantization method from a checkpoint’s config, so an FP8, AWQ, GPTQ or compressed-tensors repository needs no extra flag, and --quantization can quantize a 16-bit model online at load time:

vllm serve RedHatAI/GLM-5.2-NVFP4
vllm serve meta-llama/Llama-3.1-8B-Instruct --quantization fp8_per_block

Producing such checkpoints with llm-compressor is covered on Quantization for Inference. See vLLM — Quantization.

OpenAI and Anthropic endpoints

The server implements /v1/chat/completions, /v1/completions, /v1/responses, /v1/embeddings and /v1/models in the OpenAI shape, and /v1/messages and /v1/messages/count_tokens in the Anthropic shape. The Anthropic endpoint is what lets Claude Code talk to a vLLM server (see Local Models in Claude Code):

curl http://localhost:8000/v1/messages \
  -H "Content-Type: application/json" \
  -H "x-api-key: $VLLM_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "max_tokens": 256,
       "messages": [{"role": "user", "content": "Hello"}]}'

Security note. vLLM is unauthenticated unless you pass --api-key, and even then, its security guide states, the key protects only the /v1, /v2, /inference and /cohere path prefixes; other endpoints on the same server (health, metrics, tokenization and others) stay open. Never expose vLLM directly: bind it to a private network and put a reverse proxy in front that allowlists the routes you intend to serve (see authentication in front of local servers and Running Local LLMs in Production). See vLLM — Security.

Tensor parallelism

A model too large for one GPU is split across several: tensor parallelism shards every layer’s matrices across GPUs in one node (fast interconnect such as NVLink matters); pipeline parallelism puts consecutive layers on different GPUs or nodes. For a 70B model at FP8 on two 80 GB GPUs:

vllm serve meta-llama/Llama-3.3-70B-Instruct --tensor-parallel-size 2 --kv-cache-dtype fp8

Multi-node clusters, data and expert parallelism are out of scope here; see vLLM — Parallelism and scaling.

Docker and Kubernetes

The official image is vllm/vllm-openai. Mount the Hugging Face cache so weights are downloaded once, and use --ipc=host (or a large --shm-size) for the inter-process shared memory the workers need:

docker run --rm --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN -e VLLM_API_KEY \
  -p 127.0.0.1:8000:8000 --ipc=host \
  vllm/vllm-openai:latest \
  meta-llama/Llama-3.1-8B-Instruct --max-model-len 32768

On Kubernetes, a Deployment requests nvidia.com/gpu resources (the NVIDIA device plugin or GPU Operator must be installed), mounts a persistent volume for the model cache and probes /health:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: docsassistant-vllm
spec:
  replicas: 1
  selector:
    matchLabels: {app: docsassistant-vllm}
  template:
    metadata:
      labels: {app: docsassistant-vllm}
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args: ["meta-llama/Llama-3.1-8B-Instruct", "--max-model-len", "32768"]
          env:
            - name: HF_TOKEN
              valueFrom: {secretKeyRef: {name: hf-token, key: token}}
            - name: VLLM_API_KEY
              valueFrom: {secretKeyRef: {name: vllm-api-key, key: key}}
          ports: [{containerPort: 8000}]
          resources:
            limits: {nvidia.com/gpu: "1"}
          readinessProbe:
            httpGet: {path: /health, port: 8000}
            periodSeconds: 10
          volumeMounts:
            - {name: hf-cache, mountPath: /root/.cache/huggingface}
      volumes:
        - name: hf-cache
          persistentVolumeClaim: {claimName: hf-cache}

See vLLM — Docker and vLLM — Kubernetes. Health checks, monitoring and the rest of the operational picture are on Running Local LLMs in Production.

SGLang and RadixAttention

SGLang is the other high-throughput open engine. Its distinguishing feature, RadixAttention, stores the KV cache of all requests in a radix tree keyed by token sequences, so any shared prefix — a system prompt, few-shot examples, a multi-turn history, the branches of an agent’s tool loop — is found and reused automatically, with LRU eviction (Zheng et al.). It also has a front-end language for structured LLM programs, but most teams use it simply as a server:

pip install sglang
sglang serve --model-path meta-llama/Llama-3.1-8B-Instruct --port 30000
# equivalent, and what older guides show:
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 127.0.0.1 --port 30000

The default port is 30000. The OpenAI endpoints are under /v1, and the Anthropic /v1/messages endpoint is registered automatically. --tp-size sets tensor parallelism, --context-length the context, --api-key a bearer key, and --disable-radix-cache turns RadixAttention off. The Docker image is lmsysorg/sglang.

docker run --gpus all --shm-size 32g --ipc=host \
  -p 127.0.0.1:30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface -e HF_TOKEN \
  lmsysorg/sglang:latest \
  python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000

See SGLang — Documentation. The choice between vLLM and SGLang is workload-specific: benchmark both on your prompts with the harness below rather than trusting published numbers.

TGI is in maintenance mode

Hugging Face’s Text Generation Inference, which several books use, is now in maintenance mode (the latest release is v3.3.7): its README says the project will accept only minor bug fixes, documentation and light maintenance, and recommends vLLM, SGLang, llama.cpp or MLX going forward. Existing TGI deployments keep working and expose an OpenAI-compatible /v1/chat/completions, so the client snippets below work against them too, but new deployments should start on vLLM or SGLang. See text-generation-inference.

A benchmark harness

This site does not quote throughput or latency numbers — they depend on the GPU, model, quantization, prompt lengths and concurrency, and they change with every engine release. Measure your own. vLLM ships a benchmark client:

vllm bench serve --backend vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --endpoint /v1/completions \
  --dataset-name random --random-input-len 1024 --random-output-len 256 \
  --num-prompts 200 --max-concurrency 16

Repeating the run against the same server can hit the prefix cache and inflate throughput; vary the prompts or restart between runs. For production-grade load tests the vLLM docs recommend GuideLLM; SGLang has python3 -m sglang.bench_serving. See vLLM — Benchmark CLI.

For an engine-neutral check against any OpenAI-compatible endpoint (Ollama, llama.cpp, vLLM, SGLang, NIM), this Python script fires N concurrent streaming requests and reports time to first token and aggregate output throughput:

"""Concurrent load test for any OpenAI-compatible endpoint (no quoted numbers -- run it yourself)."""
import asyncio
import os
import statistics
import time

from openai import AsyncOpenAI

BASE_URL = os.environ.get("BASE_URL", "http://localhost:8000/v1")
MODEL = os.environ.get("MODEL", "meta-llama/Llama-3.1-8B-Instruct")
CONCURRENCY = int(os.environ.get("CONCURRENCY", "16"))
REQUESTS = int(os.environ.get("REQUESTS", "64"))

client = AsyncOpenAI(base_url=BASE_URL, api_key=os.environ.get("API_KEY", "not-needed"))


async def one(i: int) -> tuple[float, int, float]:
    start = time.perf_counter()
    ttft, chunks = None, 0
    stream = await client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": f"Request {i}: explain continuous batching in 150 words."}],
        max_tokens=256,
        stream=True,
    )
    async for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            if ttft is None:
                ttft = time.perf_counter() - start
            chunks += 1  # roughly one token per chunk on most servers
    return ttft or 0.0, chunks, time.perf_counter() - start


async def main() -> None:
    sem = asyncio.Semaphore(CONCURRENCY)

    async def bounded(i: int):
        async with sem:
            return await one(i)

    t0 = time.perf_counter()
    results = await asyncio.gather(*(bounded(i) for i in range(REQUESTS)))
    wall = time.perf_counter() - t0
    ttfts = sorted(r[0] for r in results)
    tokens = sum(r[1] for r in results)
    print(f"requests={REQUESTS} concurrency={CONCURRENCY} wall={wall:.1f}s")
    print(f"TTFT p50={statistics.median(ttfts):.3f}s p95={ttfts[int(0.95 * (len(ttfts) - 1))]:.3f}s")
    print(f"aggregate output throughput ~ {tokens / wall:.1f} tokens/s")


asyncio.run(main())

Run it at several CONCURRENCY values to find the point where TTFT starts to climb — that knee, not the peak throughput, is your server’s usable capacity. The metrics behind it are defined on Inference Optimization. See openai-python.

Client snippets

curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct",
       "messages": [{"role": "user", "content": "What is retrieval-augmented generation?"}]}'

For SGLang, use port 30000; everything else is identical.

Python:

import os

from openai import OpenAI
from langchain_openai import ChatOpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key=os.environ["VLLM_API_KEY"])
reply = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "What is retrieval-augmented generation?"}],
)
print(reply.choices[0].message.content)

chat = ChatOpenAI(base_url="http://localhost:8000/v1", api_key=os.environ["VLLM_API_KEY"],
                  model="meta-llama/Llama-3.1-8B-Instruct")
print(chat.invoke("What is retrieval-augmented generation?").content)

Spring AI — the OpenAI starter (spring-ai-starter-model-openai):

spring:
  ai:
    openai:
      base-url: http://localhost:8000/v1
      api-key: ${VLLM_API_KEY}
      chat:
        model: meta-llama/Llama-3.1-8B-Instruct

The base URL includes /v1: Spring AI 2.0 builds its OpenAI client on the official OpenAI Java SDK, which appends /chat/completions to the base URL as given, and vLLM serves only /v1/… routes. The extra-body examples on the Spring AI reference page still show http://localhost:8000, the 1.x form; on 2.0.1 that URL sends requests to /chat/completions, which vLLM does not serve (checked with a request-recording server, see Spring AI: Local Models). The starter’s extra-body property passes vLLM-specific sampling parameters such as top_k. The ChatClient code is the same as on Ollama. See Spring AI — OpenAI Chat and LangChain — ChatOpenAI.