NVIDIA NIM

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page describes NVIDIA NIM for LLM and VLM 2.0.13 as of 2026-09-29, checked against the NIM for LLM and VLM documentation (last updated 2026-09-24), with Python 3.12 (openai 3.20.0) and Spring AI 2.0.1 for the clients (see the Running LLMs Locally baseline). NIM releases often, so check the current release, image tags and licence terms there before you rely on them.

NIM ("NVIDIA Inference Microservices") packages an open model together with a tuned inference engine and an OpenAI-compatible API server in one container. Since 2.0, NIM for LLM and VLM is built directly on vLLM (earlier 1.x releases used several backends, including TensorRT-LLM), and some images ship an SGLang backend instead. At start-up, the container detects the GPUs and picks a matching model profile — engine, precision and tensor/pipeline-parallel degree — then downloads the matching curated weights. The result is roughly what you would build yourself from vLLM & SGLang plus quantization and tuning, but pre-assembled, validated and supported by NVIDIA.

NIM containers for LLMs

There are two kinds of container, both pulled from NVIDIA’s NGC registry (nvcr.io):

Model-specific NIMs — one image per model under nvcr.io/nim/<publisher>/<model>:<version>, for example nvcr.io/nim/meta/llama-3.1-8b-instruct:2.0.13, with curated weights and validated profiles for that model.
The model-free NIM — nvcr.io/nim/nvidia/model-free-nim:2.0.13, which serves a model you choose at runtime (NIM_MODEL_PATH) from NGC, Hugging Face, S3, Google Cloud Storage or a local directory; use it for custom or newly released models.

The exact image and tag for each model and GPU are in the Support Matrix. Most current public-catalog images can be pulled without logging in. An NGC Personal API key (legacy API keys are not supported) is only required for Production Branch (-pb<version>) NIMs, NIMs released before 2.0.10, and private or gated images or model artifacts:

export NGC_API_KEY=...           # from your secrets manager; never commit it
echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

The literal username $oauthtoken (in single quotes, so the shell does not expand it) is how NGC accepts an API key as a password. See Prerequisites and Installation.

Running a NIM

The weights are cached in a host directory mounted at /opt/nim/.cache, so they are downloaded only once, and the API listens on port 8000:

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"

docker run -it --rm \
  --gpus all \
  -e NGC_API_KEY \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -u "$(id -u):0" \
  -p 127.0.0.1:8000:8000 \
  nvcr.io/nim/meta/llama-3.1-8b-instruct:2.0.13

-e NGC_API_KEY passes the variable through from your shell without writing its value on the command line; drop it for keyless images. When you run as your own user, keep the :0 group in -u "$(id -u):0": the container’s writable directories are owned by GID 0, and without it the container fails with a PermissionError. The host needs NVIDIA driver 580+, CUDA 12.9+, Docker 24+ and the NVIDIA Container Toolkit 1.14+ (see containerising).

The first positional argument selects an action (the default is nim-serve, which forwards any further arguments to vLLM). list-model-profiles shows the profiles available on your hardware, download-to-cache pre-fetches weights without starting the server, and NIM_MODEL_PROFILE pins a profile by ID or by description (for example vllm-bf16-tp1-pp1). See Configuration, CLI Reference and Model Profiles and Selection.

Readiness and liveness endpoints are GET /v1/health/ready and GET /v1/health/live; the container is live before the model is loaded, and ready returns 503 until it is, so wait for ready before sending traffic:

until curl -sf http://localhost:8000/v1/health/ready >/dev/null; do sleep 5; done
curl http://localhost:8000/v1/models

NVIDIA’s deployment examples call the API without any credential, so treat it as unauthenticated: publish the port on loopback or a private network only, and front it with an authenticating proxy or gateway (see authentication in front of local servers). See the Quickstart and NVIDIA/nim-deploy (Helm charts and cloud examples).

The OpenAI API

A running NIM exposes /v1/chat/completions, /v1/completions, /v1/responses and /v1/models in the OpenAI shape, so the client snippets below — and every tool on this site that accepts an OpenAI base URL — work unchanged. The model field is the id reported by /v1/models: for a model-specific NIM, the model identifier (for the image above, meta/llama-3.1-8b-instruct); for the model-free NIM, the value of NIM_SERVED_MODEL_NAME. Clients migrating from 1.x must move sampling parameters out of the removed nvext object into top-level fields.

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "meta/llama-3.1-8b-instruct",
       "messages": [{"role": "user", "content": "What is retrieval-augmented generation?"}],
       "max_tokens": 256}'

See the Quickstart and 1.x migration guide.

Licensing: NIM offerings and NVIDIA AI Enterprise

To download and deploy NIM containers you need either a free NVIDIA Developer Program membership or an NVIDIA AI Enterprise (NVAIE) licence (a free 90-day evaluation licence is available), and use is governed by the NVIDIA AI Product Agreement. NVIDIA publishes NIMs under two offerings:

Offering	Availability	Support
NIM (new models, published within about 72 hours of upstream release)	No charge, under the applicable licence terms	Community (NVIDIA Developer Forums)
NIM Certified — Feature Branch	No charge, under the applicable licence terms	Community; NVIDIA Enterprise Support needs an NVAIE subscription
NIM Certified — Production Branch (stable baseline, CVE SLAs, STIG/FIPS where applicable)	Requires an active NVAIE subscription	NVIDIA Enterprise Support, included

Offering

Availability

Support

NIM (new models, published within about 72 hours of upstream release)

No charge, under the applicable licence terms

Community (NVIDIA Developer Forums)

NIM Certified — Feature Branch

No charge, under the applicable licence terms

Community; NVIDIA Enterprise Support needs an NVAIE subscription

NIM Certified — Production Branch (stable baseline, CVE SLAs, STIG/FIPS where applicable)

Requires an active NVAIE subscription

NVIDIA Enterprise Support, included

So NVAIE is what you pay for a Production Branch baseline and enterprise support, not the entry ticket for development. The model’s own licence (see Open-Weight Models) still applies on top. Prices and terms change; this page quotes none, so read the current terms before planning around them. See NIM Offerings, Prerequisites and Legal.

When an enterprise needs NIM

NIM and a self-assembled open stack serve the same OpenAI-compatible API; the difference is who owns the work in between:

Concern	Self-assembled (vLLM / SGLang)	NIM
Engine tuning per GPU	You choose quantization, parallelism and flags	Pre-built profiles picked at start-up
Model support	Any Hugging Face checkpoint the engine supports	Curated model-specific NIMs, or any supported checkpoint through the model-free NIM
Updates and CVEs	You track engine releases and base images	NVIDIA ships patched images (CVE SLAs on the Production Branch)
Support	Community	Community, or enterprise support under NVAIE
Cost	GPU and people	GPU and people; plus an NVAIE subscription for the Production Branch or enterprise support
Lock-in	None beyond the model	NVIDIA GPUs and the NGC registry

Concern

Self-assembled (vLLM / SGLang)

NIM

Engine tuning per GPU

You choose quantization, parallelism and flags

Pre-built profiles picked at start-up

Model support

Any Hugging Face checkpoint the engine supports

Curated model-specific NIMs, or any supported checkpoint through the model-free NIM

Updates and CVEs

You track engine releases and base images

NVIDIA ships patched images (CVE SLAs on the Production Branch)

Support

Community

Community, or enterprise support under NVAIE

Cost

GPU and people

GPU and people; plus an NVAIE subscription for the Production Branch or enterprise support

Lock-in

None beyond the model

NVIDIA GPUs and the NGC registry

NIM makes sense when an organisation runs NVIDIA data-centre GPUs and wants a validated artefact instead of maintaining engine flags itself — and, with an NVIDIA AI Enterprise subscription, a supported Production Branch for regulated or production use. It is overkill for a laptop, for AMD or Apple hardware, or for a team that is comfortable operating vLLM. A managed cloud service is the other alternative; see AI, Data and IoT Services Overview for Azure’s options.

Client snippets

curl: see The OpenAI API above.

Python:

from openai import OpenAI
from langchain_openai import ChatOpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-used")  # the proxy in front holds the real key
reply = client.chat.completions.create(
    model="meta/llama-3.1-8b-instruct",
    messages=[{"role": "user", "content": "What is retrieval-augmented generation?"}],
    max_tokens=256,
)
print(reply.choices[0].message.content)

chat = ChatOpenAI(base_url="http://localhost:8000/v1", api_key="not-used", model="meta/llama-3.1-8b-instruct")

Spring AI — the OpenAI starter (spring-ai-starter-model-openai):

spring:
  ai:
    openai:
      base-url: http://localhost:8000
      api-key: not-used
      chat:
        model: meta/llama-3.1-8b-instruct

The ChatClient code is the same as on Ollama. See openai-python, LangChain — ChatOpenAI and Spring AI — OpenAI Chat.