Desktop Runners

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page was written against LM Studio 0.4 (the series that introduced the headless llmster daemon), Jan v0.8.4, Docker Model Runner v1.2.8, LocalAI v4.10.0 and mlx-lm 0.31.3, with Python 3.12 (openai 3.20.0) and Spring AI 2.0.1 for the clients, as checked on 2026-09-28 (see the Running LLMs Locally baseline). Each tool’s documentation was read from its public source repository on that date.

Ollama is the default choice on this site, but it is not the only way to run a model on one machine. The runners below differ in who they are for: LM Studio and Jan are desktop apps with a chat UI, Docker Model Runner fits models into the Docker workflow, LocalAI is an all-in-one self-hosted API, and MLX-LM is Apple’s own engine for Apple silicon. All of them serve an OpenAI-compatible API, so the client code does not change.

LM Studio

LM Studio comes in three pieces that share one engine:

  • the desktop app — model search and download from the Hugging Face Hub, a chat UI, document chat, MCP servers and a local API server;

  • llmster, the headless daemon — the same engine without the GUI, for a Linux server, a GPU box without a screen, CI, or a service that starts at boot;

  • lms, the CLI, which drives either of them.

# headless install (Linux / macOS); the desktop app installs lms too
curl -fsSL https://lmstudio.ai/install.sh | bash
lms daemon up                    # start llmster
lms get openai/gpt-oss-20b       # download a model
lms load openai/gpt-oss-20b      # load it into memory
lms server start --port 1234     # OpenAI + Anthropic compatible server
lms ps                           # loaded models
lms log stream                   # watch requests and responses

The server listens on http://localhost:1234, binds to 127.0.0.1 by default (--bind 0.0.0.0 or "Serve on Local Network" opens it up) and serves /v1/chat/completions, /v1/responses, /v1/embeddings and /v1/models in the OpenAI shape, plus POST /v1/messages in the Anthropic shape. Authentication is off by default: enable Require Authentication in the server settings and create an API token, which clients then send as Authorization: Bearer (or x-api-key on the Anthropic endpoint). Enable it before binding to anything but loopback. Books written before 2026 show LM Studio as an OpenAI-compatible server in its GUI-only form; the headless daemon and the Anthropic endpoint are newer.

curl http://localhost:1234/v1/messages \
  -H "Content-Type: application/json" \
  -H "x-api-key: $LM_API_TOKEN" \
  -d '{"model": "openai/gpt-oss-20b", "max_tokens": 256,
       "messages": [{"role": "user", "content": "Say hello from LM Studio."}]}'

See LM Studio — Developer docs, Run LM Studio as a service (headless) and Anthropic-compatible Messages.

Jan

Jan is an open-source desktop chat app (built on llama.cpp) with a built-in Local API Server: Settings > Local API Server > Start Server. It listens on http://127.0.0.1:1337 with an /v1 prefix and requires an API key if you set one — set one, especially before switching the host to 0.0.0.0, which the docs call out as the insecure option.

curl http://127.0.0.1:1337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $JAN_API_KEY" \
  -d '{"model": "YOUR_MODEL_ID", "messages": [{"role": "user", "content": "Tell me a joke."}]}'

See Jan — Local API Server.

Docker Model Runner

Docker Model Runner (DMR) runs models as part of Docker Desktop or Docker Engine. Models are OCI artifacts (ai/…​ on Docker Hub) or Hub GGUFs, pulled and run with docker model:

docker model pull ai/smollm2:360M-Q4_K_M
docker model run ai/smollm2 "What is a vector database?"
docker model pull hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF
docker model ls
docker model configure --context-size 8192 ai/qwen2.5-coder

Three inference engines are available: llama.cpp (the default, on every platform), vLLM (Linux x86_64 and Windows WSL2 with NVIDIA GPUs; installed with docker model install-runner --backend vllm --gpu cuda, models usually tagged -vllm) and Diffusers for images. The llama.cpp engine defaults to a 4096-token context, so raise it for RAG or coding use — the same pitfall as Ollama’s context length.

The API is served in three shapes on the same port — TCP 12434 on the host (enabled by default on Docker Engine, opt-in on Docker Desktop), and http://model-runner.docker.internal from containers on Docker Desktop:

Client Base URL

OpenAI SDK / tools

http://localhost:12434/engines/v1

Anthropic SDK / tools

http://localhost:12434 (/v1/messages)

Ollama-compatible tools

http://localhost:12434 (/api/chat, /api/tags, …​)

The Model Runner API is not authenticated. Docker’s documentation states that any client that can reach it — including other containers on the same Docker network — can pull, load and run models and send inference requests. Keep host-side TCP disabled unless you need it, never publish port 12434 beyond loopback, and put an authenticating proxy in front of it for anything shared (see authentication in front of local servers).

In Docker Compose, models are a top-level element, and services that list a model receive its URL and name as environment variables (LLM_URL, LLM_MODEL for a model called llm):

services:
  docsassistant:
    image: docsassistant:latest
    models:
      - llm
      - embedding-model
models:
  llm:
    model: ai/qwen2.5-coder
    context_size: 8192
  embedding-model:
    model: ai/all-minilm

That requires Docker Compose v2.38 or later. How Compose is used for the rest of a local development stack is covered on Compose for Local Development. See Docker Model Runner, DMR — API reference and Use AI models in Compose.

LocalAI

LocalAI is a self-hosted, OpenAI-compatible API server that bundles many backends (llama.cpp, vLLM, transformers, diffusers, speech and image models) behind one endpoint, with an Anthropic Messages endpoint and MCP support as well. It listens on port 8080:

docker run -ti --name local-ai -p 127.0.0.1:8080:8080 -e LOCALAI_API_KEY localai/localai:latest
# NVIDIA GPU: localai/localai:latest-gpu-nvidia-cuda-12 (or -cuda-13) with --gpus all

local-ai run llama-3.2-1b-instruct:q4_k_m          # from the LocalAI gallery
local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf

Setting LOCALAI_API_KEY (one key or a comma-separated list) turns on key authentication; without it the API is open. See LocalAI — Documentation and LocalAI — Authentication.

MLX-LM

On Apple silicon, Apple’s MLX framework runs models natively on unified memory, and mlx-lm is its LLM package: generation, chat, fine-tuning, quantization, and a small OpenAI-compatible server. Models come from the mlx-community organisation on the Hub, already converted and usually 4-bit.

pip install mlx-lm
mlx_lm.generate --model mlx-community/Mistral-7B-Instruct-v0.3-4bit --prompt "What is MLX?"
mlx_lm.server --model mlx-community/Mistral-7B-Instruct-v0.3-4bit    # http://localhost:8080

--kv-bits 4 quantizes the KV cache for long contexts (and disables batching). The project’s own server documentation says the MLX LM server is not recommended for production because it implements only basic security checks — keep it on localhost, as a development convenience. See mlx-lm and its server documentation.

Comparison

Runner Platforms API surface Engine GUI Auth by default

Ollama

macOS, Linux, Windows, Docker

native, OpenAI, Anthropic

llama.cpp-derived

app (basic)

none (local server ignores keys)

LM Studio

macOS, Linux, Windows; headless llmster

native REST, OpenAI, Anthropic

llama.cpp, MLX

full

off (opt-in API tokens)

Jan

macOS, Linux, Windows

OpenAI

llama.cpp

full

optional key

Docker Model Runner

Docker Desktop, Docker Engine

OpenAI, Anthropic, Ollama

llama.cpp, vLLM, Diffusers

Docker Desktop

none (documented as unauthenticated)

LocalAI

Docker, Linux, macOS

OpenAI, Anthropic, more

many backends

web UI

off unless LOCALAI_API_KEY

MLX-LM

macOS on Apple silicon

OpenAI (basic)

MLX

none

none (not for production)

GPT4All, which some of the books in this sub-section’s bibliography use for local models with Q4_0 GGUFs, is now a niche choice: the Hugging Face Hub integrates GGUF directly with llama.cpp, Ollama, LM Studio, Jan and Docker Model Runner, and the runners above cover the same ground with current quant types (Q4_K_M) and OpenAI-compatible servers. See GGUF on the Hugging Face Hub.

Client snippets

The clients are the same for every runner; change the base URL and model name from the table below.

Runner OpenAI base URL

LM Studio

http://localhost:1234/v1

Jan

http://127.0.0.1:1337/v1

Docker Model Runner

http://localhost:12434/engines/v1 (host), http://model-runner.docker.internal/engines/v1 (containers)

LocalAI

http://localhost:8080/v1

MLX-LM

http://localhost:8080/v1

curl (Docker Model Runner):

curl http://localhost:12434/engines/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "ai/smollm2",
       "messages": [{"role": "user", "content": "What is retrieval-augmented generation?"}]}'

Python (LM Studio):

import os

from openai import OpenAI
from langchain_openai import ChatOpenAI

client = OpenAI(base_url="http://localhost:1234/v1", api_key=os.environ.get("LM_API_TOKEN", "lm-studio"))
reply = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[{"role": "user", "content": "What is retrieval-augmented generation?"}],
)
print(reply.choices[0].message.content)

chat = ChatOpenAI(base_url="http://localhost:1234/v1", api_key=os.environ.get("LM_API_TOKEN", "lm-studio"),
                  model="openai/gpt-oss-20b")

Spring AI (MLX-LM, or any runner above) — the OpenAI starter:

spring:
  ai:
    openai:
      base-url: http://localhost:8080/v1
      api-key: not-needed
      chat:
        model: mlx-community/Mistral-7B-Instruct-v0.3-4bit

The base URL includes /v1 because Spring AI 2.0 appends /chat/completions to it as given; use the port of the runner you started (LM Studio listens on 1234, for example). For Docker Model Runner the base URL is spring.ai.openai.base-url=http://localhost:12434/engines/v1: the Spring AI reference also mentions http://localhost:12434/engines, but on 2.0.1 that form misses the /v1 route (see Spring AI: Local Models). The ChatClient code is the same as on Ollama. See Spring AI — OpenAI Chat, LangChain — ChatOpenAI and openai-python.

References