Local Models

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written against Spring AI 2.0.1, Spring Boot 4.1.x and Java 21 (verified 2026-09-29), with Ollama 0.34 and the ONNX default model all-MiniLM-L6-v2. A model that runs on your own machine or in your own cluster costs no per-token fee and sends no text to a third party, and it also has a smaller context, weaker instruction following and one request at a time. Spring AI treats a local model as one more provider behind ChatModel, so DocsAssistant does not need a different design to use one. What changes is configuration and a short list of things that no longer work the way they did with a hosted model. This page adds the capability to run DocsAssistant with no hosted model; running and sizing the model servers themselves is on Running LLMs Locally, and the Hugging Face ecosystem where many of these models come from is the subject of the Hugging Face & On-Device Models sub-section of this guide.

Choosing a route

There are four ways to reach a local model from Spring AI, and they differ in which starter they use:

Table 1. Routes to a local model
Route	Starter	Set	Use it when
Ollama, native API	`spring-ai-starter-model-ollama`	`spring.ai.ollama.base-url` (default `http://localhost:11434`)	The default choice. Ollama-only options work: `think`, `keep-alive`, `num-ctx`, model pull at start-up
Docker Model Runner	`spring-ai-starter-model-openai`	`spring.ai.openai.base-url` = `http://localhost:12434/engines/v1`	The models already live in Docker, or the stack is defined with Docker Compose or Testcontainers
Any OpenAI-compatible server: vLLM, llama.cpp, LM Studio, Foundry Local	`spring-ai-starter-model-openai`	`spring.ai.openai.base-url` with the `/v1` path	A server is already running, or you need vLLM’s throughput for many users
In-process ONNX (embeddings only)	`spring-ai-starter-model-transformers`	`spring.ai.model.embedding=transformers`	No embedding service to run, small documents, offline

flowchart TD S["Need a local model for DocsAssistant"] --> Q1{"One developer machine or a shared server?"} Q1 -- "one machine" --> Q2{"Already using Docker for the stack?"} Q2 -- yes --> DMR["Docker Model Runner + OpenAI starter"] Q2 -- no --> OL["Ollama starter"] Q1 -- "shared server, many users" --> Q3{"GPU available?"} Q3 -- yes --> VL["vLLM + OpenAI starter"] Q3 -- no --> LC["llama.cpp server + OpenAI starter"] OL --> EM{"Embeddings too?"} DMR --> EM VL --> EM LC --> EM EM -- "no service to run" --> ON["ONNX TransformersEmbeddingModel"] EM -- "same server" --> SAME["Ollama or OpenAI-compatible embeddings"]

Only one provider answers ChatModel injection unless you build models by hand. When two chat starters are on the classpath, set spring.ai.model.chat to the one you want (ollama, openai); the same applies to spring.ai.model.embedding. The practical way to keep local and hosted setups side by side is profiles, as shown at the end of this page. The property names above and below were checked against the starters' configuration metadata.

Ollama

The Ollama starter talks to Ollama’s native API and is the route with the most local-specific features. It is the model provider the whole DocsAssistant sub-section uses (llama3.1:8b for chat, nomic-embed-text for embeddings).

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-ollama</artifactId>
</dependency>

Source: Spring AI: Ollama Chat.

Table 2. The Ollama properties that matter locally
Property	Default	Meaning
`spring.ai.ollama.base-url`	`http://localhost:11434`	Where the server is
`spring.ai.ollama.chat.model`		Chat model, for example `llama3.1:8b`
`spring.ai.ollama.embedding.model`		Embedding model, for example `nomic-embed-text`
`spring.ai.ollama.init.pull-model-strategy`	`never`	`always`, `when_missing` or `never`: download the models at start-up
`spring.ai.ollama.init.timeout`	`5m`	How long to wait for a pull
`spring.ai.ollama.init.max-retries`	`0`	Retries of a failed pull
`spring.ai.ollama.init.chat.additional-models`		More chat models to pull at start-up
`spring.ai.ollama.chat.keep-alive`		How long Ollama keeps the model loaded after a request (its own default is 5 minutes)
`spring.ai.ollama.chat.think`		Ask a reasoning model to think first; renamed from `think-option` in 2.0
`spring.ai.ollama.chat.num-ctx`		Context window in tokens for the request

Pull strategy

when_missing downloads a model only if the server does not have it; always re-pulls each start, which is how you get a newer build of a tag and also how a start-up ends up waiting for gigabytes. The default, never, is right for production: the model is put on the server as a deployment step, so an application start never depends on a download and never changes the model under a running system. Use when_missing in development and in tests (the Testcontainers test on Testing relies on it) and keep init.timeout above the time the first pull takes.

application-local.yml: DocsAssistant on Ollama

spring:
  ai:
    model:
      chat: ollama
      embedding: ollama
    ollama:
      base-url: http://localhost:11434
      chat:
        model: llama3.1:8b
        keep-alive: 30m
      embedding:
        model: nomic-embed-text
      init:
        pull-model-strategy: when_missing
        timeout: 10m

Source: Spring AI: Ollama Chat.

Keep-alive, context and thinking

Ollama unloads an idle model after five minutes and loading it again takes seconds for a small model and much longer for a large one, which shows up as a slow first answer after a quiet period. keep-alive extends the time; the matching server-side settings and the memory cost are on Keeping models loaded. num-ctx sets the context window, and the pitfall around it is worth knowing: a prompt longer than the window is silently truncated rather than rejected, so a RAG prompt with five long chunks can lose its beginning, and with it the instructions. The details are on Context length and the silent-truncation pitfall.

Reasoning models can think before they answer, which improves hard answers and costs time and tokens. think is a request option, so use it per request and not globally. A model that has no thinking mode will not accept it, so a plain llama3.1:8b call must not set it.

LocalDocsChat.java: Ollama options as defaults and per request

package com.example.docsassistant;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.ollama.api.OllamaChatOptions;
import org.springframework.stereotype.Service;

/** The DocsAssistant chat path on a local Ollama model, with the Ollama-only options that matter locally. */
@Service
public class LocalDocsChat {

    private final ChatClient chatClient;

    public LocalDocsChat(ChatClient.Builder builder) {
        this.chatClient = builder
                .defaultOptions(OllamaChatOptions.builder()
                        .model("llama3.1:8b")
                        .keepAlive("30m")   // keep the weights in memory between requests
                        .numCtx(8192)       // Ollama's own context window setting; see the context-length pitfall
                        .temperature(0.2))
                .build();
    }

    public String ask(String question) {
        return this.chatClient.prompt().user(question).call().content();
    }

    /** A per-request override for a reasoning model: think is set on the request, not globally. */
    public String askWithReasoning(String question) {
        return this.chatClient.prompt()
                .user(question)
                .options(OllamaChatOptions.builder()
                        .model("qwen3:8b")
                        .keepAlive("30m")   // request options replace defaultOptions: repeat what must still apply
                        .numCtx(8192)
                        .enableThinking())
                .call()
                .content();
    }
}

Source: Spring AI: Ollama Chat.

The class compiled against 2.0.1. The qwen3:8b name is an example of a model with a thinking mode; use any model you have pulled. Options are described on Chat Options, where the rule that a request’s options replace the defaults instead of merging with them also applies: that is why askWithReasoning repeats keepAlive and numCtx, and without them the request would run with the server’s own settings, including its default context window.

Development services

During development you rarely install Ollama by hand. Two Spring AI modules start it for you:

spring-ai-spring-boot-docker-compose recognises a running ollama/ollama container from your Compose file and configures the connection, in addition to Spring Boot’s own spring-boot-docker-compose module.
spring-ai-spring-boot-testcontainers provides the same for tests with @ServiceConnection; the working test is on Testing, and the surrounding practice on Spring Boot Integration Tests with Testcontainers.

compose.yaml: Ollama for local development

services:
  ollama:
    image: ollama/ollama:0.34.4
    ports:
      - "11434:11434"
    volumes:
      - ollama-models:/root/.ollama
volumes:
  ollama-models: {}

Source: Spring AI: Docker Compose Support.

The volume keeps the downloaded models between runs; without it every fresh container pulls them again.

Docker Model Runner

Docker Model Runner (DMR) serves models pulled from Docker Hub through an OpenAI-compatible API, so Spring AI uses the OpenAI starter and points it at DMR. The engine, the model tags, the ports and the fact that its API is not authenticated are covered on Desktop Runners. From Spring AI’s side the setup is three properties:

application-dmr.yml: the OpenAI starter against Docker Model Runner

spring:
  ai:
    model:
      chat: openai
      embedding: none
    openai:
      base-url: http://localhost:12434/engines/v1
      api-key: not-used-by-model-runner
      chat:
        model: ai/gemma3:4B-F16

Source: Spring AI: Docker Model Runner.

The TCP port is opt-in on Docker Desktop (docker desktop enable model-runner --tcp 12434, per the Spring AI page). The key is a placeholder, because the OpenAI client requires a value and DMR ignores it; it is not a secret and no real key belongs in this file. Embeddings are switched off here because the starter would otherwise configure an embedding model against a server that has none loaded; add an embedding model to DMR and remove the line to use it.

For tests, Testcontainers has a DockerModelRunnerContainer that starts a small socat container in front of the host’s Model Runner, pulls the model and exposes the endpoint:

DmrTestConfig.java: Model Runner as a test dependency

package com.example.docsassistant;

import org.springframework.boot.test.context.TestConfiguration;
import org.springframework.context.annotation.Bean;
import org.springframework.test.context.DynamicPropertyRegistrar;
import org.testcontainers.containers.DockerModelRunnerContainer;

/** Test-only: start Docker Model Runner through Testcontainers and point the OpenAI starter at it. */
@TestConfiguration(proxyBeanMethods = false)
class DmrTestConfig {

    private static final String MODEL = "ai/smollm2:360M-Q4_K_M";

    @Bean(destroyMethod = "stop")
    DockerModelRunnerContainer modelRunner() {
        DockerModelRunnerContainer container =
                new DockerModelRunnerContainer("alpine/socat:1.7.4.3-r0").withModel(MODEL);
        container.start();
        return container;
    }

    @Bean
    DynamicPropertyRegistrar modelRunnerProperties(DockerModelRunnerContainer modelRunner) {
        return registry -> {
            // getOpenAIEndpoint() ends in /engines; Spring AI 2.0 appends only /chat/completions
            registry.add("spring.ai.openai.base-url", () -> modelRunner.getOpenAIEndpoint() + "/v1");
            registry.add("spring.ai.openai.api-key", () -> "not-used-by-model-runner");
            registry.add("spring.ai.openai.chat.model", () -> MODEL);
        };
    }
}

Source: Spring AI: Docker Model Runner.

This configuration compiled, but it was not run: it needs Docker Desktop with Model Runner enabled, and Model Runner was not running on the machine where these examples were checked. Treat it as the shape from the Spring AI documentation. In Testcontainers 2.0.5 getOpenAIEndpoint() returns the base URL plus /engines, without /v1, so the configuration adds /v1 to get the /engines/v1 base URL used above; check what your Testcontainers version returns. Tool calling works when the model supports it; the auto-registered ToolCallingAdvisor drives the loop as with any provider.

OpenAI-compatible servers

vLLM, llama.cpp’s llama-server, LM Studio and Foundry Local all expose the OpenAI Chat Completions API, so one Spring AI configuration covers them: the OpenAI starter, a base-url, a placeholder or real key, and the model name the server uses.

Table 3. The OpenAI starter against local servers
Server	`spring.ai.openai.base-url`	Notes
vLLM	`http://localhost:8000/v1`	Model name is the one passed to `vllm serve`; vLLM and SGLang
llama.cpp `llama-server`	`http://localhost:8080/v1`	Model name is free text on a single-model server; llama.cpp
LM Studio	`http://localhost:1234/v1`	Start the server from the app or the `lms` CLI; LM Studio
Foundry Local	the endpoint its CLI or SDK reports	Designed for one user on one device, not as a shared server; Foundry Local

The /v1 suffix is part of the base URL in 2.0. Spring AI 2.0 builds its OpenAI client on OpenAI’s official Java SDK, which appends /chat/completions to whatever base URL it is given. A request-recording server shows what each value produces, both with the client builder and with the spring.ai.openai.base-url property:

Table 4. What the starter requested for each base-url (2.0.1)
`base-url`	Request path
`http://127.0.0.1:PORT`	`POST /chat/completions`
`http://127.0.0.1:PORT/v1`	`POST /v1/chat/completions`
`http://127.0.0.1:PORT/engines/v1`	`POST /engines/v1/chat/completions`

Older examples, including configuration snippets in books and in the 1.x reference, give a base URL without /v1 and rely on a completions-path setting that the 1.x starter added. On 2.0.1 such a URL reaches a server that has no route at that path (vLLM, for one, serves only /v1/…). If a local OpenAI-compatible server returns 404, check this first.

OpenAiCompatibleLocalChat.java: one class for any OpenAI-compatible endpoint

package com.example.docsassistant;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.openai.OpenAiChatOptions;
import org.springframework.stereotype.Service;

/**
 * The OpenAI starter pointed at any OpenAI-compatible server (vLLM, llama.cpp, LM Studio, Docker Model Runner).
 * The endpoint and the key come from configuration, so the same class runs against a local server or a hosted one.
 */
@Service
public class OpenAiCompatibleLocalChat {

    private final ChatClient chatClient;

    public OpenAiCompatibleLocalChat(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    public String ask(String question, String servedModelName) {
        return this.chatClient.prompt()
                .user(question)
                .options(OpenAiChatOptions.builder().model(servedModelName).temperature(0.2))
                .call()
                .content();
    }
}

Source: Spring AI: OpenAI Chat.

application-vllm.yml: DocsAssistant on a vLLM server

spring:
  ai:
    model:
      chat: openai
      embedding: none
      image: none
      moderation: none
      audio:
        speech: none
        transcription: none
    openai:
      base-url: ${VLLM_BASE_URL:http://localhost:8000/v1}
      api-key: ${VLLM_API_KEY:not-needed}
      chat:
        model: meta-llama/Llama-3.1-8B-Instruct

Source: Spring AI: OpenAI Chat.

The OpenAI starter configures chat, embedding, image, audio and moderation models. Against a local server only chat is served, so the other model types are switched off with spring.ai.model.<type>=none; a moderation bean, for one, wants working credentials at start-up. When the server was started with an API key, supply it through VLLM_API_KEY; the not-needed fallback is only for servers with no authentication. Binding such a server beyond loopback without authentication exposes it to every client that can reach the port, which the runner pages cover under authentication in front of local servers.

ONNX embeddings, in process

TransformersEmbeddingModel runs a sentence-transformer model through ONNX Runtime inside the JVM: no embedding server, no network call per document. The starter is spring-ai-starter-model-transformers, and the default model is sentence-transformers/all-MiniLM-L6-v2 (384 dimensions). A test that embedded one sentence with the default configuration returned a 384-dimension vector.

Three things to plan for:

Dimensions. DocsAssistant’s pgvector table was created for nomic-embed-text, which produces 768 dimensions. Vectors of 384 do not fit that column, and vectors from different models are not comparable anyway. Switching the embedding model means a new table and re-embedding the corpus.
Downloads at first use. With no configuration, the model and tokenizer are fetched from GitHub the first time they are used, and the run above also downloaded PyTorch native libraries from publish.djl.ai. That is fine on a laptop and unacceptable in a locked-down or air-gapped environment. Package the files and point the model at them, as below, and check which native libraries your platform still needs at run time.
Cache directory. Downloaded files go to spring.ai.embedding.transformer.cache.directory, by default under the temporary directory. In a container with a read-only root file system this must be a writable volume.

LocalEmbeddingConfig.java: ONNX files from the classpath, no download

package com.example.docsassistant;

import org.springframework.ai.document.MetadataMode;
import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.ai.transformers.TransformersEmbeddingModel;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;

/**
 * An in-process ONNX embedding model with the ONNX files packaged on the classpath, so nothing is downloaded at
 * start-up. The 384-dimension MiniLM output does not fit a table created for nomic-embed-text (768).
 */
@Configuration(proxyBeanMethods = false)
@Profile("onnx")
class LocalEmbeddingConfig {

    @Bean
    EmbeddingModel embeddingModel() throws Exception {
        TransformersEmbeddingModel model = new TransformersEmbeddingModel(MetadataMode.NONE);
        model.setTokenizerResource("classpath:/onnx/all-MiniLM-L6-v2/tokenizer.json");
        model.setModelResource("classpath:/onnx/all-MiniLM-L6-v2/model.onnx");
        model.setResourceCacheDirectory("build/onnx-cache");
        model.afterPropertiesSet();
        return model;
    }
}

Source: Spring AI: Transformers (ONNX) Embeddings.

The class compiled; the packaged-files variant was not run, because the model file is not in the Spring AI jar and has to be supplied. The configuration form of the same thing is spring.ai.embedding.transformer.onnx.model-uri and spring.ai.embedding.transformer.tokenizer.uri, which accept classpath: and file URLs, and a different model such as an e5 variant can be set the same way. The chunking, vector-store and similarity side of embeddings is on Integrating with Spring AI.

What changes in DocsAssistant

The code of DocsAssistant stays the same; the model behind it is weaker, and a few behaviours that a hosted model hides show up.

Table 5. Local model behaviour to plan for
Area	What to do
Tool calling	Use a model that supports tools (`llama3.1:8b` does). Small models call the wrong tool or invent arguments more often, so keep tool descriptions short and precise and test them with the scripted-model tests on Testing
Structured output	Reliability varies by model; keep records small and add `StructuredOutputValidationAdvisor` for retries, see Structured Output
Context window	Set `num-ctx` (Ollama) or the server’s context flag; a value too small truncates silently. Keep RAG `topK` and chunk sizes in budget, and count tokens as on Chat Memory
Concurrency	A local server answers a few requests at a time and queues the rest. The rate limiter and concurrency cap on Production & Deployment protect it; timeouts must be longer than on a hosted model
Quality	Measure it. A local `llama3.1:8b` will not match a hosted frontier model on hard questions, so compare both on the same evaluation set from Evaluation
Privacy	Text stays on your machine, but prompt logging, memory tables and traces still store it. Logging flags are off by default, see Observability
Embeddings	Changing the embedding model invalidates the vector store; use the same model for indexing and querying

Local by profile, hosted by profile

Because the code is provider-neutral, the choice can be made by Spring profile. application-local.yml (above) selects Ollama; a hosted profile selects a hosted model with its key coming from the environment:

application-hosted.yml: the same application on a hosted model

spring:
  ai:
    model:
      chat: openai
    openai:
      api-key: ${OPENAI_API_KEY}
      chat:
        model: gpt-6-sol
      timeout: 60s
      max-retries: 3

Source: Spring AI: OpenAI Chat.

Run --spring.profiles.active=local on a laptop and hosted in a deployment that may send text to the provider. Models that appear only here as examples (gpt-6-sol and the others named in this guide) are placeholders for whatever your organisation has approved.

Where older material differs

Table 6. Older local-model material and the 2.0.1 line
Older text	Spring AI 2.0.1
`spring.ai.ollama.chat.options.model`, `.options.temperature`	`.options` is dropped: `spring.ai.ollama.chat.model`; the old form is deprecated but still accepted
`spring.ai.ollama.chat.think-option`	`spring.ai.ollama.chat.think`
OpenAI starter with `base-url: http://localhost:8080` and a `completions-path`	The official Java SDK appends `/chat/completions`, so the base URL carries `/v1`
No default temperature override: the starters used to set 0.7	The default is removed; each provider’s own default applies, so set `temperature` explicitly if a local model needs a value
Docker Model Runner not mentioned in Spring AI books	A documented route through the OpenAI starter, with a `DockerModelRunnerContainer` for tests

Local Models

Choosing a route

Ollama

Pull strategy

Keep-alive, context and thinking

Development services

Docker Model Runner

OpenAI-compatible servers

ONNX embeddings, in process

What changes in DocsAssistant

Local by profile, hosted by profile

Where older material differs

References