Chat Options

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written against Spring AI 2.0.1, Spring Boot 4.1.x and Java 21 (verified 2026-09-29). This page gives the running DocsAssistant service control over how the model answers: a deterministic setting for the extraction endpoint added later, a slightly warmer one for chat, and a reasoning setting for the occasional hard question. It uses the client from ChatModel & ChatClient. The maths of sampling is explained once, in Sampling & Decoding; here it is only what the parameters are called and where they are set. The layering rules below were observed by running the client against a stub ChatModel and reading the 2.0.1 sources, because no model server was available when the page was written.

Portable and provider options

ChatOptions is the portable options interface. It exposes eight values that almost every chat provider understands.

Table 1. The portable options in ChatOptions
Option Type Meaning

model

String

The model name, for example llama3.1:8b

temperature

Double

Randomness of sampling: low is focused and repeatable, high is varied

topP

Double

Nucleus sampling: sample only from the smallest set of tokens whose probability adds up to this value

topK

Integer

Sample only from the K most likely tokens

maxTokens

Integer

Upper bound on the generated tokens

stopSequences

List<String>

Strings that end generation when produced

frequencyPenalty, presencePenalty

Double

Discourage repeating tokens or topics

Not every provider honours every portable option; a value a provider does not support is ignored or rejected by that provider, so read its page in the reference. The interface offers ChatOptions.builder() for building them and mutate() on an instance to derive a builder from it.

Each provider also has its own subclass that adds what only that provider has: OllamaChatOptions (numCtx, keepAlive, seed, minP and the reasoning switches), OpenAiChatOptions (reasoningEffort, verbosity, maxCompletionTokens, responseFormat, seed), AnthropicChatOptions (thinking, effort, caching and web-search settings) and GoogleGenAiChatOptions (thinking budget and level). Using them makes the code provider-specific, which is a real cost if you might switch models; keep provider options in configuration or in a small adapter class, and use the portable interface everywhere else.

OptionsExamples.java: portable defaults and Ollama-specific options
package com.example.docsassistant;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.prompt.ChatOptions;
import org.springframework.ai.ollama.api.OllamaChatOptions;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
class OptionsExamples {

    // Portable options: work with any provider.
    @Bean
    ChatClient deterministicClient(ChatClient.Builder builder) {
        return builder
                .defaultOptions(ChatOptions.builder()
                        .temperature(0.0)
                        .maxTokens(600))
                .build();
    }

    // Provider options: Ollama's num_ctx and keep_alive next to the portable ones.
    OllamaChatOptions ollamaOptions() {
        return OllamaChatOptions.builder()
                .model("llama3.1:8b")
                .temperature(0.2)
                .numCtx(8192)
                .keepAlive("10m")
                .build();
    }
}

Ollama’s numCtx is the size of the context window the runtime allocates, and Ollama’s own default is modest, so a long prompt with retrieved passages can be silently truncated unless numCtx (or the runtime’s default) is raised; sizing it is part of Local Models. Source: Spring AI: Chat Model API.

Three layers of configuration

A request’s options come from three places, from the most general to the most specific.

  1. The model’s start-up options, built from the spring.ai.<provider>.chat.* properties, for example spring.ai.ollama.chat.temperature.

  2. The client’s default options, set with ChatClient.Builder.defaultOptions(…​).

  3. The request’s options, set with chatClient.prompt().options(…​).

ChatClient builds the effective options by starting from the model’s start-up options and layering the client’s customizer on top; only the values that were actually set replace the model’s (stop sequences are the exception: they are appended). That makes it a "delta" mechanism: you name what changes and the rest stays as configured. In Spring AI 2.0 the framework no longer forces a default temperature of 0.7, so a value you do not set is the provider’s own default (see the Spring AI: Upgrade Notes).

The one rule that surprises people is that a request-level options(…​) replaces the client’s defaultOptions(…​) rather than merging with it. With a stub model and a client whose default was temperature(0.0), three requests produced the following effective values.

Table 2. Effective options for three requests on a client with defaultOptions(temperature 0.0)
Request temperature maxTokens

no request options

0.0 (the client default)

not set

options(ChatOptions.builder().maxTokens(50))

not set (the client default was replaced)

50

options(ChatOptions.builder().temperature(0.9))

0.9

not set

The second row shows the trap: the request set only maxTokens, and the client’s temperature(0.0) was gone. If a request needs the client defaults plus something else, repeat the defaults or derive the options from a shared base with mutate() (below). Both layers still sit on top of the model’s start-up options, which is why an unset value falls back to the properties rather than to nothing. When you skip ChatClient and call ChatModel.call(Prompt) directly, the options on the Prompt are used as a whole and replace the model’s start-up options entirely, so a prompt with only temperature set loses the configured model name and every other default; pass a complete options object there. Sources: Spring AI: Chat Model API and Spring AI: Chat Client API.

Immutable options and mutate()

Options objects are immutable in Spring AI 2.0. The mutable copy constructors and copy() / fromOptions() methods of the 1.x line are gone; the way to derive a variant is mutate(), which returns a builder pre-filled with the current values. Because the original is untouched, a shared options bean is safe to hand to concurrent requests.

OptionVariants.java: deriving a variant with mutate()
package com.example.docsassistant;

import org.springframework.ai.ollama.api.OllamaChatOptions;

class OptionVariants {

    // The base is never modified: mutate() copies it into a new builder.
    OllamaChatOptions creativeVariant(OllamaChatOptions base) {
        return base.mutate().temperature(0.9).build();
    }
}

For a base with temperature 0.2, the variant had 0.9 and the base kept 0.2 in a test. getDefaultOptions() on a model is deprecated in favour of getOptions(), and N() became n() on the builders that have it, both from the 2.0 upgrade notes. Source: Spring AI: Upgrade Notes.

Per-request options on ChatClient

options(…​) on a request accepts an options builder, not a built options object; in 2.0.1 the method’s parameter is ChatOptions.Builder, and code that ends the chain with .build() (some examples in the reference and in blog posts do) does not compile against that signature. A provider builder is fine wherever a ChatOptions.Builder is expected.

PerRequestOptions.java: a one-off creative answer and a one-off deep-thinking answer
package com.example.docsassistant;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.prompt.ChatOptions;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.ollama.api.OllamaChatOptions;
import org.springframework.ai.openai.OpenAiChatOptions;

class PerRequestOptions {

    // Pass the builder, not the built options.
    String oneOffCreative(ChatClient chatClient, String question) {
        return chatClient.prompt()
                .user(question)
                .options(ChatOptions.builder().temperature(0.9).maxTokens(200))
                .call()
                .content();
    }

    // A provider-specific builder also works per request.
    String deepThought(ChatClient chatClient, String question) {
        return chatClient.prompt()
                .user(question)
                .options(OpenAiChatOptions.builder().reasoningEffort("high").maxCompletionTokens(4000))
                .call()
                .content();
    }

    // Options on the Prompt for a direct ChatModel call replace the model's defaults completely.
    String directCall(ChatModel chatModel, OllamaChatOptions options, String question) {
        return chatModel.call(new Prompt(question, options)).getResult().getOutput().getText();
    }
}

If several clients need different settings, the cleaner design is several ChatClient beans, each with its own defaultOptions, as in More than one ChatClient; a request override is for the exceptions. Source: Spring AI: Chat Client API.

Choosing values

Temperature rescales the model’s raw scores before they become probabilities. For raw scores over the vocabulary,

so a temperature below 1 sharpens the distribution toward the top token and is (in practice) greedy decoding, while above 1 flattens it. Top-p and top-k then cut the tail before sampling. For DocsAssistant the working defaults are below; the right values depend on the model, so treat them as starting points and confirm with the tests in Evaluation.

Table 3. Starting points for DocsAssistant
Endpoint temperature maxTokens Why

/api/extract (structured metadata)

0.0

small

Repeatable, shape-sensitive output; see Structured Output

/api/chat

0.2

600

Factual answers with a little fluency; a hard cap keeps cost and latency predictable

Brainstorming a page outline

0.8

larger

Variety is the point

Set maxTokens deliberately: it bounds both cost and the time a request can hold a thread. It is one of the cheapest guardrails there is, and Cost, Latency & Throughput explains why output length dominates both.

Reasoning and effort

Reasoning models spend extra tokens thinking before they answer, and providers expose a control for how much. The knob has a different name and shape for each provider, so it is a provider option, not a portable one. The builder methods below were read from the compiled 2.0.1 classes and the properties from the starter metadata.

Table 4. Reasoning controls by provider in Spring AI 2.0.1
Provider Builder methods Property

OpenAI

reasoningEffort(String), maxCompletionTokens(Integer), verbosity(String)

spring.ai.openai.chat.reasoning-effort, spring.ai.openai.chat.max-completion-tokens

Anthropic

thinkingEnabled(long budgetTokens), thinkingAdaptive(), thinkingDisabled(), effort(OutputConfig.Effort) with LOW, MEDIUM, HIGH, XHIGH and MAX

spring.ai.anthropic.chat.max-tokens for the reply ceiling; thinking is configured in code

Ollama

thinkLow(), thinkMedium(), thinkHigh(), enableThinking(), disableThinking(), or thinkOption(…​)

spring.ai.ollama.chat.think with true, false, low, medium or high

Google GenAI

thinkingBudget(Integer), thinkingLevel(…​)

see the Google GenAI page of the reference
ReasoningOptions.java: reasoning settings for three providers
package com.example.docsassistant;

import com.anthropic.models.messages.OutputConfig;
import org.springframework.ai.anthropic.AnthropicChatOptions;
import org.springframework.ai.ollama.api.OllamaChatOptions;
import org.springframework.ai.openai.OpenAiChatOptions;

class ReasoningOptions {

    OpenAiChatOptions openAi() {
        return OpenAiChatOptions.builder()
                .model("gpt-6-sol")
                .reasoningEffort("medium")
                .maxCompletionTokens(2000)
                .build();
    }

    AnthropicChatOptions anthropicFixedBudget() {
        return AnthropicChatOptions.builder()
                .model("claude-sonnet-5")
                .maxTokens(4096)
                .thinkingEnabled(2000)
                .build();
    }

    AnthropicChatOptions anthropicAdaptive() {
        return AnthropicChatOptions.builder()
                .model("claude-sonnet-5")
                .maxTokens(4096)
                .thinkingAdaptive()
                .effort(OutputConfig.Effort.MEDIUM)
                .build();
    }

    OllamaChatOptions ollama() {
        return OllamaChatOptions.builder().model("qwen3:8b").thinkLow().build();
    }
}

All three compile against the 2.0.1 jars; whether a particular model accepts adaptive thinking, an effort level or a budget depends on that model, so check the provider’s documentation for the model you name. Thinking tokens are generally billed as output tokens and count toward maxTokens, so raise the ceiling when you enable thinking. Reasoning models can also emit their reasoning trace in the reply, which matters for structured output: the reference warns that Ollama models with a thinking mode can return their trace as plain text instead of JSON. How reasoning models work is covered in Reasoning Models. Sources: Spring AI: OpenAI Chat, Spring AI: Anthropic Chat and Spring AI: Ollama Chat.

Options from configuration

Everything above can be given as properties instead of code, which is usually the better home for model names and sampling defaults because it changes without a rebuild.

application.yml: chat options as properties
spring:
  ai:
    ollama:
      chat:
        model: llama3.1:8b
        temperature: 0.2
        num-ctx: 8192
        keep-alive: 10m
        think: low
    openai:
      chat:
        model: gpt-6-sol
        reasoning-effort: medium

Source: Spring AI: Ollama Chat and Spring AI: OpenAI Chat.

These keys use the flattened 2.0 paths. The old spelling with an .options segment (spring.ai.ollama.chat.options.temperature) still resolves as a deprecated alias, and some blog posts and the books use it; migrate when you see it. spring.ai.ollama.chat.think-option from 1.x is now spring.ai.ollama.chat.think. Each key above was found in the 2.0.1 configuration metadata.

References