Reasoning Models

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Examples on this page target the anthropic Python SDK 1.8.0 and claude-sonnet-5 (adaptive thinking, on by default, high effort by default), the openai Python SDK 3.19.2 and a current GPT-6-family reasoning model (gpt-6-sol, reasoning.effort), and Ollama 0.34.x with qwen3:8b, a thinking-capable local model alongside DocsAssistant’s usual llama3.1:8b chat model (The Model Landscape covers the model landscape more broadly). All parameter names and model IDs below were re-verified against each vendor’s own documentation on 2026-09-28; where a name has changed recently, this page says so explicitly rather than guessing.

Thinking: extended and adaptive reasoning

A model that answers in a single forward pass has to get everything right on the first try — no scratch work, no checking an intermediate result, no abandoning an approach that turns out to be wrong. Thinking (also called extended reasoning) removes that constraint: before producing its final answer, the model generates a separate stream of reasoning tokens in which it restates the problem, tries an approach, checks its own intermediate results, and backtracks when something doesn’t hold up. That reasoning is billed as output tokens and shows up in the response as one or more dedicated blocks, distinct from the final answer text.

Two designs for turning this on exist across vendors and even across a single vendor’s own model generations:

  • Manual, budget-driven thinking — the caller sets a fixed token budget for reasoning up front, and the model always spends against it. Anthropic calls this extended thinking (thinking.type: "enabled" with budget_tokens); it is the only mode some older reasoning-capable models support.

  • Adaptive, effort-driven thinking — the model decides for itself whether to think at all and how deeply, steered by a separate, coarser effort knob rather than a token count. Anthropic’s current generation (claude-sonnet-5 and later) calls this adaptive thinking; OpenAI’s reasoning.effort parameter is the same idea under a different name.

The rest of this page covers both designs, because "reasoning model" in practice can mean either one depending on which model and vendor a request targets.

Anthropic: adaptive thinking, and printing thinking and text blocks separately

On claude-sonnet-5, thinking is on by default and needs no parameter to enable it, but the reasoning text is hidden by default (the display field defaults to "omitted" on this model generation). Opting in with display: "summarized" returns a human-readable summary of the model’s reasoning — never the raw internal chain of thought — as separate thinking content blocks ahead of the final text block:

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=4096,
    thinking={"type": "adaptive", "display": "summarized"},
    messages=[
        {
            "role": "user",
            "content": (
                "DocsAssistant question: our HNSW index's recall dropped after we raised "
                "ef_construction. Should ef_search also change, and why?"
            ),
        }
    ],
)

for block in response.content:
    if block.type == "thinking":
        print("Thinking:\n", block.thinking)
    elif block.type == "text":
        print("\nAnswer:\n", block.text)

The response’s content array holds the two block types side by side: a thinking block (with an encrypted signature field that must be passed back unchanged in any follow-up turn) and, after it, the ordinary text block with the answer DocsAssistant would show the user. See the Anthropic Thinking documentation, which this example and the block shapes above derive from.

Extended thinking’s manual type: "enabled" plus budget_tokens form still exists, but its status is model-dependent and changed recently enough to call out explicitly: it is the only thinking mode on Claude Opus 4.5 and earlier reasoning-capable models, it is deprecated but still functional on the Claude 4.6 generation, and Claude 4.7 and later models — including claude-sonnet-5 — reject it with an HTTP 400 error. A request against an older, extended-thinking-only model looks like this instead:

response = client.messages.create(
    model="claude-opus-4-5",  # extended-thinking-only model
    max_tokens=16000,
    thinking={"type": "enabled", "budget_tokens": 10000},  # rejected on 4.7+ models
    messages=[{"role": "user", "content": "..."}],
)

budget_tokens must be at least 1,024 and less than max_tokens (the one exception is interleaved thinking, covered below, where the budget can span an entire turn’s tool calls). See the Anthropic Extended Thinking documentation for the full budget rules and the migration mapping from budget_tokens to adaptive thinking plus effort.

Effort and budget parameters

"How hard should the model try" is governed by two different, vendor-specific knobs, and the exact parameter names are worth getting right because they move fast:

Vendor Parameter (verified 2026-09-28) Notes

Anthropic (extended thinking, older models)

thinking: \{"type": "enabled", "budget_tokens": N\}

A hard token budget for reasoning; deprecated on the 4.6 generation, rejected on 4.7 and later.

Anthropic (adaptive thinking, current models)

output_config: \{"effort": "…​"\} on the request (a per-message output_config on an effort-only system message is in beta on some models)

A coarse effort level (low, medium, high, xhigh, max), not a token count; steers whether and how deeply the model thinks. claude-sonnet-5 defaults to "high".

OpenAI

reasoning: \{"effort": "…​"\} in the Responses API (i.e. reasoning.effort, not reasoning_effort)

Allowed values are model-dependent. The API defines none, minimal, low, medium, high, xhigh and max; the current GPT-6 and GPT-5.6 models accept none through max but not minimal, and GPT-6 Astra also rejects none.

An Anthropic request that lowers effort on the baseline model for the same kind of cheap classification task:

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    output_config={"effort": "low"},  # adaptive thinking stays on, but thinks less (or not at all)
    messages=[
        {
            "role": "user",
            "content": (
                "Classify this DocsAssistant page as one of: foundations, similarity-search, rag, "
                "agents, frameworks, operations. Page title: 'HNSW Indexes'."
            ),
        }
    ],
)

for block in response.content:
    if block.type == "text":
        print(block.text)

See the Anthropic Effort documentation for output_config.effort, its allowed levels and each model’s default.

An OpenAI reasoning request with an explicit effort level, run against DocsAssistant’s page-classification task:

from openai import OpenAI

client = OpenAI()

response = client.responses.create(
    model="gpt-6-sol",  # current OpenAI reasoning model, verified 2026-09-28
    reasoning={"effort": "low"},  # cheap, fast classification -- no need for deep reasoning here
    input=[
        {
            "role": "user",
            "content": (
                "Classify this DocsAssistant page as one of: foundations, similarity-search, rag, "
                "agents, frameworks, operations. Page title: 'HNSW Indexes'."
            ),
        }
    ],
)

print(response.output_text)

See the OpenAI Reasoning models guide, which this reasoning.effort shape and the current allowed levels derive from.

Because Anthropic’s effort and thinking are separate controls — thinking decides whether the model reasons in dedicated blocks, effort decides how much work the whole response gets, thinking included — lowering effort is the first lever to pull for cost or latency, and raising it is the first lever to pull when the model is thinking too little or too shallowly. Sampling & Decoding covers which other sampling parameters a reasoning model ignores or restricts once thinking is active.

Interleaved thinking with tools

Interleaved thinking lets a reasoning model think again between tool calls in the same turn: it reasons about a tool’s result before deciding whether to call another tool or answer. Without it, a model that uses tools reasons once up front and then executes a fixed plan of tool calls; with it, each tool result can change what the model does next, which matters for a DocsAssistant agent that retrieves a document, notices the retrieved chunk doesn’t actually answer the question, and decides to retrieve again with a different query before answering.

On Anthropic’s current adaptive-thinking models, interleaving happens automatically, with no extra parameter. Older models (Claude Opus 4.5, Claude Sonnet 4.5 and earlier Claude 4 models) need the interleaved-thinking-2025-05-14 beta header on the request to get the same behaviour under manual extended thinking. See the Anthropic Thinking documentation’s interleaved-thinking section for the turn structure and the per-model beta-header requirement.

Reasoning locally with Ollama

Ollama exposes thinking as a single request field, think, which accepts true, false, null (model default) or a named level such as "low" / "medium" / "high" on models that support graded effort. As of 2026-09-28, Ollama’s own thinking documentation lists qwen3, gpt-oss and deepseek-r1 as thinking-capable model families; this page uses qwen3:8b, a size that runs comfortably on a laptop, as DocsAssistant’s local reasoning model, alongside the plain llama3.1:8b chat model this section uses elsewhere for fast, non-reasoning requests:

from ollama import chat

response = chat(
    model="qwen3:8b",
    messages=[
        {
            "role": "user",
            "content": "Why would raising ef_construction on an HNSW index also call for retuning ef_search?",
        }
    ],
    think=True,
)

print("Thinking:\n", response.message.thinking)
print("\nAnswer:\n", response.message.content)

The equivalent request against Ollama’s native /api/chat endpoint with curl:

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [{"role": "user", "content": "Why retune ef_search after raising ef_construction?"}],
  "think": true,
  "stream": false
}'

See the Ollama thinking documentation, which the think parameter, its accepted values and the model list above derive from.

When reasoning helps — and when it only adds latency and cost

Thinking is not a free quality upgrade. It trades latency and token spend (reasoning tokens are billed as output tokens even when hidden from the response) for a chance at a better answer, and that trade only pays off on certain kinds of requests:

  • Reasoning tends to help on multi-step math and logic, non-trivial debugging, planning that requires weighing several options against constraints, and any DocsAssistant question whose answer depends on combining several retrieved facts rather than restating one of them (for example, "which of these three index types should we use given our recall and latency targets" rather than "what does HNSW stand for").

  • Reasoning tends to only add cost on lookup-style requests that a fast model already answers correctly: classification against a short fixed label set, extracting a field that is already present verbatim in the input, simple rephrasing or summarisation, and any request running at high volume and tight latency budgets where the fast model’s accuracy is already acceptable. Escalating from a fast model to a reasoning model only on low-confidence or failed responses — the cascade pattern Choosing a Model covers — gets most of the quality benefit without paying the reasoning cost on every request.

Prompting reasoning models: state goals and constraints, don’t script the steps

A reasoning model already plans its own intermediate steps; writing it a numbered chain-of-thought recipe — "first do X, then do Y, then check Z" — fights the thing the model is designed to do on its own and, in practice, tends to produce worse results than simply stating what a good answer looks like. The prompting shift this page’s techniques companion, Prompting Techniques, calls out for reasoning models specifically is: describe the goal, the constraints and any facts the model must ground its answer in, and let the model decide how to get there, rather than hand-authoring the chain of thought yourself the way a non-reasoning model often needs.

A DocsAssistant example of the shift:

  • Non-reasoning-style prompt (scripts the steps): "First list every index type mentioned in the retrieved chunks. Then, for each one, note its build time and query latency. Then compare them pairwise. Then pick the best one and explain why."

  • Reasoning-model prompt (states goals and constraints): "Given the retrieved chunks about HNSW, IVF and flat indexes, recommend one index type for a corpus of 2 million vectors with a 50ms p99 query latency budget. State the trade-off you’re accepting."

The second prompt gives the model the outcome and the constraints it must satisfy, and leaves the decomposition — which facts to compare, in what order — to the model’s own thinking.

Choosing between a reasoning model and a fast model

Use a reasoning model when…​ Use a fast (non-reasoning) model when…​

The task has several steps that depend on each other (planning, multi-hop retrieval, debugging)

The task is a single lookup, extraction or classification against a short, well-defined answer

Getting the answer wrong is costly, and a slower, more expensive answer is acceptable

The request runs at high volume, or a user is waiting on a tight latency budget

The input requires weighing trade-offs or constraints against each other

The input already contains the answer verbatim, or the mapping from input to output is close to mechanical

Early testing on your own evaluation set shows the fast model missing cases the reasoning model gets right

Your evaluation set shows the fast model already at acceptable accuracy for the task

The workload can tolerate escalation latency (a cascade’s second call)

The workload is the first call in a cascade, deciding whether escalation is even needed

Cost, Latency & Throughput covers the concrete latency and pricing impact of turning thinking on, and how a lower effort level or a smaller model trades against it.

Related pages

References