Production & Deployment

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written against Spring AI 2.0.1, Spring Boot 4.1.x and Java 21 (verified 2026-09-29). A model call is a slow, rate-limited, metered and sometimes failing remote request whose output changes with every run. In development that is a curiosity; under load it decides whether the service stays up and what it costs. This page takes DocsAssistant from a working sample to a service you can operate: what to retry and for how long, how to keep one busy client from starving the others, when a cached answer is safe, how the timeouts nest, how to package and run it on Kubernetes without baking in a secret, and how to estimate and govern the spend. The classes were compiled against 2.0.1 and the behaviours described as tested were run with scripted models, so they prove the mechanics of Spring AI and Spring, not the behaviour of any model provider under real load.

flowchart LR C["Client"] --> I["Ingress: timeout, TLS"] I --> S["Spring Security: identity"] S --> RL["Rate limiter: 429 when busy"] RL --> CC["Concurrency limit"] CC --> CA{"Cache hit?"} CA -- yes --> R["Answer"] CA -- no --> RT["Retry with backoff"] RT --> M["Model provider: timeout"] M --> R M -. "429/503" .-> RT

Retries

Two different retry mechanisms exist in 2.0.1 and the provider decides which one applies.

Providers built on Spring’s RestClient, such as Ollama, use the RetryTemplate that spring.ai.retry.* configures. It retries TransientAiException and ResourceAccessException (connection problems) and gives up immediately on anything else.
Providers built on an official vendor SDK, OpenAI and Anthropic among them since 2.0, use the SDK’s own retries, set per provider: spring.ai.openai.max-retries (default 3) and spring.ai.openai.timeout (default 60 seconds), and spring.ai.anthropic.max-retries and spring.ai.anthropic.timeout. The spring.ai.retry.* properties do not govern those calls.

Check which applies to your provider before tuning: setting spring.ai.retry.max-attempts while running on the OpenAI starter changes nothing.

What spring.ai.retry does

An HTTP error becomes a TransientAiException (retried) or a NonTransientAiException (not retried) according to these rules, applied in this order:

A status listed in spring.ai.retry.on-http-codes is always transient.
A 4xx status is non-transient unless spring.ai.retry.on-client-errors is true. So a 400 or a 401 fails at once, and a 429 fails at once too, unless you list it.
A status in spring.ai.retry.exclude-on-http-codes is non-transient.
Everything else, meaning 5xx, is transient.

Table 1. The spring.ai.retry properties and their defaults
Property	Default	Meaning
`max-attempts`	`10`	Number of retries, not attempts: the request is made once plus this many times
`backoff.initial-interval`	`2000ms`	First wait
`backoff.multiplier`	`5`	Each wait is the previous one times this
`backoff.max-interval`	`3m`	Ceiling on one wait
`on-client-errors`	`false`	Retry 4xx as well
`on-http-codes`	empty	Statuses to retry whatever their class, for example `429`
`exclude-on-http-codes`	empty	Statuses never to retry

Two of those defaults surprise people, and both were checked by running the auto-configured template. With max-attempts=3 a failing request is sent four times; a non-transient error is sent once. And the defaults sleep between attempts for seconds, 1322 seconds in total, about 22 minutes, before the caller sees the failure. A request thread, and a user, can wait that long on a provider that is down. The wait before retry (counting from 0) is

with the initial interval, the multiplier and the ceiling. The total sleeping time for retries is . If each call fails independently with probability , the expected number of calls per user request is

so with and it is about 1.25: retries cost 25% more calls when one call in five fails, and nothing when none do. During an outage approaches 1 and the cost approaches calls per request, which is one more reason to keep small.

A sensible interactive profile is three retries starting at one second and doubling, with a ceiling of ten seconds, and a 429 treated as transient:

application-prod.yml: retry, timeouts, tool limits and logging defaults

server:
  shutdown: graceful
spring:
  lifecycle:
    timeout-per-shutdown-phase: 45s
  threads:
    virtual:
      enabled: true
  http:
    clients:
      connect-timeout: 5s
      read-timeout: 120s
  reactor:
    context-propagation: auto
  ai:
    model:
      chat: ollama
    ollama:
      base-url: ${OLLAMA_BASE_URL}
      chat:
        model: llama3.1:8b
        keep-alive: 30m
      init:
        pull-model-strategy: never
    retry:
      max-attempts: 3
      on-http-codes: [429, 503]
      backoff:
        initial-interval: 1s
        multiplier: 2
        max-interval: 10s
    tools:
      limits:
        max-total-tool-calls: 20
        max-calls-per-tool-default: 5
    chat:
      observations:
        log-prompt: false
        log-completion: false
      client:
        observations:
          log-prompt: false
          log-completion: false
management:
  endpoints:
    web:
      exposure:
        include: health,prometheus
  endpoint:
    health:
      probes:
        enabled: true

Source: Spring AI: Upgrade Notes for the retry and tool-limit keys, and Spring Boot: Graceful Shutdown for the shutdown keys.

Every spring.ai.* key in the file was checked against the starters' configuration metadata. Worst-case sleeping with these values is 1 + 2 + 4 = 7 seconds. The Ollama URL comes from the environment; nothing secret is in the file.

A ceiling on total retry time

The properties cannot express a total deadline. Spring Framework 7’s RetryPolicy can (timeout(Duration)), and Spring AI’s template is @ConditionalOnMissingBean, so defining your own replaces it:

RetryConfig.java: retry at most three times, but never for more than 45 seconds

package com.example.docsassistant;

import java.time.Duration;

import org.springframework.ai.retry.TransientAiException;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.retry.RetryPolicy;
import org.springframework.core.retry.RetryTemplate;
import org.springframework.web.client.ResourceAccessException;

/**
 * Replaces the auto-configured template (which is @ConditionalOnMissingBean) to add what the properties cannot
 * express: a ceiling on the total time spent retrying one request.
 */
@Configuration(proxyBeanMethods = false)
class RetryConfig {

    @Bean
    RetryTemplate retryTemplate() {
        RetryPolicy policy = RetryPolicy.builder()
                .maxRetries(3)
                .includes(TransientAiException.class, ResourceAccessException.class)
                .delay(Duration.ofSeconds(1))
                .multiplier(2)
                .maxDelay(Duration.ofSeconds(8))
                .timeout(Duration.ofSeconds(45))   // give up after 45 s in total, whatever is left of maxRetries
                .build();
        return new RetryTemplate(policy);
    }
}

Source: Spring Framework: Resilience features.

A test with a 450 ms timeout, a 100 ms delay and up to 50 retries made 5 calls and then aborted with a message saying the pending sleep would exceed the timeout. The timeout limits the retry loop, not a single call that hangs, which is what the HTTP timeouts below are for. This bean applies to RestClient-based providers only.

Retrying the same request is safe for a chat call and not for a tool that changes something. A model call that fails midway may already have consumed tokens, and a tool such as openIssue that the model calls twice will create two issues unless it is idempotent. Give write tools a request key that makes a repeat harmless, or the approval queue described on Security & Guardrails will show the duplicate.

Retries can also be added around your own methods with Spring Framework 7’s @Retryable and @EnableResilientMethods, which is how the self-correcting evaluation loop on Evaluation works. That is a retry of a whole service call after a bad answer, a different thing from re-sending a request after a transport failure.

Rate limiting and concurrency

Providers limit requests and tokens per minute, and they answer 429 when you exceed the limit. A service should apply its own limit first, for three reasons: to stay under the provider’s quota, to stop one client using all of it, and to protect a local model server that answers only a few requests at a time. Spring AI has no rate limiter; use one from the Spring ecosystem. The example uses Resilience4j’s RateLimiter for a per-instance limit and Spring Framework 7’s @ConcurrencyLimit for the number of calls in flight. The limiter itself is explained on RateLimiter. For per-user token quotas that must hold across every replica (a bucket per user or API key, with each call costing the tokens it used), see Resilience4j versus Bucket4j.

ProductionConfig.java: a limiter bean and the switches for caching and resilience annotations

package com.example.docsassistant;

import java.time.Duration;

import io.github.resilience4j.ratelimiter.RateLimiter;
import io.github.resilience4j.ratelimiter.RateLimiterConfig;
import org.springframework.cache.annotation.EnableCaching;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.resilience.annotation.EnableResilientMethods;

@Configuration(proxyBeanMethods = false)
@EnableCaching
@EnableResilientMethods
class ProductionConfig {

    /** At most 5 model calls per second for this instance; a caller waits up to 2 s for a permit. */
    @Bean
    RateLimiter modelRateLimiter() {
        return RateLimiter.of("model", RateLimiterConfig.custom()
                .limitForPeriod(5)
                .limitRefreshPeriod(Duration.ofSeconds(1))
                .timeoutDuration(Duration.ofSeconds(2))
                .build());
    }
}

Source: Resilience4j: RateLimiter and Spring Framework: Resilience features.

CachedDocsAnswers.java: the anonymous answer path with a limit, a concurrency cap and a cache

package com.example.docsassistant;

import io.github.resilience4j.ratelimiter.RateLimiter;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.prompt.ChatOptions;
import org.springframework.cache.annotation.Cacheable;
import org.springframework.resilience.annotation.ConcurrencyLimit;
import org.springframework.stereotype.Service;

/**
 * Anonymous, stateless documentation answers: no memory, no per-user tools, temperature 0, so an identical
 * question is a safe cache hit. Per-user conversations never go through this class.
 */
@Service
public class CachedDocsAnswers {

    private final ChatClient chatClient;
    private final RateLimiter rateLimiter;

    public CachedDocsAnswers(ChatClient.Builder builder, DocsTools docsTools, RateLimiter modelRateLimiter) {
        this.chatClient = builder
                .defaultSystem("Answer only from the documentation tool results. Be brief.")
                .defaultOptions(ChatOptions.builder().temperature(0.0).maxTokens(400))
                .defaultTools(docsTools)
                .build();
        this.rateLimiter = modelRateLimiter;
    }

    @Cacheable(cacheNames = "docs-answers", key = "#question.strip().toLowerCase()")
    @ConcurrencyLimit(8)   // at most 8 model calls in flight in this instance
    public String answer(String question) {
        // throws RequestNotPermitted when no permit arrives within the timeout
        return RateLimiter.decorateSupplier(this.rateLimiter, () -> this.chatClient.prompt()
                .user(question)
                .call()
                .content()).get();
    }
}

Source: Spring Boot: Caching and Spring Framework: Resilience features for @ConcurrencyLimit.

When the limiter cannot grant a permit within its timeout it throws RequestNotPermitted. Turn that into an honest HTTP answer with a Retry-After header, not a 500:

RateLimitAdvice.java: 429 instead of 500

package com.example.docsassistant;

import io.github.resilience4j.ratelimiter.RequestNotPermitted;
import org.springframework.http.HttpStatus;
import org.springframework.http.ProblemDetail;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice;

@RestControllerAdvice
class RateLimitAdvice {

    @ExceptionHandler(RequestNotPermitted.class)
    ResponseEntity<ProblemDetail> tooManyRequests(RequestNotPermitted ex) {
        ProblemDetail problem = ProblemDetail.forStatusAndDetail(HttpStatus.TOO_MANY_REQUESTS,
                "The assistant is busy; retry shortly.");
        return ResponseEntity.status(HttpStatus.TOO_MANY_REQUESTS)
                .header("Retry-After", "2")
                .body(problem);
    }
}

Source: Resilience4j: RateLimiter.

Per-instance limits multiply with the number of replicas: five per second on each of three pods is fifteen per second against the provider. If the provider’s quota is global, divide it by the replica count or enforce the limit in a shared gateway, such as the ones on LLM Gateways. Limit by user or tenant too, not only per instance, when one client must not consume the budget of the rest.

The limiter grants a fixed number of permits per refresh period (limitForPeriod per limitRefreshPeriod), and a caller that cannot get one within timeoutDuration is refused. A short timeout is deliberate: a queue with no bound only turns overload into slow failures.

Caching

A cache is safe where the same input must give an acceptable answer for anyone who asks. For DocsAssistant that is the anonymous question "what does keep-alive do?", and it is not "summarise my open issues". The rules for @Cacheable on a model call:

Deterministic setup. Use a temperature of 0 and no per-user context. A cached answer is one sampled reply, and with a higher temperature it would freeze one arbitrary reply for everyone.
No memory, no per-user tools. Anything derived from the caller’s identity or history makes the same question produce different right answers. The CachedDocsAnswers class above has neither and is a separate path from the chat endpoint; conversations never go through it.
A normalised key. The key strips and lower-cases the question. Without it, What is keep-alive? and `what is KEEP-ALIVE ` would be two cache entries; matching questions that differ by more than case and spacing is semantic caching, below.
An expiry and an eviction. Cache for as long as the documentation stays the same, and clear the cache when the corpus is re-ingested, or answers describe pages that no longer say that.
Tools with side effects never run from a cache hit, because the cache skips the call entirely. Only cache paths whose tools read.

Caffeine as the cache provider

spring:
  cache:
    type: caffeine
    cache-names: docs-answers
    caffeine:
      spec: maximumSize=1000,expireAfterWrite=1h

Source: Spring Boot: Caching.

A test with a counting model showed the mechanics: two spellings of one question made one model call, and a limiter with two permits refused the third distinct question with RequestNotPermitted while the model had been called twice. Spring’s cache abstraction, providers and pitfalls are on Spring Boot Caching.

A semantic cache treats near-identical questions as the same by comparing embeddings. It hits more often and it can return a wrong answer to a question that only looks similar, so it needs a similarity threshold you have measured. The design, the threshold trade-off and the invalidation problem are on Semantic caching. Separately, some providers cache the prompt prefix on their side and bill cached tokens at a lower rate; in 2.0 Usage exposes getCacheReadInputTokens() and getCacheWriteInputTokens() (populated by Anthropic, Bedrock Converse, OpenAI and Google GenAI, as far as each reports them), so you can see whether a long stable system prompt is being reused.

Timeouts

Every remote call needs a timeout, and the timeouts need to nest, or a user waits for a request the service has already given up on, or the service keeps spending tokens for a client that has left.

Table 2. Where each timeout is set
Layer	Setting	Default
OpenAI and Anthropic SDK calls	`spring.ai.openai.timeout`, `spring.ai.anthropic.timeout`	60 s for OpenAI; unset for Anthropic (SDK default)
`RestClient` providers such as Ollama	`spring.http.clients.connect-timeout`, `spring.http.clients.read-timeout`	none
Total retry time	`RetryPolicy.timeout`, in `RetryConfig` above	none (`PT0S`)
Image generation on a slow model	the provider timeout above; an image call can take up to two minutes, see Image Generation	60 s for OpenAI
Servlet async request	`spring.mvc.async.request-timeout`	the container’s default; it did not cut an SSE `Flux` in a test
Reverse proxy or ingress	its own read timeout	set by your platform

The worst case for one request with a read timeout , retries and sleeps is

With read-timeout: 120s and the retry profile above that is 4 × 120 + 7 = 487 seconds, far above any proxy timeout, so the caller sees a 504 while the service goes on retrying. For interactive traffic choose and so that fits inside the proxy’s timeout, or add the total deadline from RetryConfig. Streaming answers are different: the connection stays open for as long as tokens keep arriving, so use an idle timeout, not a total one; see Streaming & Reactive.

Virtual threads

A blocking model call spends nearly all its time waiting, which is what virtual threads are for. With Java 21 set spring.threads.virtual.enabled=true and Boot runs request handling and @Async work on virtual threads, so a few hundred slow model calls in flight do not need a few hundred platform threads. It removes the thread pool as the limit; it does not remove the provider’s limit, so the rate limiter and concurrency cap above are still needed. Two details from the streaming work apply:

Reactor’s boundedElastic scheduler runs on platform threads unless you start the JVM with -Dreactor.schedulers.defaultBoundedElasticOnVirtualThreads=true, which in a test produced loomBoundedElastic virtual threads.
Tools run on Reactor threads while streaming, and the SecurityContext does not follow them unless spring.reactor.context-propagation=auto is set; without it a @PreAuthorize check failed with "An Authentication object was not found". The prod profile above sets it.

Spring Boot’s general treatment of threading is on Concurrency Alternatives.

Containers and Kubernetes

DocsAssistant is a stateless Spring Boot service, so it packages like any other. Build an image with the buildpacks (mvn spring-boot:build-image) or a layered Dockerfile as described on Java and Spring Boot Images, and scan and harden it as on Image Best Practices and Security. Nothing model-specific goes into the image: no key, no model file, and no ONNX download at first start (package the files, as on Local Models).

The Deployment below shows the parts specific to an AI service: secrets from secretKeyRef and not from the manifest, probes on the Actuator liveness and readiness groups, a termination grace period longer than the graceful-shutdown timeout, a non-root user with a read-only root file system and a writable /tmp, and resource limits.

docs-assistant.yaml: Deployment, Service and autoscaler

apiVersion: apps/v1
kind: Deployment
metadata:
  name: docs-assistant
  labels:
    app: docs-assistant
spec:
  replicas: 3
  selector:
    matchLabels:
      app: docs-assistant
  template:
    metadata:
      labels:
        app: docs-assistant
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: app
          image: registry.example.com/docs-assistant:1.0.0
          ports:
            - containerPort: 8080
          env:
            - name: SPRING_PROFILES_ACTIVE
              value: prod
            - name: OLLAMA_BASE_URL
              value: http://ollama.ai.svc.cluster.local:11434
            - name: SPRING_DATASOURCE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: docs-assistant-db
                  key: password
            - name: GITHUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: docs-assistant-github
                  key: token
          resources:
            requests:
              cpu: 500m
              memory: 768Mi
            limits:
              memory: 1Gi
          startupProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            failureThreshold: 30
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            periodSeconds: 5
          securityContext:
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: docs-assistant
spec:
  selector:
    app: docs-assistant
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: docs-assistant
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: docs-assistant
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Source: Spring Boot: Kubernetes probes and Kubernetes: Secrets.

The YAML parses and follows the Kubernetes API shapes; it was not applied to a cluster. The image name is a placeholder. Points that matter for AI workloads:

Readiness must not depend on the model. If readiness calls the provider, a provider outage takes every pod out of service at once and turns a partial failure into a full one. Keep the model out of the probes, serve a clear error when it fails, and alert on the error rate instead.
Graceful shutdown. server.shutdown=graceful and spring.lifecycle.timeout-per-shutdown-phase=45s let in-flight answers finish. The grace period (60 s) is longer than that, and an answer that takes longer is cut off, so keep the two aligned with your model timeouts.
Scale on load, not only CPU. The service spends its time waiting for the model, so CPU stays low while requests queue. Scale on concurrent requests or request rate from the metrics on Observability, within the provider’s quota.
A self-hosted model is a separate workload, with GPUs, warm-up and its own scaling; see Running local LLMs in production. The application reaches it by service name, as OLLAMA_BASE_URL does above.
Egress. If pods may reach only the model provider and the MCP servers they use, a prompt-injected instruction to send data elsewhere fails at the network.

The Kubernetes basics are on Orchestration: Swarm and Kubernetes; for Azure see AKS Clusters and AKS Microservices, and the managed alternative on Container Apps.

Secrets

The rule is that no secret is written in a file that is committed, baked into an image, printed in a log or sent to the model. In practice:

Provider keys and tokens (OPENAI_API_KEY, GITHUB_TOKEN for the MCP client) reach the process as environment variables, injected from a secret store by the platform: Kubernetes Secrets synced from a vault, or a CSI driver. On Azure that is Key Vault. Properties refer to them as ${OPENAI_API_KEY}.
Use a separate key per environment and per service, with the smallest scope the provider offers and a spending limit, so a leaked key has a ceiling.
Rotate keys on a schedule and on suspicion; a rotation should need a restart at most, not a rebuild.
Keep keys out of prompts and out of tool results. A system prompt that contains a credential can be read back by the model.
Turn prompt and completion logging off (the prod profile does), since prompts hold user text and tool results hold whatever the tools returned.
Never pass the caller’s token through to a tool server, see MCP Client.

What it costs

A hosted model bills by tokens, input and output at different rates. For one request with input tokens and output tokens, at prices and per million tokens,

Retries and caching change the effective cost per user request. With the expected calls from above and a cache hit rate ,

Take illustrative prices of 1 and 4 per million input and output tokens (not any vendor’s), a request of 1500 input and 300 output tokens, a 20% failure rate with three retries, and a 30% hit rate on the anonymous path. Then , , and per request. The arithmetic is simple and the inputs are what matter: measure and from the token counters (gen_ai.client.token.usage) instead of guessing, remember that grows with the memory window and with every retrieved chunk (the window arithmetic is on Chat Memory), and that an agent multiplies the number of calls per user request. Ways to lower the bill, roughly in order of payoff: shorter prompts and a smaller topK, a cheaper model for classification and routing, the cache, the provider’s prompt cache, and caps on output length (maxTokens) and on tool calls (spring.ai.tools.limits.*). Latency and throughput trade-offs are on Cost, Latency and Throughput, and cloud budgets on Cost Management.

Finally, set a hard budget outside the application, at the provider or the gateway. A retry loop or an agent bug can spend money faster than a dashboard shows it.

Governance and LLMOps

A model in production is a dependency that changes without a release: the provider can update a model behind a name, deprecate it, or change its filters. The checklist below is the minimum for a service that shows model output to users. The disciplines behind it, evaluation pipelines, prompt and model versioning, drift monitoring and incident response for AI systems, are the subject of LLMOps: Operational Governance and LLMOps: Prompt & Model Management, and the security side is the AI Security & Responsible AI sub-section, also planned separately.

Table 3. A production checklist for DocsAssistant
Area	What to have in place
Versions	Pin the model name and the Spring AI BOM version; upgrade deliberately, with the evaluation suite as the gate. Record which model version produced which answer
Evaluation in CI	A fixed question set scored by evaluators on every change to prompts, tools or the model, see Evaluation
Change control	Prompts and tool descriptions are code: review them and deploy them like code
Data handling	Decide what user text may be sent to which provider, how long memory and logs are kept, and how a user’s data is deleted; document the provider’s retention terms
Human oversight	Write tools go through an approval step; someone owns the approval queue and answers for it
Transparency	Users are told they are talking to an AI, and generated answers show their sources where the corpus provides them
Monitoring	Latency, error rate, token use, cost and refusal rate on a dashboard with alerts; sampled review of real conversations under the privacy rules above
Incidents	A way to switch the model, disable a tool or turn the assistant off without a deployment, and a named person who can
Compliance	Check which regulations apply to the use case, for example the EU AI Act, and whether the service falls in a regulated category; ask legal, this guide gives no legal advice

Where older material differs

Table 4. Older production material and the 2.0.1 line
Older text	Spring AI 2.0.1 and Spring Framework 7
`spring-retry`, `@EnableRetry`, `@Retryable` from `org.springframework.retry`	Framework 7 has its own: `org.springframework.core.retry.RetryTemplate` and `RetryPolicy`, and `@Retryable` with `@EnableResilientMethods` in `org.springframework.resilience.annotation`; Spring AI’s template is the core one
`spring.ai.retry.*` governs every provider	Only the `RestClient`-based ones; OpenAI and Anthropic use the SDK’s `max-retries` and `timeout`
`spring.http.client.read-timeout`	`spring.http.clients.read-timeout` in Boot 4
Default temperature 0.7 in the starters	Removed; the provider’s own default applies, so set `temperature` for cacheable paths
`@Async` and thread pools sized for model calls	Virtual threads with a concurrency cap, `spring.threads.virtual.enabled=true`
A tool loop inside every model call, unbounded	`ToolCallingAdvisor` with call limits: 150 in total and 40 per tool by default in 2.0.1, configurable with `spring.ai.tools.limits.*`

References

Spring AI: Upgrade Notes (retry, tool call limits, defaults, cache usage metrics)
Spring AI: Tool Calling
Spring AI: Observability
Spring Framework: Resilience features
Spring Boot: Task Execution and Scheduling (virtual threads)
Spring Boot: Caching
Spring Boot: Graceful Shutdown
Spring Boot: Kubernetes probes
Resilience4j: RateLimiter
Kubernetes: Secrets
OWASP: Top 10 for LLM Applications