Java & Spring Boot

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page was written on 2026-09-30 against Java 21, Spring Boot 4.1, Spring AI 2.0.1, LangChain4j 1.20.2 (the in-process embedding artifacts are published as 1.20.2-beta30), DJL 0.38.0, ONNX Runtime 1.30.0 (Maven Central), Jlama 0.8.4 and java-llama.cpp 4.2.0 (Maven Central) as checked that day. Java has no equivalent of the Python transformers library, so a Spring Boot service reaches Hub models in one of three ways: a small in-process model exported to ONNX (embeddings, classifiers, tokenizers), an HTTP call to a local server that already runs the model (the recommended path for chat LLMs), or a sidecar service for audio. This page walks through each option and says where it stops being a good idea. It continues the DocsAssistant goes offline scenario of this sub-section: transcribe a spoken question, answer it with a small local model, speak the answer, and classify or describe a screenshot. Model formats are in Model Formats; producing ONNX files is in Conversion & Quantization; the models themselves are in Small LLMs & Embeddings, Speech Models: STT & TTS and Vision Models. No benchmark numbers are quoted.

The shape of the integration

The JVM runs ONNX graphs well, runs PyTorch through DJL, and can call native llama.cpp through JNI, but the Hub’s Python-first tooling (pipelines, processors, audio feature extractors) has no Java port. The consequence is a split of responsibilities, shown below.

flowchart LR U["Browser or mobile client"] --> A["Spring Boot 4.1 app
Spring AI 2.0.1"] A -->|"in process"| E["ONNX embeddings
MiniLM via ONNX Runtime"] A -->|"HTTP, OpenAI or Ollama API"| L["Local LLM server
Ollama, vLLM or llama-server"] A -->|"HTTP"| S["Speech sidecar
Whisper and Kokoro"] A -->|"in process, optional"| V["ONNX classifier
for screenshots"] L --> H[("Hub weights
GGUF or safetensors")] S --> H E --> H
Figure 1. DocsAssistant on Spring Boot with no cloud dependency
Table 1. Which Java option fits which DocsAssistant task
Task Preferred Java option Alternative Main cost

Transcribe a spoken question

Spring AI TranscriptionModel with the OpenAI starter pointed at an OpenAI-compatible Whisper server

Whisper exported to ONNX and run with OrtSession

Sidecar to operate, or audio feature extraction and decoding loop written by hand

Answer it

Spring AI Ollama or OpenAI starter with a local base-url

ORT GenAI or java-llama.cpp in process

A second process to run, or a native library with its own release cadence

Speak the answer

HTTP call to a TTS sidecar (Kokoro or Piper behind an API)

Kokoro ONNX run with OrtSession

Phonemization and voice-pack handling are not in the ONNX file

Classify or describe a screenshot

A vision-capable chat model on the same local server

SigLIP 2 ONNX through ONNX Runtime or DJL

Image preprocessing has to match the model card exactly

Retrieve passages (embeddings)

In-process ONNX: Spring AI TransformersEmbeddingModel or LangChain4j

Embedding model on the local server

384 dimensions do not fit a table created for another model

The rest of the page takes the options in turn, from the libraries that run tensors, to the framework integrations, to the two JNI and pure-Java LLM engines, and finally to the recommended server path.

DJL: tokenizers, model zoo and the ONNX engine

Deep Java Library (DJL) is an engine-agnostic Java API with PyTorch, TensorFlow, ONNX Runtime and other engines behind it. Three parts matter for Hub models. The ai.djl.huggingface:tokenizers artifact wraps the Rust Hugging Face tokenizers library, so a Java service tokenizes text exactly as the model card’s Python code does, from the model’s tokenizer.json. The model zoo and Criteria load a model from a URL (djl://, file://, https://, jar://, s3:// and others) with a chosen engine. The ai.djl.onnxruntime:onnxruntime-engine artifact lets Criteria select the ONNX Runtime engine. All three were at 0.38.0 on Maven Central on 2026-09-30, and the DJL BOM keeps them aligned.

pom.xml: the DJL BOM with the tokenizers and ONNX Runtime engine
<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>ai.djl</groupId>
      <artifactId>bom</artifactId>
      <version>0.38.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>ai.djl</groupId>
    <artifactId>api</artifactId>
  </dependency>
  <dependency>
    <groupId>ai.djl.huggingface</groupId>
    <artifactId>tokenizers</artifactId>
  </dependency>
  <dependency>
    <groupId>ai.djl.onnxruntime</groupId>
    <artifactId>onnxruntime-engine</artifactId>
  </dependency>
</dependencies>

Source: DJL: Hugging Face tokenizers.

HuggingFaceTokenizer.newInstance accepts either a Hub model id (the tokenizer files are downloaded and cached) or a path to a local tokenizer.json. Prefer the local path in production so start-up does not depend on the network.

Tokenizing with the Hub tokenizer of a model
import ai.djl.huggingface.tokenizers.Encoding;
import ai.djl.huggingface.tokenizers.HuggingFaceTokenizer;
import java.nio.file.Paths;

try (HuggingFaceTokenizer tokenizer =
         HuggingFaceTokenizer.newInstance(Paths.get("models/all-MiniLM-L6-v2/tokenizer.json"))) {
    Encoding encoding = tokenizer.encode("How do I configure a Spring profile?");
    long[] ids = encoding.getIds();
    long[] attentionMask = encoding.getAttentionMask();
    String roundTrip = tokenizer.decode(ids);
}

Source: DJL: Hugging Face tokenizers.

Loading a model goes through Criteria. The DJL documentation shows a zoo URL (djl://ai.djl.pytorch/resnet) and an explicit engine selection with optEngine("OnnxRuntime"); for a Hub model the DJL tokenizers page documents the djl://ai.djl.huggingface.pytorch/ zoo and Hub URLs whose repository contains a tokenizer.json. Treat the exact zoo model names as something to look up in the DJL model zoo listing for your version, because the zoo is a curated subset of the Hub. The translator (how a String becomes tensors and back) must be supplied or inferred, which is the part that differs per task.

Loading a model with Criteria, from a local directory and with the ONNX Runtime engine
import ai.djl.inference.Predictor;
import ai.djl.repository.zoo.Criteria;
import ai.djl.repository.zoo.ZooModel;
import ai.djl.huggingface.translator.TextEmbeddingTranslatorFactory;
import java.nio.file.Paths;

Criteria<String, float[]> criteria = Criteria.builder()
        .setTypes(String.class, float[].class)
        .optModelPath(Paths.get("models/all-MiniLM-L6-v2"))   // model.onnx + tokenizer.json
        .optEngine("OnnxRuntime")
        .optTranslatorFactory(new TextEmbeddingTranslatorFactory())
        .build();

try (ZooModel<String, float[]> model = criteria.loadModel();
     Predictor<String, float[]> predictor = model.newPredictor()) {
    float[] vector = predictor.predict("How do I configure a Spring profile?");
}

Source: DJL: Load a model (the Criteria builder, engine selection and URL schemes); the TextEmbeddingTranslatorFactory name is taken from DJL’s Hugging Face extension and should be confirmed against the 0.38.0 Javadoc before use.

DJL’s strength is one API over several engines and the tokenizer; its cost is that each engine downloads or bundles native libraries (Spring AI’s ONNX embedding model uses the DJL tokenizer for this reason, see below), and the zoo covers far fewer models than the Hub.

ONNX Runtime Java

The ONNX Runtime Java API runs any ONNX graph, which makes it the direct route for models exported by optimum-onnx or downloaded from the onnx-community organisation (see Model Formats). The CPU artifact is com.microsoft.onnxruntime:onnxruntime; the CUDA artifact is com.microsoft.onnxruntime:onnxruntime_gpu. Both were at 1.30.0 on Maven Central on 2026-09-30, and the documentation states Java 8 or newer. You use one of the two artifacts, not both.

pom.xml: the ONNX Runtime CPU artifact (swap the artifactId for onnxruntime_gpu on CUDA machines)
<dependency>
  <groupId>com.microsoft.onnxruntime</groupId>
  <artifactId>onnxruntime</artifactId>
  <version>1.30.0</version>
</dependency>

Source: ONNX Runtime: Get started with Java.

The core types are OrtEnvironment (one per process), OrtSession (one loaded model, thread-safe for run), OrtSession.SessionOptions (threads, optimization level, execution providers), OnnxTensor (an input or output value) and OrtSession.Result (the outputs, to be closed). The example runs a MiniLM sentence encoder: it feeds the token ids and attention mask produced by the DJL tokenizer above and mean-pools the last hidden state.

Running a sentence-embedding ONNX model with OrtSession
import ai.djl.huggingface.tokenizers.Encoding;
import ai.djl.huggingface.tokenizers.HuggingFaceTokenizer;
import ai.onnxruntime.OnnxTensor;
import ai.onnxruntime.OrtEnvironment;
import ai.onnxruntime.OrtSession;
import java.nio.LongBuffer;
import java.nio.file.Paths;
import java.util.Map;

OrtEnvironment env = OrtEnvironment.getEnvironment();
try (OrtSession.SessionOptions options = new OrtSession.SessionOptions();
     OrtSession session = env.createSession("models/all-MiniLM-L6-v2/model.onnx", options);
     HuggingFaceTokenizer tokenizer =
         HuggingFaceTokenizer.newInstance(Paths.get("models/all-MiniLM-L6-v2/tokenizer.json"))) {

    Encoding enc = tokenizer.encode("How do I configure a Spring profile?");
    long[] ids = enc.getIds();
    long[] mask = enc.getAttentionMask();
    long[] shape = {1, ids.length};

    try (OnnxTensor idsTensor = OnnxTensor.createTensor(env, LongBuffer.wrap(ids), shape);
         OnnxTensor maskTensor = OnnxTensor.createTensor(env, LongBuffer.wrap(mask), shape);
         OnnxTensor typeTensor = OnnxTensor.createTensor(env, LongBuffer.wrap(new long[ids.length]), shape);
         OrtSession.Result result = session.run(Map.of(
                 "input_ids", idsTensor,
                 "attention_mask", maskTensor,
                 "token_type_ids", typeTensor))) {

        float[][][] hidden = (float[][][]) result.get(0).getValue();   // [1][tokens][384]
        float[] sentence = new float[hidden[0][0].length];
        for (int t = 0; t < ids.length; t++) {
            for (int d = 0; d < sentence.length; d++) {
                sentence[d] += hidden[0][t][d] / ids.length;           // mean pooling
            }
        }
    }
}

Source: ONNX Runtime: Get started with Java. Input names vary by export: read them with session.getInputNames() and follow the model card, and normalize the vector if the card says so. The pooling loop averages all tokens because this input has no padding; a batch needs the attention mask in the average.

Execution providers are configured on SessionOptions. The official Java page documents CPU and CUDA (options.addCUDA(0) with the onnxruntime_gpu artifact) and does not list the others for Java, so check the execution providers page for your platform before relying on CoreML, DirectML, TensorRT or OpenVINO from Java; a provider that is not compiled into the artifact throws at session creation.

Enabling the CUDA execution provider
try (OrtSession.SessionOptions options = new OrtSession.SessionOptions()) {
    options.addCUDA(0);                       // device id; needs the onnxruntime_gpu artifact and CUDA libraries
    OrtSession session = env.createSession("model.onnx", options);
}

Source: ONNX Runtime: Get started with Java.

The same machinery runs any other exported model, for instance the ONNX export of SigLIP 2 (onnx-community/siglip2-base-patch16-224-ONNX) for the screenshot task, but then the pre-processing (resize, normalization, patching) and post-processing are your code, written to match the model’s processor configuration.

ORT GenAI for Java

ONNX Runtime GenAI wraps the generation loop (tokenization, KV-cache, sampling) around ONNX models exported with its model builder, such as microsoft/Phi-4-mini-instruct-onnx. Its Java API lives in the ai.onnxruntime.genai package with Model, Tokenizer, GeneratorParams and Generator classes. As of 2026-09-30 the Java API page states that package publication is pending and says to build from source, and no onnxruntime-genai artifact exists under com.microsoft.onnxruntime on Maven Central. The project’s GitHub release v0.17.0 attaches an onnxruntime-genai-0.17.0.jar next to the native archives, but it is not a Maven dependency, so treat Java use as a manual-jar or build-it-yourself option and re-check the page before adopting it. The documented API is also described as being in preview.

Generating text with ORT GenAI (current source API; verify against the version you build)
import ai.onnxruntime.genai.*;

try (Model model = new Model("models/Phi-4-mini-instruct-onnx/cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4");
     Tokenizer tokenizer = new Tokenizer(model);
     GeneratorParams params = new GeneratorParams(model)) {

    params.setSearchOption("max_length", 512);

    try (Generator generator = new Generator(model, params)) {
        generator.appendTokenSequences(tokenizer.encode("Summarize Spring profiles in one sentence."));
        while (!generator.isDone()) {
            generator.generateNextToken();
        }
        String text = tokenizer.decode(generator.getSequence(0));
    }
}

Source: ONNX Runtime GenAI: Java API. The model folder name is an example; list the repository’s folders on the Hub to find the one for your hardware, and note that the published API page still shows the older GeneratorParams.setInput and computeLogits calls, while the current source uses Generator.appendTokenSequences as above, so the method names may differ from your build.

For a service that needs a chat LLM today, the server path below is less work than a source build.

Spring AI: TransformersEmbeddingModel

The Spring AI spring-ai-starter-model-transformers starter runs a sentence-transformer ONNX model in the JVM, using the DJL tokenizer and ONNX Runtime underneath. Its default model is sentence-transformers/all-MiniLM-L6-v2. The starter is selected with spring.ai.model.embedding=transformers, and the official page lists these properties.

Table 2. Properties of the ONNX embedding model
Property Meaning Default

spring.ai.embedding.transformer.onnx.model-uri

The ONNX model file (classpath, file or remote URL)

onnx/all-MiniLM-L6-v2/model.onnx

spring.ai.embedding.transformer.tokenizer.uri

The tokenizer.json for the model

onnx/all-MiniLM-L6-v2/tokenizer.json

spring.ai.embedding.transformer.tokenizer.options

Tokenizer options such as padding and truncation

empty

spring.ai.embedding.transformer.onnx.model-output-name

Output node used to compute the embedding

last_hidden_state

spring.ai.embedding.transformer.onnx.gpu-device-id

GPU device, needs the onnxruntime_gpu dependency

-1 (CPU)

spring.ai.embedding.transformer.cache.enabled and .cache.directory

Caching of remote resources, and where they are stored

true, under the temporary directory

spring.ai.embedding.transformer.metadata-mode

What parts of a document are embedded

NONE
application.yml: a packaged ONNX model with no download and a writable cache
spring:
  ai:
    model:
      embedding: transformers
    embedding:
      transformer:
        onnx:
          model-uri: classpath:/onnx/all-MiniLM-L6-v2/model.onnx
        tokenizer:
          uri: classpath:/onnx/all-MiniLM-L6-v2/tokenizer.json
          options:
            padding: "true"
        cache:
          directory: /var/cache/docsassistant/onnx

Source: Spring AI: Transformers (ONNX) Embeddings.

The same model can be built by hand, which is useful in tests and in non-Boot code.

Building the model manually
import org.springframework.ai.transformers.TransformersEmbeddingModel;

TransformersEmbeddingModel embeddingModel = new TransformersEmbeddingModel();
embeddingModel.setTokenizerResource("classpath:/onnx/all-MiniLM-L6-v2/tokenizer.json");
embeddingModel.setModelResource("classpath:/onnx/all-MiniLM-L6-v2/model.onnx");
embeddingModel.setResourceCacheDirectory("/var/cache/docsassistant/onnx");
embeddingModel.afterPropertiesSet();

float[] vector = embeddingModel.embed("How do I configure a Spring profile?");

Source: Spring AI: Transformers (ONNX) Embeddings. The official page’s example returns lists of doubles from embed(List<String>); the single-string embed returning float[] is the EmbeddingModel interface method.

Three operational points are covered in Local Models with Spring AI, with a worked configuration. Briefly, the embedding dimension follows the model (384 for MiniLM), so switching models means re-embedding the corpus and a new vector column; with no configuration the model and tokenizer are fetched on first use, which an air-gapped environment cannot do; and the cache directory must be writable in a read-only container. The ONNX file itself has to be supplied when it is not the default. Use an onnx-community or original repository with an onnx/ folder, and read the model card for the input and output names. Environment-specific property values belong in profiles, see Configuration and Profiles.

LangChain4j: in-process ONNX embeddings

LangChain4j ships ready-made in-process embedding models, each a Maven artifact that contains the ONNX file and tokenizer, plus a generic OnnxEmbeddingModel for models you export yourself. The official page shows version 1.20.2-beta30 for these artifacts on 2026-09-30, while the LangChain4j BOM is 1.20.2. The -beta suffix is a stability label, not a missing release, so pin it explicitly and do not expect the BOM to manage it. Available artifacts include langchain4j-embeddings-all-minilm-l6-v2, its quantized variant langchain4j-embeddings-all-minilm-l6-v2-q, and langchain4j-embeddings-bge-small-en-v15-q; the complete list is in the embeddings folder of the LangChain4j repository.

pom.xml: the bundled MiniLM model and the generic ONNX embedding model
<dependency>
  <groupId>dev.langchain4j</groupId>
  <artifactId>langchain4j-embeddings-all-minilm-l6-v2</artifactId>
  <version>1.20.2-beta30</version>
</dependency>
<dependency>
  <groupId>dev.langchain4j</groupId>
  <artifactId>langchain4j-embeddings</artifactId>
  <version>1.20.2-beta30</version>
</dependency>

Source: LangChain4j: In-process embedding models.

Using the bundled model and a custom ONNX export
import dev.langchain4j.data.embedding.Embedding;
import dev.langchain4j.model.embedding.EmbeddingModel;
import dev.langchain4j.model.embedding.onnx.OnnxEmbeddingModel;
import dev.langchain4j.model.embedding.onnx.PoolingMode;
import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
import dev.langchain4j.model.output.Response;

EmbeddingModel bundled = new AllMiniLmL6V2EmbeddingModel();
Embedding first = bundled.embed("How do I configure a Spring profile?").content();

EmbeddingModel custom = new OnnxEmbeddingModel(
        "/models/e5-small/model.onnx",
        "/models/e5-small/tokenizer.json",
        PoolingMode.MEAN);
Response<Embedding> second = custom.embed("query: How do I configure a Spring profile?");

Source: LangChain4j: In-process embedding models. The model path and the query: prefix are examples; the prefix convention is model specific, so follow the model card. More on LangChain4j is in LangChain4j and Local Models.

Table 3. In-process embedding options compared
Option Good for Model choice Dependency weight

Spring AI TransformersEmbeddingModel

Spring Boot apps that already use Spring AI VectorStore and EmbeddingModel

Any sentence-transformer ONNX export, set with properties

ONNX Runtime and the DJL tokenizer, native libraries on first use

LangChain4j bundled or OnnxEmbeddingModel

LangChain4j pipelines or plain Java

Bundled models, or any ONNX file with a tokenizer and a pooling mode

ONNX Runtime; the bundled artifacts embed the model in the jar

Hand-written ONNX Runtime plus DJL tokenizer

Non-embedding models, custom pre- and post-processing

Anything ONNX

Smallest dependency set, most code

Jlama and java-llama.cpp

Two projects run LLMs inside the JVM, and both carry caveats that the server path below avoids.

Jlama (https://github.com/tjake/Jlama) is an inference engine written in Java that loads safetensors models from the Hub and targets the JDK Vector API. Its README requires Java 20 or later and says it uses Java 21 preview features, so the JVM needs --enable-preview --add-modules jdk.incubator.vector. The Maven artifacts are com.github.tjake:jlama-core and jlama-native, and the latest at the time of writing is 0.8.4. That release dates from January 2025 and the repository’s last push was October 2025, so the project is slow-moving; model support covers a limited list of architectures, so a current model family such as Gemma 4 or Qwen3.5 may not load. Check the supported models list in the README before designing around it.

JVM flags needed by Jlama on Java 21
export JDK_JAVA_OPTIONS="--add-modules jdk.incubator.vector --enable-preview"
java -jar docsassistant.jar

Source: Jlama README.

java-llama.cpp (https://github.com/kherud/java-llama.cpp) is a JNI binding that embeds llama.cpp and loads GGUF files. It is published as de.kherud:llama; Maven Central lists 4.2.0 while the README snippet and latest GitHub release are 4.1.0, so confirm which one you take. The README badge pins the bundled llama.cpp to build b4916, which dates from March 2025, so the binding lags upstream llama.cpp by well over a year: a recent Hub GGUF for a new architecture may need a newer llama.cpp than the binding contains. Native memory is allocated outside the JVM heap, so LlamaModel is AutoCloseable and must be closed.

Generating with java-llama.cpp
import de.kherud.llama.InferenceParameters;
import de.kherud.llama.LlamaModel;
import de.kherud.llama.LlamaOutput;
import de.kherud.llama.ModelParameters;

ModelParameters modelParams = new ModelParameters().setModel("/models/qwen3-4b-q4_k_m.gguf");
InferenceParameters inferParams = new InferenceParameters("Explain Spring profiles in one sentence.");

try (LlamaModel model = new LlamaModel(modelParams)) {
    for (LlamaOutput output : model.generate(inferParams)) {
        System.out.print(output);
    }
}

Source: java-llama.cpp README. The model file name is an example.

Both are reasonable for experiments and offline demos; for a service that must follow the model ecosystem they put you behind two separate release trains.

The recommended LLM path: a local server behind Spring AI

Run the model in a process built for it, and let Spring AI talk to it. Ollama, vLLM and llama.cpp’s llama-server already follow the Hub’s release cadence, handle quantized GGUF or safetensors weights, batching and GPU offload, and expose an HTTP API. Your Spring Boot app keeps the same ChatClient code it would use against a hosted model. See Running LLMs Locally for choosing and operating the server, Ollama, llama.cpp and vLLM & SGLang, and Spring AI for the framework. The Spring AI side of this, with pull strategy, keep-alive and context options, is in Local Models with Spring AI.

application.yml: Ollama starter, or OpenAI starter against an OpenAI-compatible server
spring:
  ai:
    model:
      chat: ollama            # use "openai" for the second variant below
    ollama:
      base-url: http://localhost:11434
      chat:
        model: qwen3:4b
---
spring:
  config:
    activate:
      on-profile: vllm
  ai:
    model:
      chat: openai
    openai:
      base-url: ${VLLM_BASE_URL:http://localhost:8000/v1}
      api-key: ${VLLM_API_KEY:not-needed}
      chat:
        model: Qwen/Qwen3-4B

Source: Spring AI: Ollama Chat and Spring AI: OpenAI Chat. The starters are spring-ai-starter-model-ollama and spring-ai-starter-model-openai. Models are named in the server’s own naming (an Ollama tag, or the Hub id for vLLM). A server bound to 0.0.0.0 has no authentication of its own, so keep it on localhost or behind a gateway, see LLM Gateways.

For the screenshot task, a vision-capable model on the same server needs no extra Java code: Spring AI’s multimodal messages carry the image, see Multimodality, and the choice of model is in Vision Models. Model calls block for seconds; handle them with the techniques in Concurrency Alternatives to Reactive Programming and expose them through REST APIs.

Whisper and text-to-speech

Java has no maintained Hub-native Whisper or Kokoro client, so there are two workable designs.

A sidecar is the simplest. Run a small service that wraps faster-whisper or transformers for speech-to-text and Kokoro or Piper for text-to-speech output, and call it from Spring Boot. The sidecar can be a Python FastAPI service you write (see the Python backends page of this sub-section, Python Backends, and Speech Models: STT & TTS for the models), or an existing OpenAI-compatible audio server such as Speaches, whose latest release on 2026-09-30 was a release candidate (0.9.0-rc.3), so pin a version and test it. When the sidecar speaks the OpenAI audio API, Spring AI’s transcription model needs only a base URL. Transcription-specific properties override the common spring.ai.openai.* ones.

application.yml: transcription through a local OpenAI-compatible Whisper server
spring:
  ai:
    model:
      audio:
        transcription: openai
    openai:
      audio:
        transcription:
          base-url: http://localhost:8001
          api-key: not-needed
          model: Systran/faster-whisper-small

Source: Spring AI: OpenAI Transcriptions. The model id is an example in the naming the server expects; whether a given server needs a /v1 suffix on the base URL depends on the server, see the table in Local Models with Spring AI. Code that calls the TranscriptionModel is in Audio Transcription & Speech.

Transcribing an uploaded question
import org.springframework.ai.audio.transcription.TranscriptionModel;
import org.springframework.core.io.Resource;

String transcribe(TranscriptionModel transcriptionModel, Resource question) {
    return transcriptionModel.transcribe(question);
}

Source: Spring AI: Transcription API. Speech output has no Spring AI path to a local non-OpenAI server beyond the OpenAI speech model with a base URL, so a plain HTTP client (RestClient) to the sidecar is the honest default for text-to-speech, see REST APIs.

ONNX in process is possible: onnx-community/whisper-small and onnx-community/Kokoro-82M-v1.0-ONNX exist on the Hub, and both run with OrtSession. The Kokoro repository declares Apache-2.0, while the Whisper ONNX repository declares no licence on its own card and points to openai/whisper-small (Apache-2.0) as its base, so read the base card before shipping it. The effort is the catch. Whisper needs log-Mel feature extraction in Java, a decoder loop with KV-cache inputs and token suppression rules; Kokoro needs text phonemization (the ONNX file does not include it) and voice-style vectors. Budget for this as a small project of its own, and validate the output against the Python reference for the same audio. Unless a single deployable jar is a hard requirement, the sidecar is cheaper.

Packaging, operations and the DocsAssistant picture

Models are large files, so decide how they reach the JVM. Small embedding models (tens of megabytes) can live on the classpath or in a volume. Anything larger is downloaded at image build time or mounted, using the download techniques of Hub Essentials, and referenced by file: URLs so the service never reaches the Hub at run time. Embedding inference is CPU-bound and benefits from caching repeated queries, see Caching. Expose model-call latency and failures through the tools of Metrics and Observability.

Table 4. DocsAssistant goes offline, Java edition
Step Wiring Model from the Hub

Transcribe

TranscriptionModel with spring.ai.openai.audio.transcription.base-url

A Whisper variant served by the sidecar (see Speech Models: STT & TTS)

Retrieve

TransformersEmbeddingModel in process

sentence-transformers/all-MiniLM-L6-v2 (or an ONNX export of another embedding model)

Answer

ChatClient over the Ollama or OpenAI starter with a local base-url

A small model such as Qwen3 or Gemma 4 E2B, see Small LLMs & Embeddings

Speak

RestClient to the TTS sidecar

Kokoro-82M or a Piper voice

Describe the screenshot

Multimodal message to a vision-capable local model, or SigLIP 2 ONNX in process for classification only

See Vision Models

If you change an ONNX embedding model, or quantize a model you host yourself, follow Conversion & Quantization and keep the original safetensors checkpoint as the source of truth. Never load pickle-based checkpoints or run trust_remote_code code from a service that handles user data; in Java the ONNX and GGUF paths avoid both.

References