Speech to Text

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against faster-whisper 1.2.1 (MIT), whisper.cpp 1.9.4 (MIT), Moonshine (moonshine-voice 0.1.5, MIT), NVIDIA Parakeet parakeet-tdt-0.6b-v3 (CC-BY-4.0), Kyutai STT (CC-BY-4.0), Deepgram Flux, AssemblyAI streaming, Spring AI 2.0.1 on Spring Boot 4.1.x and Java 21, and Python 3.12 (see the version baseline). The code is illustrative and was written from the documentation linked after each block; it was not run against live services here. Hosted model names change quickly, so check each provider’s models page before copying an identifier.

Speech to text (STT, also called automatic speech recognition or ASR) is the first stage of a cascaded voice agent, and its errors are the ones retrieval inherits: if "ConversationRelay" is heard as "conversation really", the search finds nothing. This page is about choosing and configuring the recogniser; the catalogue of open models with their cards, sizes and licences is on the Hugging Face speech models page and is not repeated here.

Streaming versus batch

Batch (offline) transcription takes a complete recording and returns the text. It is right for voicemail, recorded calls, ingesting audio into a knowledge base (see Multimodal RAG) and post-call analytics. Whisper-family models are designed this way: they process 30-second windows.
Streaming transcription accepts audio as it is produced and returns interim results that are revised, followed by final results for each segment. It is mandatory for a live agent, because the transcript has to be ready when the caller stops speaking. Streaming recognisers expose events for interim text, final text, speech start and end, and sometimes the end of the turn (see Turn-Taking and Interruptions).

A batch model can be approximated as a streaming one by running it on short overlapping windows every few hundred milliseconds. This works but costs more compute, gives less stable interim text, and lacks end-of-turn signals; frameworks such as Pipecat and LiveKit wrap it behind a VAD for that reason.

For a live agent only the final transcript goes to the LLM. Interim text is useful for displaying captions, for starting retrieval speculatively, and for deciding whether an interruption contains real words.

Hosted providers

Provider What it offers for agents Where to check

Provider	What it offers for agents	Where to check
Deepgram	Streaming recognition with the Nova family and Flux, a conversational model that emits end-of-turn events (`EndOfTurn`, `EagerEndOfTurn`) on the `/v2/listen` endpoint; keyterm prompting for custom vocabulary.	Deepgram: Flux quickstart
AssemblyAI	Universal streaming with turn-level messages that carry an `end_of_turn` field and formatting options.	AssemblyAI: Streaming
OpenAI	Transcription models (`gpt-4o-transcribe`, `gpt-4o-mini-transcribe`, `whisper-1`) over REST, and streaming through the Realtime API’s transcription sessions.	OpenAI: Realtime API
Azure AI Speech	Real-time and batch recognition in many languages, phrase lists and custom models; also the recogniser inside the Voice Live API.	Azure Speech documentation
Google, ElevenLabs, others	Streaming recognition services are offered by Google Cloud Speech-to-Text and ElevenLabs Scribe among others; agent platforms bundle one of them.	The vendors' own documentation

Deepgram

Streaming recognition with the Nova family and Flux, a conversational model that emits end-of-turn events (EndOfTurn, EagerEndOfTurn) on the /v2/listen endpoint; keyterm prompting for custom vocabulary.

Deepgram: Flux quickstart

AssemblyAI

Universal streaming with turn-level messages that carry an end_of_turn field and formatting options.

AssemblyAI: Streaming

OpenAI

Transcription models (gpt-4o-transcribe, gpt-4o-mini-transcribe, whisper-1) over REST, and streaming through the Realtime API’s transcription sessions.

OpenAI: Realtime API

Azure AI Speech

Real-time and batch recognition in many languages, phrase lists and custom models; also the recogniser inside the Voice Live API.

Azure Speech documentation

Google, ElevenLabs, others

Streaming recognition services are offered by Google Cloud Speech-to-Text and ElevenLabs Scribe among others; agent platforms bundle one of them.

The vendors' own documentation

Weigh a hosted STT on five criteria: streaming latency (time from speech to interim and final text), languages and accents you actually need, telephone-quality (8 kHz) accuracy, vocabulary adaptation, and data handling (is audio retained, can it be excluded from training, where is it processed). Prices vary by plan and are not quoted here. The Azure service’s modes (real-time, fast, batch and LLM speech), diarization and custom speech are covered in detail on Foundry Tools: Speech to Text.

Local models

Running STT yourself removes per-minute fees and keeps audio on your infrastructure, at the cost of GPU or CPU capacity (see Production Voice Agents).

Model or runtime Strengths Caveats

Model or runtime	Strengths	Caveats
`faster-whisper` (CTranslate2, MIT)	The most used local option: Whisper accuracy at several times the speed, CPU or GPU, quantised weights.	Offline by design; streaming is simulated with a VAD and windows.
`whisper.cpp` (MIT)	Dependency-free C and C++ runtime that runs on CPUs, Apple silicon and small devices; bindings for many languages.	Same windowed behaviour as Whisper.
Moonshine (MIT)	Small models built for low-latency, on-device streaming of short utterances.	Model cards warn of invented text on very short or clipped audio; validate on your calls.
NVIDIA Parakeet (CC-BY-4.0)	High accuracy and throughput on GPUs through NeMo; multilingual TDT model in the 0.6B class.	Needs the NeMo toolchain; attribution is required.
Kyutai STT (CC-BY-4.0)	A streaming model with built-in semantic voice activity, designed for conversational agents.	Newer; check supported languages and server requirements.

faster-whisper (CTranslate2, MIT)

The most used local option: Whisper accuracy at several times the speed, CPU or GPU, quantised weights.

Offline by design; streaming is simulated with a VAD and windows.

whisper.cpp (MIT)

Dependency-free C and C++ runtime that runs on CPUs, Apple silicon and small devices; bindings for many languages.

Same windowed behaviour as Whisper.

Moonshine (MIT)

Small models built for low-latency, on-device streaming of short utterances.

Model cards warn of invented text on very short or clipped audio; validate on your calls.

NVIDIA Parakeet (CC-BY-4.0)

High accuracy and throughput on GPUs through NeMo; multilingual TDT model in the 0.6B class.

Needs the NeMo toolchain; attribution is required.

Kyutai STT (CC-BY-4.0)

A streaming model with built-in semantic voice activity, designed for conversational agents.

Newer; check supported languages and server requirements.

The Whisper family (openai/whisper) is multilingual and robust, but it can hallucinate text on silence or noise, which a VAD in front of it prevents. Model sizes trade accuracy for speed: use the smallest that meets your word error rate on your audio, and benchmark on a sample of real calls, not on the published benchmarks.

Accuracy factors

Sample rate and codec. Telephone audio is 8 kHz with a compressed codec (mu-law or A-law). Models trained mostly on 16 kHz or wideband audio lose accuracy on it. Do not upsample and hope: prefer a recogniser documented for telephony, or fine-tune on 8 kHz material. Send the audio at its native rate and encoding where the API allows it.
Accents and languages. Accuracy varies strongly by language and accent. Configure the language explicitly when you know it, use a multilingual model with language detection when you do not, and test the accents of your callers.
Noise and overlap. Background speech, music and speakerphone echo raise errors; noise suppression helps modestly and can hurt with some models, so test both ways.
Custom vocabulary. Product names, identifiers and acronyms ("ARI", "AudioSocket", "chan_websocket") are rarely in the training data. Hosted providers accept keyterms, boosted phrases or phrase lists; Whisper-style models accept an initial prompt that contains the terms. For the DocsAssistant, generate the list from the page titles and API names in the index.
Numbers and formats. "Two oh one" versus "201" matters for identifiers; use the provider’s formatting options and normalise before searching.

Word error rate

The standard accuracy measure is the word error rate (WER), computed after aligning the recogniser’s output with a reference transcript:

where is the number of substituted words, the deleted words, the inserted words and the number of words in the reference. It can exceed 100 percent when there are many insertions. Compute it on a held-out sample of your own transcribed calls, with the same text normalisation (case, punctuation, number format) applied to reference and hypothesis, for example with the jiwer package. A better predictor for a RAG agent is the retrieval hit rate on the transcribed question, because a wrong filler word costs nothing while a wrong product name loses the answer; track both. Sampling and monitoring WER in production is covered on Production Voice Agents.

Examples

Spring AI: TranscriptionModel

Spring AI 2.0 transcribes through TranscriptionModel, with StreamingTranscriptionModel as its streaming twin. The SpeechModel name of Spring AI 1.0 and of books written against it applies to text to speech and no longer exists in 2.0; see Audio Transcription & Speech for the full treatment, options and the OpenAI vocabulary prompt. The service below turns a recorded utterance into the text passed to the DocsAssistant core.

TranscriberService.java: transcribing an utterance with Spring AI

import org.springframework.ai.audio.transcription.TranscriptionModel;
import org.springframework.core.io.Resource;
import org.springframework.stereotype.Service;

@Service
class TranscriberService {

    private final TranscriptionModel transcription;

    TranscriberService(TranscriptionModel transcription) {
        this.transcription = transcription;
    }

    String transcribe(Resource utterance) {
        return this.transcription.transcribe(utterance);
    }
}

Source: Spring AI: Transcription API. The audio interface does not stream microphone audio over a socket by itself: for a live agent the transcription step is a stream processor fed by a VAD, which is what the hand-rolled pipeline builds, or a framework service.

Python: faster-whisper on a finished utterance

local_stt.py: local transcription with faster-whisper

from faster_whisper import WhisperModel

model = WhisperModel("small", device="cpu", compute_type="int8")


def transcribe(path: str, vocabulary: list[str]) -> str:
    segments, info = model.transcribe(
        path,
        language="en",
        vad_filter=True,                       # drop silence to avoid hallucinated text
        initial_prompt=", ".join(vocabulary),   # bias towards product and API names
        beam_size=1,
    )
    return " ".join(segment.text.strip() for segment in segments)

Source: faster-whisper repository (usage and transcribe options). Pick the model size and compute_type from the same page; float16 on a GPU is typical for production.

Python: streaming with a hosted provider

For live audio, use the provider’s streaming SDK and feed it the caller’s audio in small chunks (20 to 100 ms). The shape is the same for Deepgram, AssemblyAI and the OpenAI transcription session: open a socket, send binary audio frames, read events. The skeleton below is provider-neutral; the event names and URL are in the provider’s reference.

stream_stt.py: provider-neutral streaming loop

import asyncio


async def run_stt(audio_chunks, stt_socket, on_final):
    async def sender():
        async for chunk in audio_chunks:          # 16 kHz PCM16 or 8 kHz mu-law, as the provider requires
            await stt_socket.send(chunk)

    async def receiver():
        async for event in stt_socket:            # interim, final and end-of-turn events
            if event.is_final and event.text.strip():
                await on_final(event.text)

    await asyncio.gather(sender(), receiver())

Source: AssemblyAI: Streaming and Deepgram: Flux quickstart show the real clients. In a framework the same stage is a service class that you configure, as on the Pipecat and LiveKit Agents pages, and LangChain’s pattern for wiring such a stage around create_agent is on Multimodality & Voice.

Choosing

Use a hosted streaming recogniser first: it is the fastest route to good latency and telephone accuracy, and its turn events remove work. Move to a local model when data residency, volume, or cost per minute says so, and run a bake-off on your own recordings with WER, retrieval hit rate, latency and GPU cost as the columns. Keep the STT behind an interface so that the change is a configuration.