Speech Models: STT & TTS

The two tables below use the same columns, so models can be compared at a glance:

The decision tree below summarises how to pick a starting point; the tables give the detail.

Reference: Hugging Face — Model Cards explains the metadata fields (licence, languages, library) that this page reads from each card.

Speech-to-text (STT, also called automatic speech recognition, ASR) maps an audio waveform to text, optionally with timestamps. The Whisper family is the default starting point because it is multilingual, well supported by every runtime, and available in many converted formats; the alternatives trade language coverage for speed, size or streaming.

Notes on the STT table:

References: transformers — Whisper, transformers — Moonshine and NVIDIA NeMo — ASR models.

Text-to-speech (TTS) maps text to a waveform. The Hub’s text-to-speech task tag is shared by very different designs: a tiny non-autoregressive model such as Kokoro, autoregressive language-model-style models such as Orpheus and Dia, and diffusion-head models such as VibeVoice. The designs differ in cost by more than an order of magnitude, so the size column is worth reading before you pick one.

Notes on the TTS table:

References: transformers — Dia, transformers — CSM, Kokoro repository and Piper repository.

The automatic-speech-recognition pipeline loads the model, its tokenizer and its feature extractor from a model ID and returns a dictionary with the transcript. In transformers v5 the precision argument is dtype, and dtype="auto" uses the dtype stored in the checkpoint. The pipeline accepts a file path, a NumPy array or a list of files.

Whisper predicts the spoken language by default. If you know it, pass it through generate_kwargs (for example generate_kwargs={"language": "english"}); pass {"task": "translate"} to translate to English instead of transcribing. Loading a checkpoint that ships custom code would need trust_remote_code=True, which runs the repository author’s Python on your machine; none of the models above need it, so leave it off.

Reference: transformers — Pipelines and the Whisper large-v3-turbo model card, which documents the chunked long-form options. The full pipeline tour is in Transformers Pipelines.

Text-to-speech in transformers goes through the text-to-audio task (text-to-speech is an alias) or directly through the model and processor classes. Dia is a good example because its card documents the model-and-processor path: tag each speaker turn with [S1] and [S2], generate, then decode and save.

Dia was not fine-tuned on a fixed voice, so each run produces different voices; pass a reference clip with its transcript to condition on a voice, which is the cloning case described above.

Reference: transformers — Dia and the Dia model card.

faster-whisper runs Whisper on CTranslate2 and is usually the quickest way to get Whisper onto a CPU, a consumer GPU or a server. WhisperModel accepts a size name such as large-v3 or turbo and downloads the converted checkpoint from the Hub, or a path to a local converted directory. compute_type selects the precision: float16 on GPU, or int8 on CPU. The transcribe method returns a lazy generator of segments, so transcription only runs as you iterate.

For many files, BatchedInferencePipeline wraps the same model and accepts a batch_size. The distil-whisper checkpoints also run here: the faster-whisper README shows distil-large-v3 with language="en" and condition_on_previous_text=False.

Reference: faster-whisper README and CTranslate2 documentation.

Kokoro is the most convenient small TTS model for local work. The kokoro package exposes KPipeline; the lang_code must match the voice (a is American English, b British English, e Spanish, f French, h Hindi, i Italian, j Japanese, p Brazilian Portuguese, z Mandarin Chinese). It phonemises text with misaki, falling back to espeak-ng, so install that system package first; Japanese and Chinese need the misaki[ja] and misaki[zh] extras. Output is 24 kHz mono audio, yielded one segment at a time.

Because the generator yields per segment, you can start playback after the first segment instead of waiting for the whole text, which gives a reasonable time to first audio for short replies. On Apple silicon, the Kokoro README suggests setting PYTORCH_ENABLE_MPS_FALLBACK=1 to use the GPU. The kokoro package constrains the Python version (the baseline for this page lists Python below 3.13), so use a dedicated virtual environment.

Reference: Kokoro repository and the Kokoro-82M model card. The same model in ONNX form, for runtimes outside Python, is onnx-community/Kokoro-82M-v1.0-ONNX.

Everything above transcribes or synthesises whole utterances, or at best a segment at a time. A conversation needs more: voice-activity detection to know when the user has stopped, streaming in both directions, barge-in (the user interrupting the reply) and a language model in the middle. Those pipelines are covered in the Voice Agents sub-section; treat this page as the model shelf it draws from. As a rule of thumb when choosing for that use, prefer the models marked as streaming (Moonshine streaming for input, Kyutai TTS or VibeVoice-Realtime for output) or the small models whose first segment arrives quickly (Kokoro, Piper), and measure time to first audio on your own hardware rather than trusting a card.

The same models appear later in this sub-section: Conversion & Quantization shows how the Whisper and Kokoro checkpoints are converted for ONNX, Core ML and CTranslate2 runtimes, and the per-platform integration pages reuse the same four-task scenario (transcribe, answer, speak, describe a screenshot) with these models.