Text to Speech

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against Spring AI 2.0.1 on Spring Boot 4.1.x and Java 21, the elevenlabs Python package 2.70.0, the openai Python package 3.22.1, Kokoro-82M (Apache-2.0), OHF-Voice/piper1-gpl 1.8.0 (GPL-3.0), the idiap/coqui-ai-TTS fork 0.27.5 with XTTS-v2 weights under the Coqui Public Model Licence, Sesame CSM-1B (Apache-2.0) and Kyutai TTS (CC-BY-4.0); the full list is the version baseline. The code was written from the documentation linked after each block and was not run against live providers. Licence statements were read from each project’s licence file or model card on that date and are not legal advice; read the licence yourself before shipping. Hosted voice and model names change quickly, so check the provider’s voices page.

Text to speech (TTS) is the last stage of a cascaded voice agent and the one the caller judges most directly. The model list of open voices with sizes and cards is on the Hugging Face speech models page; this page covers what a voice agent needs from a synthesiser.

Streaming synthesis

A non-streaming TTS call returns a whole clip, so the caller waits for the full sentence to be synthesised. A streaming call returns audio chunks while synthesis continues, which gives a short time to first byte (the term in the latency budget). There are two kinds of streaming:

  • Streaming output. You send the complete text of a sentence and receive audio in chunks. Every hosted provider and most local engines support this, and it combines with the sentence chunking from the architecture page.

  • Streaming input. You send text in pieces as the LLM produces tokens and the provider starts speaking before the sentence is complete. ElevenLabs offers this over a WebSocket, and some local models are designed for it. It lowers latency further but gives the engine less context for intonation, so measure the quality.

Whichever you use, keep a cancel path: when the caller interrupts, the request must be cancelled and the queued audio dropped (see Turn-Taking and Interruptions).

Voices, cloning and consent

A hosted provider offers a catalogue of stock voices, voice design from a text description, and cloning from recordings. Cloning needs ethical and legal care that the code cannot enforce for you:

  • Clone only voices of people who have given informed, recorded consent for that use, and keep the consent record.

  • Do not make a voice that imitates a real person without permission, and do not use an agent voice to impersonate staff or deceive callers about talking to a machine; many jurisdictions require the caller to be told it is an AI voice (see Production Voice Agents).

  • Open models that clone from a few seconds of audio carry a misuse risk you take on as the deployer; the model cards often forbid impersonation, and some projects deliberately limit cloning to pre-computed voice embeddings.

  • Store voice identifiers, not raw reference audio, in application configuration, and restrict who can create voices.

For the DocsAssistant, a single stock voice chosen for clarity on telephone lines is the right default.

SSML and expressiveness

Speech Synthesis Markup Language (SSML, a W3C recommendation) controls pauses, emphasis, pronunciation and speaking rate in synthesisers that support it, notably Azure AI Speech and Google Cloud Text-to-Speech. ElevenLabs and OpenAI take instructions in other forms: pronunciation dictionaries and voice settings for ElevenLabs, and a free-text instruction prompt for OpenAI’s gpt-4o-mini-tts model. Local models vary: Kokoro and Piper take plain text and phonemes, Piper-style engines accept pronunciation overrides, and many ignore SSML altogether.

For a voice agent the practical needs are small: spell out identifiers ("W-three-C"), read version numbers digit by digit, pronounce product names correctly, and insert short pauses between list items. Do this in a text preprocessing step that you control (a function that converts the LLM’s sentence into speakable text), because it works the same with every engine, and use SSML or the provider’s dictionary only for the cases the preprocessing cannot reach.

Hosted providers

Provider Characteristics Where to check

ElevenLabs

Expressive voices, voice design and cloning, low-latency streaming over REST and WebSocket, PCM and mu-law outputs for telephony. A Spring AI provider.

ElevenLabs: Text to speech

OpenAI

gpt-4o-mini-tts with steerable delivery instructions, stock voices, streaming responses, several formats including PCM.

OpenAI: Text to speech

Azure AI Speech

Neural voices in many languages, SSML, visemes, and custom neural voice under a gated process.

Azure Speech documentation

Google, Deepgram, Cartesia and others

Streaming synthesisers aimed at agents, some with word timestamps useful for interruption handling.

The vendors' documentation

Choose by time to first byte on your network path, voice quality on telephone-rate audio, the languages you need, streaming input support, timestamps (to know how much was actually spoken), pronunciation control and data handling. Azure’s voices, SSML extensions, batch synthesis and avatars are covered on Foundry Tools: Text to Speech and Avatars.

Local models and their licences

Model Licence Notes for agents

Kokoro-82M

Apache-2.0

A small model (82 million parameters) with good quality and fast CPU or GPU synthesis; 24 kHz output; several stock voices; yields audio per text segment. A strong default for a fully local stack.

Piper (OHF-Voice/piper1-gpl)

GPL-3.0

Very fast on CPUs and small devices, many voices and languages. The maintained repository is the GPL-licensed piper1-gpl; the original MIT repository is archived. Because the code is GPL-3.0, linking it into a distributed product has copyleft consequences; running it as a separate service you do not distribute is a different situation. Ask a lawyer about your case. Individual voices carry their own licences.

XTTS-v2 (idiap coqui-ai-TTS fork)

Code MPL-2.0; model weights under the Coqui Public Model Licence (non-commercial)

Multilingual voice cloning from a short reference clip. The weights' licence forbids commercial use, so it fits research and internal experiments, not a product.

Sesame CSM-1B

Apache-2.0

A conversational speech generation model that conditions on prior turns for more natural prosody. It generates speech only (it is not an end-to-end agent), is heavier than Kokoro and needs a GPU for real-time use.

Kyutai TTS

CC-BY-4.0

Streaming-text input, designed for low latency in conversational settings; requires attribution and a GPU server.

Licence terms and voice licences change, and model repositories can be relicensed or gated. Re-read the licence file at the commit you deploy, record it in your software bill of materials, and check each voice file’s own terms, since a model under a permissive licence can still ship voices trained on data with restrictions.

Audio formats and resampling

The synthesiser, the transport and the listener each have a preferred audio format, and a wrong guess shows up as noise, chipmunk speech or silence.

Format Rate Used by Notes

mu-law (G.711 mu-law), 8-bit

8 kHz

Twilio Media Streams, North American PSTN

64 kbit/s; one byte per sample; the format Twilio expects in media messages, base64 encoded.

A-law (G.711 A-law), 8-bit

8 kHz

European PSTN and many SIP trunks

Same size as mu-law with a different companding curve.

PCM16 (signed 16-bit little-endian, linear)

16 kHz

Most STT services, WebRTC wideband paths, ElevenLabs pcm_16000

256 kbit/s mono.

PCM16

24 kHz

Kokoro, OpenAI realtime PCM, many TTS outputs

384 kbit/s mono.

Opus

48 kHz in WebRTC

Browsers and WebRTC

Compressed; the transport encodes and decodes, your code usually sees PCM.

A typical telephone path therefore resamples the synthesiser’s 24 kHz PCM16 down to 8 kHz and companding it to mu-law, and the reverse for the caller’s audio into the STT. Resample with a proper low-pass filter (a polyphase resampler) rather than dropping samples, which aliases and makes speech harsh. For a conversion by a ratio of the polyphase resampler needs the output length ; 24 kHz to 8 kHz is .

audio_convert.py: streaming 24 kHz PCM16 to 8 kHz mu-law
import audioop  # removed in Python 3.13: use the audioop-lts package there
import numpy as np
import soxr


class Pcm24kToUlaw8k:
    """Converts a stream of TTS chunks; one instance per utterance or call, fed every chunk in order."""

    def __init__(self) -> None:
        self._resampler = soxr.ResampleStream(24000, 8000, 1, dtype="int16")   # keeps filter state across chunks
        self._pending = b""                                                     # odd trailing byte of the last chunk

    def convert(self, chunk: bytes, last: bool = False) -> bytes:
        data = self._pending + chunk
        usable = len(data) - (len(data) % 2)              # PCM16 needs whole 2-byte samples
        self._pending = data[usable:]
        samples = np.frombuffer(data[:usable], dtype=np.int16)
        pcm8k = self._resampler.resample_chunk(samples, last=last)   # 24 kHz -> 8 kHz, no seams between chunks
        return audioop.lin2ulaw(pcm8k.tobytes(), 2)       # 16-bit linear -> mu-law

Source: python-soxr: ResampleStream and the Python audioop module. Synthesisers stream audio in chunks of arbitrary size, so the converter keeps two pieces of state. The resampler’s filter history removes the click that per-chunk resampling would add at every chunk boundary. The odd trailing byte of a chunk is held back so that a sample split across two chunks is not dropped. A one-shot scipy.signal.resample_poly call is fine for a whole file, but not for a stream. Pass last=True with the final chunk to flush the filter. Frameworks do this for you; see Pipecat and Telephony with Twilio.

Examples

Spring AI: TextToSpeechModel and streaming

Spring AI 2.0 names the speech interfaces TextToSpeechModel and StreamingTextToSpeechModel; the SpeechModel of Spring AI 1.0 and of the books written against it is gone, as recorded on the Audio Transcription & Speech page, which also covers the starters, options and the OpenAI and ElevenLabs configuration in depth. The class below returns the audio of an answer as a stream of chunks that a WebSocket handler can forward as they arrive.

SpokenAnswerService.java: streaming synthesis with Spring AI
import org.springframework.ai.audio.tts.TextToSpeechModel;
import org.springframework.stereotype.Service;
import reactor.core.publisher.Flux;

@Service
class SpokenAnswerService {

    private final TextToSpeechModel speech;

    SpokenAnswerService(TextToSpeechModel speech) {
        this.speech = speech;
    }

    Flux<byte[]> speak(String sentence) {
        return this.speech.stream(sentence);   // audio chunks as they are produced
    }
}

Source: Spring AI: Text-To-Speech (TTS) API. Which provider backs the bean is chosen with spring.ai.model.audio.speech (openai or elevenlabs); with ElevenLabs the voice is selected by identifier, as described in Spring AI: ElevenLabs Text-To-Speech. Check the output format of the provider you configure against what your transport expects (the table above), and cancel the subscription when the caller interrupts.

Python: ElevenLabs streaming

elevenlabs_tts.py: streaming PCM from ElevenLabs
from elevenlabs.client import ElevenLabs

client = ElevenLabs()  # reads ELEVENLABS_API_KEY from the environment


def speak(sentence: str, voice_id: str):
    return client.text_to_speech.stream(
        voice_id=voice_id,
        text=sentence,
        model_id="eleven_flash_v2_5",     # check the models page for the current low-latency model
        output_format="pcm_16000",         # raw 16 kHz PCM16, ready for a pipeline or resampling
    )  # iterate the result to receive audio chunks

Source: ElevenLabs: Text to speech (the model names and output formats are listed there).

Python: OpenAI streaming speech

openai_tts.py: streaming PCM from OpenAI
from openai import OpenAI

client = OpenAI()


def speak(sentence: str):
    with client.audio.speech.with_streaming_response.create(
        model="gpt-4o-mini-tts",
        voice="coral",
        input=sentence,
        instructions="Speak clearly and calmly, like a helpful support engineer.",
        response_format="pcm",             # raw 24 kHz PCM16 mono
    ) as response:
        for chunk in response.iter_bytes(chunk_size=4096):
            yield chunk

Source: OpenAI: Text to speech (the text-to-speech guide lists voices, formats and the instructions parameter). The same LangChain wiring around an agent is shown on Multimodality & Voice.

Python: Kokoro, fully local

kokoro_tts.py: local synthesis with Kokoro
import numpy as np
from kokoro import KPipeline

pipeline = KPipeline(lang_code="a")   # American English; Apache-2.0 model and code


def speak(text: str):
    for _graphemes, _phonemes, audio in pipeline(text, voice="af_heart"):
        samples = audio.numpy()                        # KPipeline yields a torch.FloatTensor
        yield (np.clip(samples, -1, 1) * 32767).astype(np.int16).tobytes()   # 24 kHz PCM16

Source: Kokoro-82M model card (installation, language codes and voices). The pipeline yields the audio as a PyTorch tensor at 24 kHz, so it is converted to a NumPy array before the PCM16 conversion.

Choosing

Start with a hosted voice and the sentence-chunking pattern, measure time to first byte from your own servers, and make the engine replaceable behind one function. Move to Kokoro when you need on-premises audio under a permissive licence, treat Piper as GPL software, and keep XTTS out of commercial products. Always run the speakable-text preprocessing before the engine.

References