Voice Architectures

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against LangChain 1.4.3 on Python 3.12, Spring AI 2.0.1 on Spring Boot 4.1.x and Java 21, Pipecat 1.12.0, LiveKit Agents 1.8.3, OpenAI GPT-Live (gpt-live-1) and the Realtime API (gpt-realtime-2.1), and the Gemini Live API (gemini-3.8-live); the complete list is the version baseline. This page is conceptual: it contains no vendor calls, and the snippets that do appear were not run against a live audio service. The design is the one the vendors themselves describe in the guides linked under References.

Three architectures

Every voice agent turns sound into a decision and a decision into sound. What differs is whether the decision is taken on text or on audio.

flowchart LR subgraph Cascaded["1. Cascaded (STT, LLM, TTS)"] A1[Audio in] --> S1[STT] --> L1["LLM + RAG core"] --> T1[TTS] --> O1[Audio out] end subgraph S2S["2. Speech to speech"] A2[Audio in] --> M2["Speech model (audio tokens in and out)"] --> O2[Audio out] M2 -. "tool call" .-> R2["RAG core"] end subgraph Hybrid["3. Hybrid"] A3[Audio in] --> M3["Speech model (hears and speaks)"] M3 -. "text question" .-> L3["LLM + RAG core (text)"] L3 -. "text answer" .-> M3 M3 --> O3[Audio out] end

Cascaded. A speech-to-text (STT) service transcribes the caller, a text LLM (the DocsAssistant core) produces the answer, and a text-to-speech (TTS) service speaks it. This is the "sandwich" that LangChain’s voice-agent guide describes. Its strengths are control and reuse: the LLM step is exactly the text agent you already operate, with its retriever, memory, access-control filters and tests; each stage can be swapped for another vendor or a local model; and the transcript exists at every step, which helps logging, redaction and evaluation. Its weakness is latency, because three services and two network hops sit between the caller and the answer, and information loss, because the LLM never hears tone, hesitation or background noise.

Speech to speech. One model takes audio in and produces audio out, keeping prosody and handling interruptions inside the model. OpenAI GPT-Live and the Realtime API, Gemini Live and Azure Voice Live are of this kind (see Realtime APIs). It is the lowest-latency and most natural option. The price is that the model is the vendor’s, the reasoning is that model’s rather than your chosen LLM’s, and the retrieval has to be reached through tool calls the model decides to make.

Hybrid. A speech model owns the conversation (listening, turn-taking, speaking) but delegates the question to a text backend, which is the DocsAssistant core. OpenAI’s GPT-Live is designed this way: the live model handles the conversation and delegates reasoning to your backend. A cascaded pipeline with a small fast model for chit-chat and the core for real questions is the same idea built from parts.

Table 1. Choosing between the architectures
Concern Cascaded Speech to speech Hybrid

Typical latency to first audio

Highest, a sum of three services.

Lowest.

Between the two.

Choice of reasoning model

Any model behind the LLM step.

The vendor’s speech model.

The vendor’s for conversation, yours for answers.

RAG integration

The LLM step itself, with the same code as the text channel.

Tool call or remote MCP, chosen by the model.

Delegated backend call.

Transcript for audit and redaction

Native, at every stage.

Provided as a side channel by most APIs.

Native for the delegated part.

On-premises

Possible with local STT, LLM and TTS.

No.

Partly.

Where RAG fits

There are two places to put retrieval, and the choice follows the architecture.

  • Context injection. Before the LLM sees the question, retrieval runs and its results are placed in the prompt. It adds a fixed delay to every turn, including turns that need no documents ("yes", "thanks"). In a cascaded pipeline that is simply the DocsAssistant core being called with the transcript.

  • Tool calls. The model decides whether a search is needed and calls a search_docs tool. Greetings cost nothing and hard questions get one or more searches. This is the only option for speech-to-speech models, and it is how the RAG over MCP server is consumed by realtime APIs. The cost is that a tool round trip adds a pause, so the agent should say something short ("let me check the docs") while the tool runs.

The retrieval tool for a voice agent returns less than the text version: two or three short passages, not ten, because the model has to speak the result and the listener cannot skim it. Citations come back as titles ("the installation guide") for the model to say, not as URLs. The retrieval stack itself is covered in RAG Systems in Production; the conversation memory for a voice thread is the same checkpointer or JDBC memory described in Conversational RAG and Memory.

Stream every stage

A pipeline that waits for the end of each stage before starting the next adds the full duration of each stage to the response time. Streaming overlaps them:

  • STT runs while the caller speaks, emitting interim and final transcripts, so by the time the caller stops, the text is almost ready.

  • The LLM streams tokens, and the first token is what matters.

  • TTS starts as soon as it has a first sentence, and returns audio in chunks.

  • Playback starts at the first audio chunk, and the rest arrives while the first sentence is being spoken.

Timeline of one spoken turn. After the user stops speaking, network transit, end-of-turn detection, the language model’s time to first token including retrieval, the speech synthesiser’s time to first byte, and the network transit back add up to the response time. Below, with streaming, the language model, the sentence chunker, the synthesiser and playback overlap so the first sentence is heard while later sentences are still being generated.

Sentence chunking

A TTS engine needs enough text to choose intonation but not so much that you wait for it. The usual compromise is to buffer LLM tokens until a sentence boundary (a full stop, question mark, exclamation mark or newline followed by a space), speak that sentence, and keep buffering. Two refinements matter in practice: do not split on abbreviations and decimal numbers ("v1.4.3", "e.g."), and flush a first clause early if the first sentence is long, because the first chunk decides the time to first audio.

sentences.py: a minimal sentence chunker for a token stream
import re
from collections.abc import AsyncIterator

_BOUNDARY = re.compile(r"(?<=[.!?])\s+(?=[A-Z0-9])|\n+")


async def sentences(tokens: AsyncIterator[str], first_flush: int = 80) -> AsyncIterator[str]:
    """Yield speakable chunks from a stream of text tokens."""
    buffer, emitted_first = "", False
    async for token in tokens:
        buffer += token
        while (match := _BOUNDARY.search(buffer)):
            chunk, buffer = buffer[: match.start()], buffer[match.end():]
            if chunk.strip():
                emitted_first = True
                yield chunk.strip()
        # Long first sentence: flush at a comma so audio can start earlier.
        if not emitted_first and len(buffer) > first_flush and "," in buffer:
            head, _, buffer = buffer.rpartition(",")
            emitted_first = True
            yield head.strip() + ","
    if buffer.strip():
        yield buffer.strip()

This is illustrative code and is not a library API. Pipecat and LiveKit ship their own aggregators that do the same job; see Pipecat: Introduction and LiveKit Agents documentation.

Because the audience hears the text, the prompt of a voice agent differs from a chat prompt: ask for one or two short sentences, plain text without markdown, numbers and identifiers written so they can be read aloud, and a spoken lead-in before any tool call. The Spring AI version of this idea is the SpokenText.forSpeech helper on the Audio Transcription & Speech page.

The latency budget

The response time the caller perceives, from the end of their sentence to the first audible word, is, for a cascaded pipeline with streaming stages:

where is the audio round trip between the caller and the voice server (including capture and playout buffering), is the end-of-turn delay (the silence the system waits before deciding the caller has finished, plus any final-transcript delay), is the LLM’s time to first token (which includes retrieval and any tool round trip) and is the synthesiser’s time to first audio byte for the first sentence.

Three observations follow from the formula.

  1. The terms add. A fast TTS does not help a slow end-of-turn detector. Optimise the largest term first, and measure before guessing.

  2. is a choice, not a cost. Waiting for 800 ms of silence is cheap to implement and adds 800 ms to every turn; a semantic turn detector can answer sooner when the sentence is clearly finished and wait longer when it is not. See Turn-Taking and Interruptions.

  3. Retrieval sits inside . A search before the first token delays everything. Speak a short filler while the search runs, cache frequent queries (Caching, Latency and Cost), and keep the reranker small.

A speech-to-speech model has a different budget, , where the model term covers hearing, thinking and the first audio together; tool calls add their own round trip on top.

Measuring it

Measure per stage, on real audio, at the voice server, and report percentiles rather than averages because the tail is what callers remember. Record these timestamps for every turn:

  • t_user_end: the moment the turn detector declared the turn finished.

  • t_stt_final: the final transcript arrived.

  • t_llm_first_token: the first LLM token arrived.

  • t_tts_first_byte: the first synthesised audio byte arrived.

  • t_audio_sent: the first audio frame was written to the transport.

The differences give each term, and t_audio_sent - t_user_end is the server-side part of . Add the transport round trip (the WebRTC stats currentRoundTripTime, or a ping on a WebSocket) to estimate the rest. Frameworks expose these as metrics events; production dashboards and alert thresholds belong to Production Voice Agents.

Barge-in and the architecture

Whichever architecture you choose, the caller may interrupt. A cascaded pipeline must cancel the running LLM stream and the TTS request, flush the audio already queued in the transport, and tell the conversation memory how much of the answer was actually heard. A speech-to-speech API does the first two itself and usually asks the client to report where playback stopped. The details are on the next page.

Choosing

Start with the architecture that matches the constraint that hurts most: a cascaded pipeline when you must reuse the text core, run on premises or pick each model; speech to speech when naturalness and latency dominate and the vendor’s model is acceptable; a hybrid when you want the vendor’s conversation layer but your own answers. The decision map on the landing page turns this into products, and Pipecat and LiveKit Agents show the cascaded and speech-to-speech modes in one framework.

References