Voice Agents

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This sub-section was written on 2026-10-01 against LangChain 1.4.3 (with langchain-core 1.6.6 and langgraph 1.2.12) and FastAPI 0.142.2 on Python 3.12, Spring AI 2.0.1 on Spring Boot 4.1.x with Java 21, Pipecat 1.12.0, LiveKit Agents 1.8.3, OpenAI GPT-Live (gpt-live-1) and the Realtime API (gpt-realtime-2.1), the Gemini Live API (gemini-3.8-live), the ElevenLabs Python SDK 2.70.0, Twilio ConversationRelay and Media Streams, and Asterisk 23.5, 22.11 and 20.21; the full list is in the version baseline below. Versions and model names were read from PyPI, npm, Maven Central, GitHub releases and the vendors' own documentation on that date, and hosted model names change quickly, so every page tells you which model page to check before you copy an identifier. The code on the pages is illustrative: it is written from the official documentation of each platform and is not compiled by this repository. Real-time audio, Twilio and Asterisk could not be tested here, and each page says in its lead paragraph which steps were documented from official documentation only.

A retrieval-augmented assistant that only answers in a text box is not yet a product for people who are driving, cooking, walking through a warehouse or phoning a help line. The RAG Systems in Production sub-section builds the assistant, and Conversational Channels puts it behind chat surfaces. This sub-section adds the spoken channel: how audio gets to the assistant, how the answer gets back as speech quickly enough to feel like a conversation, and how that works in a browser and on a telephone.

Voice is not chat with a microphone

A text conversation tolerates a two-second pause, long answers, tables and links. A spoken conversation does not. People expect a reply to start within a second or so of finishing a sentence, they interrupt when the answer is wrong or too long, they cannot skim, and they cannot see a citation. Five consequences shape every page that follows:

  • Latency is the user experience. The time from the end of the user’s speech to the first audible word is a sum of network, end-of-turn detection, the model’s first token and the speech synthesiser’s first audio byte; each stage has to stream. The Voice Architectures page builds the budget.

  • Turn-taking is a feature, not a detail. Deciding when the user has finished, and stopping instantly when they interrupt, decides whether the agent feels attentive or rude. See Turn-Taking and Interruptions.

  • Answers are written for the ear. Short sentences, no markdown, numbers and identifiers spelled so that they can be spoken, and citations read as "according to the installation guide" rather than as links.

  • The channel is lossy. Telephone audio is 8 kHz mu-law, browsers add echo cancellation, and recognition errors propagate into retrieval, so the assistant must confirm what it understood when the stakes are high.

  • Recording and consent are legal matters. Calls are regulated differently in each jurisdiction; the production page links the rules rather than summarising them.

The idea: one core, many voices

Every page uses the same running scenario, call DocsAssistant. The assistant answers questions about this site’s documentation and its core is the same channel-agnostic operation used by the other sub-sections:

ask(conversationId, userId, text) -> stream of { token | citation | done }

Source: Serving RAG as an API (the operation and its streaming contract) and Channel-Agnostic Architecture (the same core behind chat channels).

The core is implemented twice, as in the RAG Systems sub-section: in Python with LangChain create_agent, a retriever tool, a Postgres checkpointer and FastAPI, and in Java with the Spring AI ChatClient, the RAG advisor, JDBC chat memory and Spring Boot. The voice layer sits in front of it and has to turn audio into the text argument and the token stream back into audio. The scenario follows four paths:

  1. Browser. A WebRTC voice widget on a documentation page speaks to a Pipecat or LiveKit worker, or directly to a realtime API.

  2. ElevenLabs Agent. A hosted agent calls the DocsAssistant core as its custom LLM or as a tool.

  3. Phone call through Twilio. First with ConversationRelay, where Twilio does the speech work and your WebSocket exchanges text; then with Media Streams, where raw audio goes through Pipecat or a hand-written Spring pipeline.

  4. Company PBX. An Asterisk extension streams audio through External Media or chan_websocket to a Python (Pipecat) or Java (Spring) voice service.

flowchart LR subgraph Callers["Callers"] B[Browser widget] P[Phone through Twilio] X[Company PBX / Asterisk] E[ElevenLabs Agent] end subgraph Voice["Voice layer"] RT[Realtime API] FW[Pipecat / LiveKit worker] HR[Hand-rolled Spring or FastAPI pipeline] CR[ConversationRelay text WebSocket] end B --> RT B --> FW P --> CR P --> FW X --> FW X --> HR E -->|custom LLM or tool| CORE RT -->|tool call or MCP| CORE FW -->|LLM step| CORE HR --> CORE CR --> CORE CORE["DocsAssistant core: ask(conversationId, userId, text)"] --> RAG[(Retriever, memory, ACL filter)]

Choosing an approach

Decision map with four options for giving the DocsAssistant core a voice. A hosted voice platform such as ElevenLabs Agents gives the fastest start and receives the core as a custom LLM or tool. A realtime speech-to-speech API from OpenAI, Google or Azure gives the lowest latency and reaches the core through tool calls. An open-source framework such as Pipecat or LiveKit Agents lets you choose each model and run some locally, with the core as the LLM step. A hand-rolled pipeline gives full control and the least footprint. Delivery then branches into the web over WebRTC or WebSocket and the telephone network over Twilio, SIP or an Asterisk PBX.

The table below turns the map into the questions that usually decide the matter.

Question Hosted voice platform Realtime API Open-source framework Hand-rolled pipeline

Examples

ElevenLabs Agents

OpenAI GPT-Live and Realtime, Gemini Live, Azure Voice Live

Pipecat, LiveKit Agents

Spring Boot or FastAPI with a WebSocket, VAD, STT, your LLM call and TTS

Time to a first working agent

Hours.

Hours to days.

Days.

Weeks.

Control of the speech models

Chosen from the platform’s catalogue.

One speech-to-speech model per vendor.

Any STT, LLM and TTS service, hosted or local.

Anything you can call.

How the RAG core is reached

Custom LLM endpoint, server tool or MCP tool.

Function tools, remote MCP or a delegated backend.

The LLM step of the pipeline, or a tool the LLM calls.

A direct call between STT and TTS.

On-premises or air-gapped

No.

No.

Yes, with local models.

Yes, with local models.

Telephony

Native Twilio integration and SIP trunking.

SIP to the vendor, or a bridge through a partner.

Twilio, Telnyx and other serializers, LiveKit SIP.

Whatever you implement.

What you operate

Almost nothing.

Session-minting endpoint and tools.

Voice workers and their scaling.

Everything, including turn detection and echo handling.

Main cost driver

Per conversation minute (see the platform’s pricing page).

Per audio token or per second of audio.

Model hosting plus per-minute STT and TTS.

Model hosting plus your engineering time.

Page

ElevenLabs Agents

Realtime APIs

Pipecat and LiveKit Agents

Building a Voice Pipeline by Hand

Prices are deliberately not quoted: they change often and differ by plan and region, so each page links the vendor’s own pricing page and the production page gives the method for estimating cost per minute instead of figures.

Reading path

Read the pages in navigation order on a first pass. Shorter paths:

A voice widget on a web page

  1. Voice Architectures and Turn-Taking and Interruptions.

  2. Voice in the Browser.

  3. Either Realtime APIs or Pipecat.

A phone number that answers questions

  1. Voice Architectures.

  2. Telephony with Twilio, starting with ConversationRelay.

  3. ElevenLabs Agents if you prefer a hosted agent with a native Twilio link.

  4. Production Voice Agents.

An agent behind your company PBX

  1. Telephony with Asterisk and SIP.

  2. Pipecat or Building a Voice Pipeline by Hand.

  3. Speech to Text and Text to Speech for local models.

A fully on-premises agent

  1. Speech to Text and Text to Speech, with the speech models catalogue.

  2. Pipecat with faster-whisper, a local LLM and Kokoro.

  3. Production Voice Agents for GPU sizing.

What’s covered

Foundations

  • Voice Architectures — cascaded, speech-to-speech and hybrid designs, where RAG fits in each, streaming every stage, sentence chunking and the latency budget.

  • Turn-Taking and Interruptions — voice activity detection, semantic turn detection, barge-in with history truncation, echo cancellation, DTMF and silence handling.

Speech models

  • Speech to Text — streaming versus batch, hosted and local recognisers, accuracy factors and the word error rate.

  • Text to Speech — streaming synthesis, voices and cloning, SSML, hosted and local voices with their licences, audio formats and resampling.

Hosted platforms

  • Realtime APIs — OpenAI GPT-Live and Realtime, Gemini Live and Azure Voice Live, ephemeral credentials, tools and remote MCP.

  • ElevenLabs Agents — agent configuration, a custom LLM pointing at the DocsAssistant core, tools, workflows, SDKs, Twilio and SIP.

Frameworks

  • Pipecat — pipelines and frames, services, transports, a LangChain agent as the LLM step, a fully local stack, Smart Turn and Flows.

  • LiveKit Agents — rooms, workers, AgentSession, the turn detector, tools and LiveKit SIP.

  • Building a Voice Pipeline by Hand — a minimal cascaded pipeline in Spring Boot and in Python asyncio, and what frameworks save you.

Web delivery

  • Voice in the Browser — getUserMedia, WebRTC to a voice server, the WebSocket PCM alternative, the Web Speech API and mobile caveats.

Telephony

Operations

  • Production Voice Agents — scaling, GPUs, cost per minute, recording and consent, PII, observability, testing, human fallback and accessibility.

Cheat sheet

LLMOps & Evaluation covers evaluating and operating voice agents in production; AI Security & Responsible AI covers prompt injection and governance for them.

Version baseline

Every version below was read on 2026-10-01 from PyPI, npm, Maven Central, Hugging Face or GitHub releases, or from the vendor’s own documentation. Each page repeats the subset it uses in its lead paragraph; if a page and this table ever disagree after an update, this table wins.

Area Versions

LangChain (Python)

langchain 1.4.3, langchain-core 1.6.6, langgraph 1.2.12, on Python 3.12. LangChain’s own voice guidance describes the STT, agent, TTS "sandwich" and the realtime alternative.

Web layer (Python)

FastAPI 0.142.2; aiortc 1.15.0 (the WebRTC implementation under Pipecat’s SmallWebRTC transport).

Spring AI

2.0.1 through org.springframework.ai:spring-ai-bom:2.0.1 on Spring Boot 4.1.x and Java 21; a 2.1.0-M1 milestone exists and is not used. The audio interfaces are TranscriptionModel (extending StreamingTranscriptionModel) and TextToSpeechModel (extending StreamingTextToSpeechModel); the SpeechModel name used by Spring AI 1.0 and by books written against it no longer exists in 2.0.1, as the existing Audio Transcription & Speech page records. The pages in this sub-section use the 2.0 names. ElevenLabs is a Spring AI text-to-speech provider.

OpenAI

Two voice APIs. GPT-Live (model gpt-live-1, session created with POST /v1/live/sessions, full duplex, conversation handled by the live model and reasoning delegated to a backend through Responses or client delegation) and the Realtime API (speech-to-speech models including gpt-realtime, gpt-realtime-1.5, gpt-realtime-2 and gpt-realtime-2.1; ephemeral client secrets for browsers, WebRTC, WebSocket and SIP, function tools and remote MCP). openai Python package 3.22.1. The models page lists gpt-realtime-2.1 as the reasoning model with tool use and gpt-realtime as the first generally available realtime model.

Google

Gemini Live API over WebSockets with function calling, session management and ephemeral tokens; the models page lists gemini-3.8-live (stable) alongside gemini-3.1-flash-live-preview and gemini-2.5-flash-native-audio-preview-12-2025. google-genai Python package 2.26.0.

Azure

Voice Live API (speech recognition, generative model and speech synthesis behind one interface, with noise suppression, echo cancellation, semantic end-of-turn detection and avatars) and the Azure OpenAI realtime audio route; documented under Foundry Tools on Microsoft Learn.

ElevenLabs

ElevenAgents documentation under /docs/eleven-agents/ (the older /docs/agents-platform/ addresses redirect there); custom LLM through an OpenAI-compatible Chat Completions or Responses endpoint; elevenlabs Python package 2.70.0.

Pipecat

pipecat-ai 1.12.0 (BSD-2-Clause). Pipecat Flows is part of core in the pipecat.flows module; the separate pipecat-ai-flows package is at 1.4.0. Smart Turn is the pipecat-ai/smart-turn project.

LiveKit

livekit-agents 1.8.3 and livekit-plugins-turn-detector 1.8.3 (Apache-2.0), LiveKit server 1.13.7, with LiveKit SIP for telephony.

Other frameworks

TEN Framework 0.11.73 and TEN VAD 1.0; Kyutai Unmute and Moshi (the kyutai-labs/moshi repository is active); Vocode is not recommended because it is no longer actively developed.

Speech to text

faster-whisper 1.2.1 (MIT), whisper.cpp 1.9.4 (MIT), Moonshine (moonshine-voice 0.1.5, MIT), NVIDIA Parakeet parakeet-tdt-0.6b-v3 (CC-BY-4.0), Kyutai STT (CC-BY-4.0), Silero VAD 6.2.3 (MIT). Deepgram Flux (flux-general-en and flux-general-multi on the /v2/listen endpoint, with EndOfTurn and EagerEndOfTurn events) and AssemblyAI streaming (with an end_of_turn field) as hosted options.

Text to speech

Kokoro-82M (Apache-2.0); Piper through OHF-Voice/piper1-gpl 1.8.0 (GPL-3.0; the original repository is archived); XTTS-v2 through the idiap/coqui-ai-TTS fork 0.27.5 (code under MPL-2.0) whose model weights carry the Coqui Public Model Licence, which is non-commercial; Sesame CSM-1B (Apache-2.0, speech generation only); Kyutai TTS (CC-BY-4.0).

Twilio

Programmable Voice with <Connect><ConversationRelay> (text WebSocket messages setup, prompt, interrupt, dtmf, text, play, sendDigits, language and end) and <Connect><Stream> Media Streams (8 kHz mu-law base64 JSON with connected, start, media, mark, clear, dtmf and stop); twilio Python helper 9.11.2 and twilio-java 13.0.2.

Asterisk

Current releases 23.5.0, 22.11.0 and 20.21.0 (a 24.0.0 release candidate exists). The chan_websocket channel driver is available from 23.0.0, 22.6.0, 21.11.0 and 20.16.0; AudioSocket and ARI External Media are older.

Other telephony

Jambonz (its llm verb supports OpenAI Realtime and GPT-Live models), the third-party FreeSWITCH module mod_audio_stream 1.1.0 (MIT), Vonage and Telnyx WebSocket audio.

Web platform

Browser APIs as specified by the W3C WebRTC recommendation, MDN for getUserMedia, RTCPeerConnection, AudioWorklet and the Web Speech API.

Where the books are out of date

No requester-provided book covers real-time voice in depth. Only the chapters named in the bibliography are used, and only for concepts. The differences that matter:

  • Spring AI audio. Spring AI in Action (Chapter 8) targets Spring AI 1.0. In 2.0 TranscriptionModel extends StreamingTranscriptionModel, the text-to-speech interfaces are TextToSpeechModel and StreamingTextToSpeechModel (the SpeechModel of the book is gone), OpenAI speech streams real chunks, ElevenLabs is a provider, and the Azure OpenAI module was removed. The pages show the current API.

  • Hugging Face pipelines. Hugging Face in Action covers ASR pipelines and Gradio audio widgets; the speech catalogue and current idioms live on the speech models page.

  • Real-time voice. Turn-taking, barge-in, realtime APIs, WebRTC audio and telephony are not covered by any of the five books; those pages rely on official documentation alone.

  • OpenAI. Anything written about the Realtime API before 2026 describes the earlier preview models and event names; GPT-Live is a newer, separate API. The Realtime APIs page follows the current documentation.

Bibliography

The following sources were consulted while preparing the pages of this sub-section. Every code example was written from the current official documentation of the platform it uses — never copied from a book. Every documentation page cited on the pages is listed here once. All links below were requested on 2026-10-01 and resolved, except the two oreilly.com publisher pages, which answer automated fetches with an HTTP 403 response. Several addresses redirect to a newer location, and this list gives the current one: the ElevenLabs custom LLM page moved from agents-platform to eleven-agents, the OpenAI realtime SIP guide is now the Telephony and SIP guide, the Pipecat Flows page moved under pipecat/flows, the ElevenLabs examples repository is now elevenlabs/examples, and the Gemini ephemeral tokens page now lives under live-api.

Requester-provided books

No requester-provided book covers real-time voice agents in depth. Only the chapters named below are consulted, for concepts. Their text and code are not reproduced; concepts are paraphrased and credited on the page that uses them.

oreilly.com answers automated fetches with an HTTP 403 response, but the publisher pages above are live pages when opened in a browser.

Official documentation

ElevenLabs

OpenAI

Google

Azure

Pipecat

LiveKit

Other open-source frameworks and voice activity detection

Speech to text

Text to speech

Spring AI

LangChain

Web platform

Twilio

Asterisk

Other telephony

WebRTC in Java

Specifications and standards

Papers and engineering articles

The requester-provided books are consulted references, not the primary source of any example on these pages. On any discrepancy between a book and the official documentation of a tool or framework, the official documentation is authoritative. No book PDF is committed or attached anywhere in this repository.

References