Production Voice Agents

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against Pipecat 1.12.0, LiveKit Agents 1.8.3 with LiveKit server 1.13.7, faster-whisper 1.2.1, Kokoro-82M, Silero VAD 6.2.3, Twilio ConversationRelay and Media Streams, Asterisk 23.5, Spring AI 2.0.1 on Spring Boot 4.1.x, LangChain 1.4.3 and FastAPI 0.142.2; the full list is the version baseline on the landing page. This page is method rather than recipe: it gives the formulas and checklists to size, price, measure, test and operate the agent built on the previous pages, and it quotes no prices, no hardware benchmarks and no legal conclusions, because all three change and depend on your region and contract. Nothing here could be run against real telephone traffic from this repository, so the figures in the worked examples are placeholders to show the arithmetic, not measurements.

Sizing concurrent calls

A voice call holds resources for its whole duration: a WebSocket or WebRTC connection, a pipeline object with its buffers, and, with local models, a share of a CPU or GPU. Capacity is therefore a question of how many calls are open at once, not of requests per second. Little’s law gives the average:

where is the call arrival rate (calls per second) and is the mean call duration in seconds. One call per ten seconds ( ) with a mean of 180 s gives calls open. Real traffic is bursty, so plan for the busy-hour peak rather than the daily mean, and add headroom for a failed node:

where is the peak concurrent calls, is the measured number of concurrent calls one instance sustains while keeping inside its budget (see Voice Architectures), and is the headroom fraction. The one number that matters is : find it with a load test that adds simulated callers until the 95th-percentile time to first audio degrades, then set the admission limit a little below that point.

Practical consequences:

  • Admit or reject, never queue silently. A caller who is placed in an unbounded queue hears silence. When a pod is at its limit, refuse the call so the load balancer or the telephony provider can try another instance or play a busy message.

  • Scale on concurrent sessions, not CPU alone. A custom metric such as active_calls drives the autoscaler; CPU lags behind it. Drain a pod on scale-in: stop accepting calls and let open ones finish before the process exits.

  • Cap what one call can consume. Maximum duration, maximum silence and a ceiling on tool calls per call protect both the bill and the fleet (the telephony page lists the abuse controls for SIP).

  • Hosted platforms have their own concurrency limits. ElevenLabs, OpenAI, Gemini and Twilio each cap simultaneous sessions per account or plan; check the limit page of the platform and ask for an increase before a launch, not during one.

  • The core is part of the fleet. Every call also loads the DocsAssistant core and its vector store; size them for the same peak, and use the caches from Caching, Latency and Cost so repeated questions do not each pay for retrieval.

  • Frameworks give you the process model. Pipecat runs a long-lived asyncio bot per call or per few calls and LiveKit runs a job per room on an agent server; both are described under Deployment and Self-hosted or LiveKit Cloud on the Pipecat and LiveKit Agents pages.

GPU needs for local speech models

A fully local stack (see the on-premises stack on the Pipecat page) replaces per-minute fees with hardware you must size. Speech models differ from LLMs in that they run per audio stream in real time, so what matters is the real-time factor (RTF): seconds of compute per second of audio.

A model with on your hardware can serve about ten audio streams at the limit of one device, before you reserve headroom for bursts and for the other models sharing it. The same arithmetic applies to the speech-to-text and the text-to-speech sides separately, and a call uses both directions only while that side is active: recognition runs while the caller speaks, synthesis while the agent speaks, so the sum is lower than the naive count.

Component What drives its cost How to size it

STT (faster-whisper, Parakeet, Kyutai STT)

Model size, precision (quantised weights), and whether you decode windows repeatedly to simulate streaming.

Measure RTF with the production audio (8 kHz telephone audio decoded the same way as in production), then divide capacity by the number of callers speaking at once, not by the number of calls.

TTS (Kokoro, Piper, Sesame CSM, Kyutai TTS)

Output seconds generated per answer, and the time to the first audio chunk of the first sentence.

Measure time to first byte under load, because the latency budget depends on it more than on throughput.

Turn and VAD models

Silero VAD and the Smart Turn or LiveKit turn detector are small and run per stream.

Usually CPU; verify that they do not share a thread with the event loop that moves audio.

A local LLM

Parameters, context length and concurrent decoding sequences.

Use the serving engine’s own capacity guidance; this is the largest consumer by far and is outside this sub-section.

The general rules are stable even though specific cards and speeds change: keep models resident in memory and load them once per process; batch only where the engine supports it without delaying the first token; pin GPU memory per replica so that one noisy call cannot evict a model; and measure with real calls, because published benchmarks use clean wideband audio. The licences of the open models (Apache-2.0, MIT, CC-BY-4.0, GPL-3.0 for Piper’s maintained repository, non-commercial weights for XTTS-v2) are tabulated on the Text to Speech and Speech to Text pages and belong in the software bill of materials before production.

Cost per minute

Voice agents are billed in many units: minutes of conversation, seconds of audio, characters synthesised, tokens, telephone minutes and GPU hours. The method is to convert every component to cost per conversation minute and add them up, using the unit prices from each vendor’s own pricing page at the time you do the exercise.

where is the telephony or transport cost per minute (Twilio, the SIP trunk, or the WebRTC platform), and are the recognition and synthesis prices per minute of audio, and are the fractions of the call during which the user and the agent speak, is the language-model cost per call minute (input and output tokens times their prices, including the retrieved context that is resent on every turn) and is the amortised servers, GPUs and operations. For a hosted voice platform or a speech-to-speech API the formula collapses to one or two unit prices per minute or per audio token, and the exercise is to find out what the price includes.

Three effects are easy to miss:

  • Silence is billed. Some services bill the whole connected time, so the fractions and do not reduce the bill; cap idle time and total duration.

  • Context grows each turn. The input tokens of turn include the earlier turns and the retrieved passages, so the LLM cost is not linear in call length. Truncate history and keep the retrieved context small.

  • Failed and abandoned calls cost money. Ring time, voicemail, transfers and retries are all billable on some platforms.

Multiply by the expected minutes per month, then compare it with the cost of the alternative (a human agent minute, or a chat session). The question the result answers is usually which calls should not go to the voice agent.

Recording, consent and disclosure

Recording a call, transcribing it, using the audio to improve a model and telling the caller that they are talking to a machine are four separate obligations, and the rules differ by country, by US state, by whether one or both parties must consent, by whether the call is crossing a border, and by sector. This site does not summarise them. Start from the sources and ask a lawyer:

  • the General Data Protection Regulation (EU 2016/679), on EUR-Lex, for lawful basis, retention, access and erasure of recordings and transcripts, which are personal data and sometimes biometric data;

  • Article 50 of the EU AI Act, Transparency obligations, on telling people that they are interacting with an AI system;

  • Twilio’s recording documentation, which states what the platform leaves to you, as an example of a provider’s shared-responsibility note; every provider has an equivalent.

Engineering consequences that hold regardless of the law:

  • Disclose at the start of the call that the caller is speaking to an AI and, if recording, that the call is recorded. Greeting prompts already do this in the examples on the previous pages.

  • Make recording a per-call decision stored with the call, not a global switch; keep a record of the consent text played.

  • Do not record when you do not need to. Streaming audio to recognition without storing it is a smaller data protection surface than a recording archive.

  • Set a retention period for audio and a possibly longer one for redacted transcripts, and delete on schedule.

  • Voice cloning of a real person needs that person’s explicit permission (see Text to Speech).

PII redaction in transcripts

Callers read out names, phone numbers, account numbers and addresses, and the recogniser writes them down. Redact before the transcript reaches logs, analytics, evaluation datasets and tracing backends.

  1. Decide the order. Redact in the voice server on the final transcript, before it leaves the process. The live copy that the core needs to answer the question is a separate, short-lived value.

  2. Use a detector. Microsoft’s Presidio combines named-entity recognition, patterns and checksums; some STT providers offer built-in redaction of entities, which is worth enabling as a first layer. Test against your own transcripts, because spoken numbers ("four one five, five five five") differ from written ones.

  3. Replace with typed placeholders (<PHONE>, <ACCOUNT>) so the transcript stays useful for analysis.

  4. Redact audio too if you keep recordings: either do not retain them, or retain them encrypted with a short retention.

  5. Keep the access-control filter. The identity that retrieval uses for document-level access comes from the call setup, not from the transcript; see Access Control and Privacy.

  6. Watch the trace attributes. Span attributes, error messages and tool arguments leak the same data as the transcript.

Spoken text is also an injection channel: a caller, or a recording played into the line, can say "ignore your instructions". The controls in Guardrails and Prompt Injection apply unchanged to the transcript, and Prompt Injection in the AI Security & Responsible AI sub-section covers the underlying attack.

Observability

Instrument per turn, per stage, and report percentiles. The timestamps are defined on Voice Architectures; emit them as spans or histograms with the call id as a correlation attribute. Pipecat exposes metrics through PipelineParams(enable_metrics=True) (see Pipecat: Metrics) and LiveKit Agents emits metrics events (see LiveKit: Metrics); the OpenTelemetry GenAI semantic conventions give common attribute names for the model spans. The RAG half of the trace is described on Observability & Tracing.

Signal Definition Why it matters

Time to first audio

t_audio_sent - t_user_end, p50, p95 and p99

The caller’s perceived latency; alert on p95 above the budget.

Per-stage latency

End-of-turn delay, STT final, LLM first token, TTS first byte

Tells you which term of the budget to fix.

Interruption rate

Barge-ins per agent turn, and the share that occur in the first second of an answer

High values mean the answer is too long, too slow, or the VAD fires on noise or echo.

False cut-off rate

Turns where the caller resumed within a short window after an end-of-turn

Direct measure of an over-aggressive endpointing delay.

Silence and dead air

Seconds without agent or caller audio during a call

Reveals stalled tools and lost audio.

Call outcome

Resolved, abandoned, transferred, error

The product metric, split by channel.

Concurrency and rejections

active_calls and refused admissions

The input to capacity planning.

Transport quality

WebRTC round-trip time, jitter and packet loss; SIP MOS where the trunk reports it

Separates network problems from model problems.

Sampling the word error rate

The word error rate defined on Speech to Text needs a human reference, so compute it on a sample, not on every call:

  1. Sample a fixed fraction of calls, stratified by channel (8 kHz telephone versus wideband browser), language and accent.

  2. Have people transcribe the sample, from redacted audio or inside the retention period.

  3. Compute with identical text normalisation on both sides, and track it per channel over time, together with the retrieval hit rate on the transcribed question.

  4. Re-run the sample after every change of recogniser, model version, codec or vocabulary list.

A rising WER on one channel after a provider change is easier to catch this way than from a satisfaction score. Scoring the answers themselves (groundedness, correctness) belongs to the LLMOps: RAG & Agent Evaluation page.

Testing with simulated callers

Unit tests cover the text core (see the evaluation material in RAG Systems in Production); a voice agent additionally needs tests that run audio through the whole pipeline.

  • Recorded calls. Replay recorded and consented utterances, or synthesised ones, through the transport at real-time speed and assert on the transcript, the tool calls and the time to first audio. Include 8 kHz mu-law, accents, noise, hesitation and numbers read aloud.

  • Scripted simulated callers. A caller script (or an LLM playing a persona) speaks through a TTS into the same transport and listens to the answer through an STT. A test client can join the room or the WebRTC session like any user; LiveKit documents a testing approach for agent behaviour at LiveKit: Testing and evaluation.

  • Barge-in cases. Interrupt in the first word, mid-sentence and at the end of the answer; assert that output stops within the target time and that the memory records what was heard.

  • Telephony paths. Place a real call for each route (browser, Twilio, PBX extension) as a smoke test after every deployment, since provider behaviour cannot be fully simulated. Use a number that is excluded from billing analytics.

  • Load tests. Ramp simulated callers to find and to verify that admission control rejects excess calls cleanly.

  • Chaos tests. Kill a provider connection mid-call, drop packets, and restart a pod while calls are open.

Failure handling and fallback to a human

Every provider in the chain can fail: the recogniser, the LLM, the synthesiser, the retrieval store and the telephony leg. Decide the behaviour before it happens.

flowchart TD A[Incoming call] --> B{Capacity available?} B -- no --> R[Reject: busy message or retry elsewhere] B -- yes --> C[Greeting and AI disclosure] C --> D[Conversation turns] D --> E{Provider error or low confidence?} E -- no --> D E -- retry --> F[Retry or switch to backup provider] F --> D E -- repeated --> G{Human available?} G -- yes --> H[Transfer with context] G -- no --> I[Apologise, offer callback or message] D --> J{Caller asks for a person?} J -- yes --> G D --> K[Call ends: log outcome]

  • Backup providers. Configure a second STT, LLM or TTS provider and switch on repeated errors or timeouts; frameworks make the service a replaceable pipeline stage (see the service lists on the Pipecat and LiveKit pages).

  • Degrade, do not hang. If retrieval times out, answer from what is known and say so, or admit that the lookup failed. Use a spoken filler only for waits you expect, and bound them.

  • Transfer to a human. Offer it whenever the caller asks, after repeated misunderstandings, for sensitive requests the agent is not allowed to handle, and when the agent detects distress. Pass the conversation summary and the caller’s identity with the transfer. The Twilio mechanism is on the "Transfer to a human" section of Telephony with Twilio; LiveKit SIP and Asterisk have their own transfer operations on their pages.

  • Voicemail and machine detection. For outbound calls, start the agent only when a person answers; answering machines and voicemail greetings must not be treated as a caller. Twilio exposes answering machine detection on outbound calls (see Twilio: Call resource); others offer similar signals.

  • After hours. If no human is available, offer a callback or a message instead of a dead end, and record the request.

Accessibility

Voice is an accessibility feature and, badly done, a barrier. People who cannot use a keyboard or screen benefit from a voice interface; people who are deaf, hard of hearing, or have a speech difference need an alternative.

  • Always offer a non-voice path with equivalent capability: the chat channel on the same site (see Conversational Channels), a text relay, or a callback.

  • Let callers set the pace: a slower speech rate, a "repeat that" command, and generous endpointing for slow or atypical speech, which a semantic turn model may handle poorly; test those voices.

  • Provide captions of the agent’s speech in the browser widget and keep the transcript available to screen-reader users (the live-region and focus rules are on Voice in the Browser).

  • Do not make DTMF the only way to confirm something, and do not use a voice-only check as the only proof of identity.

  • Follow the standards summarised on Web Accessibility for the page that hosts the widget.

Limits and abuse

Beyond the controls on the telephony page: per-caller and per-account rate limits on call creation, a maximum call length and idle time, an allow-list or authentication for the endpoints that mint ephemeral credentials (see Voice in the Browser), spending caps and alerts on every hosted account, and toll-fraud protection on SIP (restricted destinations, no open relay). Treat a spike in short calls or in minutes per number as an incident signal.

Operations checklist

  1. Capacity measured with a load test; admission control returns a clean refusal.

  2. Cost per minute computed for each route from current price pages, with idle time capped.

  3. AI disclosure played; recording consent captured per call; retention and deletion scheduled.

  4. PII redacted before transcripts reach logs, traces and datasets.

  5. Per-stage latency, interruption rate and outcome dashboards; alerts on p95 time to first audio.

  6. WER and retrieval hit rate sampled per channel.

  7. Recorded-call, simulated-caller, barge-in, telephony-smoke and chaos tests in the pipeline.

  8. Backup providers configured; human transfer, callback and after-hours path tested.

  9. Non-voice alternative and captions available.

  10. Spending caps, rate limits and SIP restrictions in place.

References