Turn-Taking and Interruptions

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against Silero VAD 6.2.3, TEN VAD 1.0, livekit-agents 1.8.3 with livekit-plugins-turn-detector 1.8.3, the pipecat-ai/smart-turn project with Pipecat 1.12.0, Deepgram Flux (flux-general-en, flux-general-multi), AssemblyAI streaming, OpenAI GPT-Live and the Realtime API, and the Twilio ConversationRelay and Media Streams message sets (see the version baseline). Turn-taking behaviour was not measured on live audio here; the thresholds quoted below are starting points documented by the projects or common practice, and every one should be tuned on recordings of your own callers.

Turn-taking is the part of a voice agent that callers judge without knowing it exists. An agent that answers too early cuts people off in the middle of a sentence; one that answers too late feels dead. One that keeps talking after the caller says "no, wait" feels rude. This page covers the machinery behind those three impressions.

Voice activity detection

A voice activity detector (VAD) classifies short audio frames as speech or non-speech. It is a small neural network that runs on a CPU in well under a millisecond per frame, so it can run on every call. Two open-source choices are common:

  • Silero VAD (MIT) is the default in Pipecat and LiveKit Agents. It takes 16 kHz (or 8 kHz) audio in small windows and returns a speech probability.

  • TEN VAD (from the TEN Framework) is a low-latency detector aimed at the same job, with a lighter footprint.

A VAD alone gives two decisions, "speech started" and "speech stopped for N milliseconds". Both are parameterised:

Parameter Effect

Speech probability threshold

Higher values ignore breathing and background noise but miss quiet speakers; lower values do the opposite.

Minimum speech duration

How long speech must last before "speech started" fires. Too short and coughs and door slams interrupt the agent.

Silence (stop) duration

How long silence must last before "speech stopped" fires. This is the main contributor to in the latency budget.

Padding

Audio kept before the detected start, so the first syllable is not clipped when the transcriber receives the segment.

Telephone audio at 8 kHz with codec artefacts and line noise needs a higher threshold than studio audio, and noisy environments (cars, call centres) benefit from a noise suppression step in front of the VAD; hosted APIs such as Azure Voice Live and the OpenAI APIs offer it as a session option.

Endpointing: when is the turn over?

Detecting that the caller stopped making sound is not the same as detecting that they finished their thought. "My order number is…​" followed by a pause to look something up is silence, not a finished turn. There are three levels of sophistication.

  1. Silence endpointing. Declare the turn over after a fixed silence, typically a few hundred milliseconds up to about one second. It is simple, adds that wait to every turn, and still cuts off slow speakers.

  2. Model-assisted endpointing. The speech recogniser itself reports the end of the utterance. Deepgram Flux emits EndOfTurn and a speculative EagerEndOfTurn event on its /v2/listen endpoint, so the agent can start preparing the answer early and cancel if the caller keeps talking. AssemblyAI’s streaming API marks turn boundaries with an end_of_turn field on its turn messages.

  3. Semantic turn detection. A small model reads the transcript (or the audio) and estimates whether the sentence is complete. The LiveKit turn detector (livekit-plugins-turn-detector) is a language model over the running transcript; Pipecat’s Smart Turn is an audio model that looks at prosody and the last seconds of speech. Both run alongside the VAD and extend the wait when the utterance sounds unfinished.

flowchart TD S[Speech frames] --> V{VAD: silence for N ms?} V -- no --> S V -- yes --> T{Turn model: sentence complete?} T -- "complete" --> E[End of turn: send transcript to the LLM] T -- "unfinished" --> W[Wait up to the maximum delay] W --> V2{Speech resumed?} V2 -- yes --> S V2 -- no --> E

The trade-off is the false-positive rate (interrupting the caller) against the added delay. A semantic detector typically lets you use a short VAD silence (say 300 to 500 ms) for obviously finished sentences and a longer maximum for unfinished ones, which lowers the average delay without raising the interruption rate. Test on recordings of hesitant speakers, numbers read aloud, and non-native speech, where the models are weakest.

Speech-to-speech APIs make the same choice as a session option. OpenAI exposes server-side VAD with a semantic mode alongside plain silence detection, Gemini Live has automatic activity detection with sensitivity and silence settings, and Azure Voice Live offers semantic end-of-turn detection; see Realtime APIs and check each vendor’s current option names before copying configuration.

agent.py: LiveKit Agents session with a VAD and the multilingual turn detector
from livekit.agents import AgentSession
from livekit.plugins import silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel

session = AgentSession(
    vad=silero.VAD.load(),
    turn_detection=MultilingualModel(),
    allow_interruptions=True,
    min_endpointing_delay=0.5,
    max_endpointing_delay=3.0,
    # stt=..., llm=..., tts=... are configured as shown on the LiveKit Agents page
)

Option names evolve between releases; confirm them against LiveKit: Turn handling and the LiveKit Agents page. For Pipecat, the VAD analyzer and the Smart Turn analyzer are attached to the transport or the user aggregator; the current configuration is in the Smart Turn repository and the Pipecat page.

Barge-in

Barge-in is the caller speaking while the agent is speaking. Handling it well takes four actions, in this order and as fast as possible:

  1. Detect it. The VAD fires "speech started" while the agent is in the speaking state. Require a minimum speech duration (200 to 300 ms) or a few recognised words, otherwise a cough or an echo stops the agent.

  2. Stop the audio. Cancel the TTS request and flush every audio buffer between the synthesiser and the caller. The buffers are the usual bug: audio already sent to the transport keeps playing unless the transport is told to clear it. Twilio Media Streams has a clear message for exactly this, ConversationRelay raises an interrupt message, and WebRTC playback is stopped by the client when it learns of the interruption.

  3. Cancel the generation. Cancel the running LLM stream, and any tool call that has not started side effects.

  4. Truncate the history. The conversation memory must contain what the caller actually heard, not the full answer the LLM had planned. Otherwise the next turn is confused: the caller says "what was the second step?" and the model believes it already spoke all five.

History truncation needs a record of how much audio reached the speaker. Speech-to-speech APIs provide this: the OpenAI Realtime API accepts a conversation.item.truncate event with the audio end offset in milliseconds, and GPT-Live and Gemini Live handle interruption inside the session. In a cascaded pipeline you estimate it from the playback position or, for telephony, from the mark acknowledgements Twilio returns when queued audio reaches the caller.

barge_in.py: framework-neutral interruption handler for a cascaded pipeline
import asyncio


class TurnState:
    def __init__(self) -> None:
        self.speaking_task: asyncio.Task | None = None
        self.spoken_text = ""  # text whose audio was confirmed played


async def on_user_speech_started(state: TurnState, transport, memory, thread_id: str) -> None:
    if state.speaking_task and not state.speaking_task.done():
        state.speaking_task.cancel()          # stops LLM stream and TTS requests
        await transport.clear_output_audio()   # flush queued audio (Twilio: send "clear")
        await memory.replace_last_assistant(thread_id, state.spoken_text + " [interrupted]")

This is a sketch, not a library API: transport and memory stand for whatever your transport and memory classes provide. The Twilio clear and mark messages are specified in the Twilio Media Streams WebSocket messages reference.

stateDiagram-v2 [*] --> Listening Listening --> Thinking: end of turn Thinking --> Speaking: first audio chunk Thinking --> Listening: caller resumes (eager end of turn cancelled) Speaking --> Listening: answer finished Speaking --> Interrupted: caller speech confirmed Interrupted --> Listening: audio flushed, history truncated Listening --> Listening: backchannel or noise ignored

Echo cancellation

If the caller’s speaker output reaches their microphone, the agent hears itself and interrupts itself. Who removes the echo depends on the transport.

Transport Echo handling

Browser, WebRTC or getUserMedia

The browser applies acoustic echo cancellation when the constraint echoCancellation is on (it is by default), together with noise suppression and auto gain control. Keep these constraints enabled for headsets-free use. See Voice in the Browser.

Phone network (PSTN)

The carrier and the handset handle echo; the agent usually receives clean far-end audio. Speakerphone use on the caller’s side can still leak the agent’s voice back and needs the stricter minimum-speech rule above.

SIP and PBX

Echo can appear between the PBX and the phone; an echo canceller in the PBX or the SIP gateway is configured per channel. See Telephony with Asterisk and SIP.

Server-side audio capture

Raw audio sent over a WebSocket has no echo cancellation unless the client applied it. Apply it on the client, or accept that barge-in will need a higher threshold.

A practical fallback is to suppress barge-in detection during the first few hundred milliseconds of each agent utterance, and to require recognised words, not only energy, before treating speech as an interruption.

DTMF

Phone callers can press keys, and the keys arrive as dual-tone multi-frequency (DTMF) events, not as speech. They are the reliable way to enter an account number, confirm a choice or escape a loop. Twilio ConversationRelay delivers a dtmf message to the WebSocket and Media Streams delivers dtmf events; Asterisk raises DTMF events on ARI channels; see Telephony with Twilio. Treat DTMF as input to the same ask(conversationId, userId, text) core (for example, as the text "caller pressed 2"), and offer it as an alternative whenever speech recognition fails twice.

Silence and timeouts

Every state needs a timeout:

  • Caller silent after the agent’s question. Re-prompt once after a few seconds ("Are you still there?"), then offer DTMF, then end the call politely.

  • Caller silent in the first seconds. On outbound calls, silence may mean voicemail or an answering machine; hand over to the voicemail logic rather than speaking into it.

  • Agent thinking too long. If the LLM or a tool has not produced a first token after a threshold, say a short filler ("One moment") so the line does not go dead, and give up with an apology at a hard limit.

  • Whole-call limit. Cap the call duration so a stuck loop cannot run up cost.

Backchannels and fillers

People say "mm-hm", "right" and "okay" while the other party talks, without taking the turn. A VAD treats them as speech and may stop the agent for no reason. Options are to require a minimum number of words before an interruption counts, to list known backchannel words that the pipeline ignores while the agent speaks, or to use a model that classifies the interruption. In the other direction, an agent can emit its own short acknowledgements ("I see", "let me check") to cover tool latency; deliver them as a separate, pre-synthesised clip so they cost no synthesis time.

A tuning checklist

  1. Record 50 real calls (with consent; see Production Voice Agents).

  2. Replay them with different VAD thresholds and endpointing delays and count false interruptions and cut-offs.

  3. Check that barge-in stops audible output within about 300 ms and that the history shows what was heard.

  4. Test with a speakerphone, a headset, a noisy room and a bad mobile connection.

  5. Re-run after every change of model, codec or transport.

References