LiveKit Agents

Four terms carry the whole model:

Because each job is a subprocess, a crash or a memory leak in one call does not take down the others, and capacity is controlled by how many agent-server processes you run and how many jobs each accepts (load reporting lets the LiveKit server stop dispatching to a busy process). Dispatch is automatic by default (every new room gets an agent) or explicit: give the agent a name and dispatch it through the API or a SIP dispatch rule, which is what a telephone deployment uses.

The agent talks to the room, not to the caller’s network, so the same agent code serves a browser tab and a telephone call. The caller-to-room leg and its WebRTC details are the subject of WebRTC Data Channels and, for audio in the browser, of Voice in the Browser.

AgentSession is the orchestrator inside a job. It owns the audio input and output, turn detection and interruption handling, and the model components, and it runs one active Agent (instructions plus tools) at a time. Agent subclasses are the unit you can hand a conversation between, for example from a greeter to a specialist. The model components can be constructed with plugins (livekit.plugins.openai, deepgram, elevenlabs, cartesia, …​) or, in the current documentation, through LiveKit’s inference gateway, which selects hosted STT, LLM and TTS models by name from LiveKit Cloud.

Source: LiveKit docs: Voice AI quickstart (the model names are the quickstart’s examples and change often; pick yours from the model pages). The file registers an agent server under the dispatch name docs-assistant; python agent.py dev (or lk agent dev) runs it locally against your project, and python agent.py start runs the production mode. The greeting states that the caller is speaking to an AI, which matters on telephone lines (see Telephony with Twilio).

The same AgentSession supports the two architectures described on Voice Architectures. The pipeline form above is the cascade: separate STT, LLM and TTS, with a turn detector deciding when the user has finished. The realtime form gives the session a speech-to-speech model as its llm and lets the provider handle recognition and synthesis; provider-side turn detection then applies, and you trade control for a lower latency floor, as discussed on Realtime APIs.

Source: LiveKit docs: OpenAI Realtime API plugin. Gemini Live has an equivalent plugin. The second form is useful when the voice must be a particular cloned or branded voice while the model keeps native audio understanding; it gives up the model’s own prosody. Whether the realtime model or a pipeline hits your latency budget is measured with the methods in Production Voice Agents, not assumed.

The framework’s voice-activity detection (Silero) finds speech and silence, but silence is a poor proxy for "finished": a pause in the middle of a sentence is not an end of turn. LiveKit therefore ships a small turn detector language model that reads the transcript so far, together with the acoustic signal, and predicts whether the speaker has finished. It runs on the agent’s CPU in tens of milliseconds, is released as the livekit-plugins-turn-detector package (1.8.3, in step with livekit-agents), and has English and multilingual variants whose weights are fetched with a download-files command before the first run. The general theory, including why endpointing delay trades latency for cut-offs, is on Turn-Taking and Interruptions.

Source: LiveKit docs: Turn detection and interruptions (the interruption keys and TurnHandlingOptions; the MultilingualModel import path follows the plugin’s package layout and could not be re-checked against the current page). Install with pip install "livekit-agents[silero,turn-detector]" and run python agent.py download-files once, also in the container image build, so that the weights are not downloaded on the first call. The hosted inference.TurnDetector() in the earlier example runs the equivalent model on LiveKit Cloud. A false interruption is a cough or a "mm-hm" that stops the agent without a real barge-in; the resume option lets the agent continue the sentence it was saying.

A voice agent that answers from documentation has two designs. In the first, the voice layer is only an adapter: every user turn is forwarded to the same ask(conversationId, userId, text) operation that the text channels use, and the reply is spoken as it streams (the approach used for ElevenLabs' custom LLM). In the second, the LiveKit LLM is the front-line model and the DocsAssistant is a tool that it calls when it needs facts. The second keeps small talk, clarification and "one moment" phrases cheap and local, and fits LiveKit’s tool model.

Source: LiveKit docs: Tool definition and use (function tools are methods decorated with @function_tool, with the docstring becoming the tool description; RunContext gives access to the session and its data). The user ID should come from the verified caller identity, not from what the model passes in, so the ACL filter of Access Control and Privacy still applies end to end. The endpoint can be the FastAPI service of Serving RAG as an API. A tool call adds a model round trip plus retrieval to the first-audio latency, so have the agent say a short filler before long lookups, or run the lookup as a background tool, and return answers already shortened for the ear. Reasoning about which tool a model selects, and the injection risks of reading retrieved text aloud, are covered on Tools and in Prompt Injection.

MCP servers can also be attached to an agent (Python SDK), which is the route when the DocsAssistant already runs behind an MCP endpoint as in RAG Behind MCP.

If the DocsAssistant is already a LangChain 1.4 create_agent graph, the livekit-agents[langchain] extra provides an LLMAdapter that wraps a compiled LangGraph graph (which is what create_agent returns) so that it acts as the session’s llm. The graph keeps its tools, retrieval and checkpointed memory; LiveKit supplies speech and turn-taking.

Source: LiveKit docs: LangChain and LangGraph integration (adapter parameters graph, stream_mode, config, context, subgraphs; the import alias and the exact constructor call follow that page’s description and were not run). The adapter supports graphs only: a plain LCEL chain such as prompt | llm or a bare chat model is not accepted. The default messages stream mode forwards LLM tokens as they arrive, which gives the best time to first audio; custom mode lets a node write explicit text such as "looking that up" through the stream writer. Use the room name or call ID as the thread_id, so memory follows the call, as described in Short-Term Memory and Checkpointers and, for streaming, in Streaming.

LiveKit SIP is a service that bridges a SIP provider (a Twilio Elastic SIP trunk, Telnyx, a carrier, or your own Asterisk) to LiveKit rooms. A caller becomes a participant, so the agent does not know or care that the audio came from a telephone. Three objects configure it:

Source: LiveKit docs: Inbound trunks.

Source: LiveKit docs: Dispatch rules (the numbers field is from the SIP inbound-trunk API reference; the page example sets only the name). Create them with lk sip inbound create inbound-trunk.json and lk sip dispatch create dispatch-rule.json, then point the provider’s trunk at your LiveKit SIP endpoint. An individual rule creates a new room per caller, so every call gets its own job and its own agent, and the dispatch metadata arrives in the job as ctx.job.metadata. Create the rule once and reuse it; do not create one per call.

Source: LiveKit docs: Make outbound calls. The outbound trunk is created with lk sip outbound create from a JSON file with the provider address, numbers and credentials. Dispatching the job with the number in its metadata (lk dispatch create --agent-name docs-outbound --metadata …​ or the API) keeps the dialing code out of the front end. For an outbound call, start the speech pipeline only after the callee answers, and watch for voicemail: a machine greeting must not be treated as a person (see Production Voice Agents). Provider-side setup is on Telephony with Twilio and Telephony with Asterisk and SIP.

The LiveKit server is open source (Apache 2.0) and the same SDKs and agent code run against either deployment.

Source: LiveKit docs: Deploy to a VM (requirements: domain with a trusted certificate, load balancer or reverse proxy for TLS, embedded TURN, Redis for distributed mode, an RTC port range, use_external_ip in clouds, host networking for Docker). Key names are from the server configuration reference; verify them against 1.13.7. For a local trial, livekit-server --dev starts with built-in development keys, and the container images are covered on the pages under Docker. Agent servers are ordinary stateless containers: they need the LiveKit URL and API key and secret, plus the model-provider keys, and scale by adding replicas.

The Agents framework also exists for Node.js and TypeScript (@livekit/agents with plugin packages such as @livekit/agents-plugin-openai). The concepts are identical (agent server, job, voice.Agent, voice.AgentSession, function tools with a schema), which makes it the natural choice when the DocsAssistant back end and the front end are already TypeScript, for instance next to the Quarkus or Spring services of the earlier pages calling it over HTTP.

Source: LiveKit docs: Voice AI quickstart (Node.js tab). The Node snippet is reconstructed from the SDK’s structure and was not checked line by line. The Node SDK has fewer plugins than the Python SDK and, as noted in the documentation, no MCP support; the LangChain adapter is Python only. For a Java shop that wants no Python process at all, the practical route is the Node agent calling the Spring AI service over HTTP, or the realtime route on Realtime APIs.

Choose LiveKit Agents when you need telephony and browser or mobile clients on the same agent, want an open-source stack you can self-host later, and are comfortable running agent-server processes. Its SIP service, dispatch rules and process model are the main differentiators; the turn detector and the interruption handling are good defaults for a conversational agent.

Whatever the choice, test with recorded calls, measure latency per stage, and plan for failure paths (provider outages, dropped SIP legs, agent-server restarts mid-call); those are the topics of Production Voice Agents, which also points at the LLMOps & Evaluation material for scoring conversations.