Voice in the Browser

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against the browser APIs as documented by MDN and the W3C (Media Capture and Streams, WebRTC, Web Audio AudioWorklet, Web Speech), aiortc 1.15.0 under Pipecat’s SmallWebRTC transport, pipecat-ai 1.12.0, LiveKit Agents 1.8.3 with the LiveKit JavaScript client, the OpenAI Realtime API, the ElevenLabs Agents SDKs, FastAPI 0.142.2 on Python 3.12, and Spring AI 2.0.1 on Spring Boot 4.1.x with Java 21 (see the version baseline). Nothing here was run: real-time audio needs a microphone, a browser and, for the hosted services, paid accounts, so every snippet was written from the official documentation linked after it and is illustrative, not compiled. The Pipecat /api/offer body shape in particular follows the project’s published examples and should be checked against the pipecat-examples repository before use, and the behaviour of iOS Safari and background tabs is described from vendor and MDN documentation, not from device testing.

The browser is where most DocsAssistant voice sessions start. Whatever runs behind it (a realtime API, a hosted agent, a Pipecat or LiveKit worker, or a hand-built pipeline), the page has the same four jobs: capture the microphone with echo control, move audio to the agent with low delay, play the reply without gaps and stop it the moment the user speaks, and obtain a credential without shipping a vendor key to the client. This page covers those jobs once; the transport-specific details of each platform are on Realtime APIs, ElevenLabs Agents, Pipecat and LiveKit Agents.

sequenceDiagram participant B as Browser participant A as DocsAssistant API (FastAPI or Spring) participant V as Voice backend (SmallWebRTC, LiveKit, OpenAI, ElevenLabs) B->>B: getUserMedia (echo cancellation, noise suppression) B->>A: POST /api/voice/token (session cookie) A->>V: mint short-lived credential or room token V-->>A: token A-->>B: token B->>V: SDP offer (RTCPeerConnection, ICE candidates) V-->>B: SDP answer B-->>V: ICE connectivity checks (STUN, TURN relay if needed) B->>V: Opus audio over SRTP V-->>B: synthesised reply over SRTP B->>V: user barge-in (new speech) V-->>B: reply stops, new turn starts

Capturing the microphone

navigator.mediaDevices.getUserMedia() asks the user for the microphone and returns a MediaStream. It works only in a secure context (HTTPS or localhost) and the permission prompt must be triggered by something the user did, so the call belongs in the click handler of the "Ask DocsAssistant" button, never on page load. The audio constraints that matter for a voice agent are echoCancellation, noiseSuppression and autoGainControl. Echo cancellation is the one that cannot be skipped: without it the agent’s own voice, played from the loudspeaker, re-enters the microphone and the agent interrupts itself (the failure analysed on Turn-Taking and Interruptions).

mic.js: microphone capture with processing constraints
export async function openMicrophone() {
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: {
      echoCancellation: true,
      noiseSuppression: true,
      autoGainControl: true,
      channelCount: 1,
      sampleRate: { ideal: 48000 },   // a hint: the browser may give another rate
    },
    video: false,
  });

  const [track] = stream.getAudioTracks();
  const applied = track.getSettings();       // what the browser actually granted
  console.log("echoCancellation:", applied.echoCancellation,
              "noiseSuppression:", applied.noiseSuppression);
  return stream;
}

Source: MDN: MediaDevices.getUserMedia() and MDN: MediaTrackConstraints.echoCancellation. Constraints given as plain values are ideal requests, not guarantees; read track.getSettings() to see what was applied and show a headset hint when echo cancellation is off. Browsers apply these as part of their WebRTC audio processing, so a stream captured this way and sent over a peer connection is already cleaned; a stream read through an AudioWorklet for a WebSocket (below) is the same stream and keeps the processing. Server-side noise suppression, described on Speech to Text, is a second layer and not a substitute.

Transport choice: WebRTC or WebSocket

WebRTC WebSocket with raw PCM

Transport

UDP (SRTP, Opus), ICE negotiates the path; falls back to TURN over TCP/TLS.

One TCP connection, so head-of-line blocking after a packet loss.

Browser audio work

Done by the browser: Opus encoding, echo cancellation, packet-loss concealment, jitter buffer, clock drift.

Done by you: capture, resampling, framing, playback scheduling, jitter buffer.

Latency under loss

Best; late packets are dropped or concealed.

Stalls then bursts on a bad mobile network.

Server complexity

An SFU or a library such as aiortc, plus signalling, STUN and TURN.

A WebSocket route; every framework and proxy already supports it.

Bandwidth

Opus at about 24 to 32 kbit/s.

16 kHz 16-bit mono PCM is 256 kbit/s unless you compress it.

Fit

Production voice in browsers and phones; what the hosted platforms use.

Prototypes, corporate networks that block UDP, and hand-built pipelines.

The rule used on the rest of the sub-section: use WebRTC whenever the backend offers it, and use the WebSocket path when you own the whole pipeline (see Building a Voice Pipeline by Hand) or when a network blocks UDP. The data-channel side of WebRTC, useful for sending transcripts and tool events over the same connection, is covered on WebRTC Data Channels.

WebRTC to Pipecat SmallWebRTC

Pipecat’s SmallWebRTC transport runs a peer connection in your Python process using aiortc, so there is no media server to operate. The browser creates an RTCPeerConnection, adds the microphone track, sends the SDP offer to an HTTP endpoint of your server (the examples use /api/offer), and applies the SDP answer that comes back. The server reply carries a pc_id that identifies the connection for later renegotiation.

offer.js: browser side of a SmallWebRTC session
export async function connectToPipecat(stream, audioElement) {
  const pc = new RTCPeerConnection({
    iceServers: [{ urls: "stun:stun.l.google.com:19302" }],
  });

  stream.getAudioTracks().forEach((track) => pc.addTrack(track, stream));

  // the agent's voice arrives as a remote audio track
  pc.ontrack = (event) => {
    if (event.track.kind === "audio") {
      audioElement.srcObject = event.streams[0];
      audioElement.play().catch(() => {});      // allowed: we are inside a user gesture chain
    }
  };

  const offer = await pc.createOffer();
  await pc.setLocalDescription(offer);

  // wait for ICE gathering to finish so the offer carries all candidates
  await new Promise((resolve) => {
    if (pc.iceGatheringState === "complete") return resolve();
    pc.addEventListener("icegatheringstatechange", () => {
      if (pc.iceGatheringState === "complete") resolve();
    });
  });

  const response = await fetch("/api/offer", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    credentials: "include",
    body: JSON.stringify({
      sdp: pc.localDescription.sdp,
      type: pc.localDescription.type,
    }),
  });
  const answer = await response.json();     // { sdp, type, pc_id }
  await pc.setRemoteDescription({ sdp: answer.sdp, type: answer.type });
  return { pc, pcId: answer.pc_id };
}

Source: MDN: RTCPeerConnection for the browser calls and Pipecat: SmallWebRTC transport for the transport; the exact JSON fields of /api/offer follow the p2p-webrtc example in pipecat-examples and were not confirmed against a running server. Waiting for complete ICE gathering is the simplest signalling (one request, one response); trickle ICE, where candidates are sent as they appear, shortens connection time and uses a PATCH request in the Pipecat examples. For production, the Pipecat client SDK (@pipecat-ai/client-js with the SmallWebRTC transport package) wraps this and adds events for transcripts and bot speaking state.

WebRTC to LiveKit, OpenAI and ElevenLabs

The other platforms hide the peer connection behind an SDK or a single HTTP call. What changes between them is who issues the credential and where the SDP goes.

Platform Browser connects with Your backend’s job

LiveKit

livekit-client Room.connect(url, token) followed by room.localParticipant.setMicrophoneEnabled(true); the agent worker joins the same room (see LiveKit Agents).

Sign a JWT access token with the room name and identity using the LiveKit API key and secret.

OpenAI Realtime

Create an RTCPeerConnection, add the microphone track, open the oai-events data channel and POST the SDP offer to /v1/realtime/calls with the ephemeral client secret as the bearer token and Content-Type: application/sdp.

Mint the client secret with POST /v1/realtime/client_secrets (see Realtime APIs); never send the API key to the page.

ElevenLabs Agents

@elevenlabs/client or @elevenlabs/react startSession with an agent ID or a conversation token; WebRTC is the default for voice (see ElevenLabs Agents).

For a private agent, fetch a conversation token from ElevenLabs with the API key and return it.

Pipecat SmallWebRTC

The offer shown above, to your own endpoint.

Run the endpoint and authenticate it like any other API route.

Whichever platform is used, the browser-side control loop is the same: show the connection state, mute by disabling the audio track (track.enabled = false) rather than closing the connection, and close the peer connection and stop the tracks (track.stop()) when the call ends so the browser releases the microphone indicator.

Minting tokens on the backend

A browser must never hold the vendor API key. The pattern on every platform above is a small authenticated endpoint of the DocsAssistant backend that exchanges the user’s session for a short-lived credential scoped to one call. The authentication dependency is the same one used on the text endpoints, so the access-control rules decide who may start a voice session.

token.py: LiveKit access token for the signed-in user (FastAPI)
import uuid

from fastapi import Depends, FastAPI
from livekit import api

from docsassistant.config import settings
from docsassistant.security import User, current_user

app = FastAPI()


@app.post("/api/voice/token")
async def voice_token(user: User = Depends(current_user)) -> dict:
    room = f"docs-{uuid.uuid4().hex[:12]}"
    token = (
        api.AccessToken(settings.livekit_api_key, settings.livekit_api_secret)
        .with_identity(user.id)
        .with_grants(api.VideoGrants(room_join=True, room=room))
        .to_jwt()
    )
    return {"url": settings.livekit_url, "room": room, "token": token}

Source: LiveKit: Authentication and access tokens (token creation with AccessToken and VideoGrants). Dispatching the agent into the room is configured on the LiveKit side, as described on LiveKit Agents. Keep the token lifetime short and generate a new room per call.

VoiceTokenController.java: ElevenLabs conversation token (Spring)
@RestController
class VoiceTokenController {

    private final RestClient elevenLabs;
    private final String agentId;

    VoiceTokenController(RestClient.Builder builder,
                         @Value("${elevenlabs.api-key}") String apiKey,
                         @Value("${elevenlabs.agent-id}") String agentId) {
        this.elevenLabs = builder.baseUrl("https://api.elevenlabs.io")
                .defaultHeader("xi-api-key", apiKey)
                .build();
        this.agentId = agentId;
    }

    @PostMapping("/api/voice/token")
    Map<String, Object> token(Authentication authentication) {   // already authenticated by Spring Security
        return elevenLabs.get()
                .uri(uri -> uri.path("/v1/convai/conversation/token")
                        .queryParam("agent_id", agentId).build())
                .retrieve()
                .body(new ParameterizedTypeReference<>() {});     // {"token": "..."}
    }
}

Source: ElevenLabs: JavaScript SDK (private agents and the conversation token) with Spring Framework: REST clients for RestClient. The same shape serves OpenAI: replace the call with POST /v1/realtime/client_secrets as shown on Realtime APIs.

The WebSocket PCM alternative

When the backend is a hand-built pipeline that wants raw audio, the browser sends PCM frames over a WebSocket and plays back the PCM that returns. The server side of the socket is the same as any other (see WebSockets Next for a Java example); the work is in the browser.

Capture uses an AudioWorklet, which runs on the audio rendering thread and so is not delayed by the page’s main thread. The processor receives 128-sample blocks of 32-bit floats at the context’s own rate. It resamples them to 16 kHz, converts them to 16-bit integers and posts fixed-size frames of 512 samples (32 ms). That is the frame size the Silero VAD in Building a Voice Pipeline by Hand requires at 16 kHz.

pcm-capture.js: AudioWorklet processor and main-thread sender
// pcm-capture.worklet.js -- runs in the AudioWorkletGlobalScope (sampleRate is the context's rate)
class PcmCapture extends AudioWorkletProcessor {
  constructor({ processorOptions }) {
    super();
    this.targetRate = processorOptions.targetRate;   // 16000
    this.frame = new Int16Array(processorOptions.frameSamples);   // 512 samples per message
    this.filled = 0;
    this.step = sampleRate / this.targetRate;         // e.g. 48000 / 16000 = 3
    this.position = 0;                                // fractional read position into the current block
    this.last = 0;                                    // last sample of the previous block (for position -1..0)
  }

  process(inputs) {
    const channel = inputs[0][0];                     // first input, first channel (Float32Array, 128 samples)
    if (!channel) return true;
    // Linear-interpolation resampler; position and the last sample carry over between blocks so there are no seams.
    const at = (k) => (k < 0 ? this.last : channel[k]);
    for (; this.position < channel.length - 1; this.position += this.step) {
      const i = Math.floor(this.position);
      const t = this.position - i;
      const s = Math.max(-1, Math.min(1, at(i) * (1 - t) + at(i + 1) * t));
      this.frame[this.filled++] = s < 0 ? s * 0x8000 : s * 0x7fff;
      if (this.filled === this.frame.length) {        // a full 512-sample frame: hand it to the main thread
        this.port.postMessage(this.frame.buffer, [this.frame.buffer]);
        this.frame = new Int16Array(this.frame.length);
        this.filled = 0;
      }
    }
    this.last = channel[channel.length - 1];
    this.position -= channel.length;
    return true;                                      // keep the processor alive
  }
}
registerProcessor("pcm-capture", PcmCapture);

// main.js
export async function startCapture(stream, socket) {
  const context = new AudioContext();                 // default (device) rate: works in every browser
  await context.audioWorklet.addModule("/pcm-capture.worklet.js");
  const source = context.createMediaStreamSource(stream);
  const node = new AudioWorkletNode(context, "pcm-capture", {
    processorOptions: { targetRate: 16000, frameSamples: 512 },
  });
  node.port.onmessage = (event) => {
    if (socket.readyState === WebSocket.OPEN) socket.send(event.data);   // 1024 bytes of PCM16 per message
  };
  source.connect(node);
  return context;
}

Source: MDN: AudioWorkletProcessor, MDN: Using AudioWorklet and MDN: AudioContext() constructor. The context is created at the default rate on purpose. Firefox refuses to connect a microphone MediaStreamAudioSourceNode to an AudioContext created with a different sampleRate ("Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported"), so asking for { sampleRate: 16000 } works in Chromium but fails there. Resampling in the worklet keeps one code path for all browsers. The linear interpolator is adequate for speech recognition; use a windowed-sinc resampler if the audio is also recorded for quality. A server that needs a different frame size changes only frameSamples. Sending the raw PCM bytes needs socket.binaryType = "arraybuffer" on the receiving side for the reply.

Playback is where WebSocket pipelines usually go wrong. Chunks arrive at irregular intervals, and playing each one as it lands produces clicks and gaps. A jitter buffer queues a small amount of audio before starting (typically 60 to 120 ms) and schedules each chunk to start exactly where the previous one ended, using the audio clock rather than timers.

pcm-playback.js: scheduled playback with a small jitter buffer
export class PcmPlayer {
  constructor(sampleRate = 24000, prebufferSeconds = 0.08) {
    this.context = new AudioContext({ sampleRate });
    this.prebuffer = prebufferSeconds;
    this.nextTime = 0;
    this.sources = new Set();
  }

  enqueue(arrayBuffer) {
    const int16 = new Int16Array(arrayBuffer);
    const buffer = this.context.createBuffer(1, int16.length, this.context.sampleRate);
    const data = buffer.getChannelData(0);
    for (let i = 0; i < int16.length; i++) data[i] = int16[i] / 0x8000;

    const source = this.context.createBufferSource();
    source.buffer = buffer;
    source.connect(this.context.destination);

    const now = this.context.currentTime;
    if (this.nextTime < now) this.nextTime = now + this.prebuffer;   // underrun: re-prime the buffer
    source.start(this.nextTime);
    this.nextTime += buffer.duration;

    this.sources.add(source);
    source.onended = () => this.sources.delete(source);
  }

  // barge-in: drop everything that has not played yet
  flush() {
    for (const source of this.sources) { try { source.stop(); } catch {} }
    this.sources.clear();
    this.nextTime = 0;
  }
}

Source: MDN: AudioBufferSourceNode and MDN: BaseAudioContext.createBuffer(). The flush() method is what the page calls when the server sends an interruption event; the server must stop generating at the same moment, and the played-versus-sent bookkeeping that keeps the conversation history honest is described on Turn-Taking and Interruptions. A larger prebuffer smooths a bad network at the price of latency: the added delay is the prebuffer length , which belongs in the budget on Voice Architectures next to the network and processing terms.

The Web Speech API

The Web Speech API offers SpeechRecognition (speech to text) and speechSynthesis (text to speech) with no server code, which makes it tempting for a prototype. Its limits are the reason it is not used for the DocsAssistant product path:

  • Support varies. SpeechRecognition is exposed with a webkit prefix in Chromium-based browsers, behaves differently in Safari, and is not available everywhere; MDN marks it limited availability.

  • Audio may go to a vendor. In Chromium-based browsers recognition is typically performed by a remote service, so audio leaves the device with no contract of yours behind it. The specification and MDN note that the service is implementation-defined; some browsers offer on-device recognition for specific languages through options that are still evolving.

  • No control over models or turn detection. There is no custom vocabulary of your choosing, no VAD tuning and no streaming into your own turn-taking logic beyond interim results.

  • Synthesis voices are whatever the operating system ships, and differ per device, so the DocsAssistant would sound different for every user.

A speech-to-text step you control, described on Speech to Text, or a realtime API, is the production answer. speechSynthesis is still a reasonable fallback to read an error message aloud, and a useful check that the loudspeaker path works.

Mobile browser caveats

  • Autoplay policy. Browsers block audio playback and keep an AudioContext in the suspended state until the page has had a user gesture. Create or resume() the context, and call audioElement.play(), inside the button’s click handler.

  • iOS Safari. All iOS browsers use WebKit, and each has its own history of WebRTC and Web Audio quirks: the microphone permission is requested per page, an AudioContext may be interrupted by calls and by the silent switch, and sample rates and AudioWorklet behaviour have differed between versions. Test on a real device for every release you support, and keep a visible "tap to start" control rather than starting automatically.

  • Background tabs and locked screens. Browsers throttle timers and may suspend audio and network activity for hidden tabs, and most mobile browsers stop microphone capture when the page is not visible. Treat a hidden page as a paused call: detect document.visibilitychange, tell the user, and reconnect rather than assuming the session survived.

  • Audio routing. Switching between earpiece, speaker and Bluetooth can change the sample rate and re-trigger echo cancellation; listen for devicechange and re-open the stream if the track ends.

  • Data cost and battery. A voice call keeps the radio active; Opus over WebRTC is far cheaper than uncompressed PCM over a WebSocket.

For a call that must survive a locked screen, a native SDK (the ElevenLabs Swift and Kotlin SDKs, LiveKit’s mobile clients) or a real telephone call, covered on Telephony with Twilio, is the dependable route.

STUN and TURN

WebRTC peers sit behind NATs and firewalls. STUN servers tell a peer its public address so that a direct path can be tried. When no direct path exists (symmetric NATs, corporate firewalls that block UDP), a TURN server relays the media. ICE gathers host, server-reflexive (STUN) and relay (TURN) candidates and picks the best working pair. For a voice agent the server usually has a public address, so direct connectivity succeeds more often than for peer-to-peer calls, but the browser’s network is outside your control, and a deployment that works in the office and fails on a hotel network is a missing TURN server.

ice-servers.js: STUN plus TURN with short-lived credentials
// credentials come from your backend, never hard-coded in the page
const { iceServers } = await (await fetch("/api/voice/ice", { credentials: "include" })).json();
// e.g. [
//   { urls: "stun:turn.example.com:3478" },
//   { urls: ["turn:turn.example.com:3478?transport=udp",
//            "turns:turn.example.com:443?transport=tcp"],
//     username: "1767225600:alice", credential: "base64-hmac" }
// ]
const pc = new RTCPeerConnection({ iceServers });

Source: MDN: RTCPeerConnection() constructor (the iceServers option) and RFC 8489: STUN with RFC 8656: TURN. A self-hosted coturn server supports time-limited credentials derived from a shared secret, which the backend computes per request; offering TURN on TCP port 443 with TLS lets media pass most proxies at the cost of extra latency. The SFU used by LiveKit and the hosted platforms ship their own TURN, so this section matters most for SmallWebRTC and other self-hosted peers. Relayed media travels through your TURN server, so account for its bandwidth.

Accessibility

A voice interface is an extra way to interact, not a replacement for the page. Keep the text conversation available, show live captions of the transcript (for deaf and hard-of-hearing users and for noisy rooms), label the start and mute buttons, expose the call state to assistive technology with an aria-live region, and do not rely on colour alone for the listening and speaking indicators. The general rules and conformance levels are on Web Accessibility.

Choosing a browser voice path

Use the platform’s own WebRTC SDK (LiveKit, ElevenLabs, OpenAI) when you chose that platform: the SDK handles the peer connection, and your code is the token endpoint. Use Pipecat SmallWebRTC when the pipeline is yours and you want a single Python process without a media server. Use the WebSocket PCM path only for hand-built pipelines or networks that block UDP, and accept that jitter buffering, resampling and barge-in flushing are then your code. Add a TURN server before the first user outside your office. Reserve the Web Speech API for prototypes and fallbacks, always start audio from a user gesture, test on real iOS and Android devices, and keep the text channel beside the voice one.

References