Recording and Media Processing

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

See the section bibliography for the reference material consulted while preparing these pages.

A media track is not only something to send to a peer. The same MediaStreamTrack can be written to a file, painted onto a canvas together with other tracks, measured, filtered frame by frame, or rewritten after encoding and before packetising. This page covers those five jobs in the browser and then explains why many products record on the server instead. Every example is a small piece of Huddle (the videoconference) or of the voice notes in Chatterbox (the chat). The capture side is in Media Capture and Devices, the connection side in Peer Connections and SDP.

Baseline, checked on 2026-10-24: the W3C MediaStream Recording specification is a Working Draft of 2026-03-16, Media Capture from DOM Elements (canvas captureStream) is a Working Draft of 2025-02-12 that states it is not intended for implementation although every engine ships it, Media Capture Transform (MediaStreamTrackProcessor, VideoTrackGenerator) is a Working Draft of 2026-04-16, and WebRTC Encoded Transform (RTCRtpScriptTransform) is a Working Draft of 2026-06-25. The examples were run in headless Chrome 155 with fake devices, served from localhost.

Which scenarios use it

  • Huddle uses all of it: a local recording of the meeting grid, background blur on the camera, an encoded transform for frame metadata or encryption, and a speaking indicator from audio levels. Recording of the whole meeting normally happens on the server, see Server-side recording through the SFU.

  • Chatterbox uses MediaRecorder for voice notes, uploaded as they are recorded.

  • Pixel Arena uses captureStream to let a player record or stream a match from the game canvas, and little else.

flowchart LR cam[Camera and microphone track] --> proc[Track processor] proc --> work[Worker: per-frame filter] work --> gen[Track generator] gen --> pc[RTCPeerConnection sender] pc --> enc[Encoder] enc --> tr[Script transform: encoded frames] tr --> net[Network] gen --> rec[MediaRecorder] canvas[Canvas grid] --> cs[captureStream] cs --> rec rec --> up[Chunk upload]

MediaRecorder

new MediaRecorder(stream, options) encodes the tracks of a MediaStream with the browser’s own encoders and hands the result back as Blob chunks. The options are mimeType (container and codecs), audioBitsPerSecond, videoBitsPerSecond, bitsPerSecond (one total that overrides the other two), audioBitrateMode and the keyframe interval options. If the mimeType is not supported the constructor throws NotSupportedError; if you omit it the browser picks a format, which you read back from recorder.mimeType.

The lifecycle is a small state machine whose state is inactive, recording or paused:

  • start(timeslice) begins recording. Without timeslice one dataavailable event is fired at the end; with it, an event is fired at least every timeslice milliseconds (browsers may impose a longer minimum). It throws InvalidStateError if already started and NotSupportedError if the stream has no live tracks.

  • pause() and resume() suspend encoding. requestData() flushes what has been collected so far into a dataavailable event and starts a new blob.

  • stop() fires a last dataavailable (possibly with an empty blob) and then stop. The error event carries a DOMException. Stopping the recorder does not stop the tracks: the microphone light stays on until you call stop() on each track.

Choosing a format

MediaRecorder.isTypeSupported(type) is synchronous and answers for the container and codec string. The answer differs by browser, so test a preference list as the example does. In Chrome 155 headless the probe returned true for audio/webm;codecs=opus, video/webm with vp8, vp9, av1 or h264, and video/mp4 with avc1, and false for audio/ogg;codecs=opus. Firefox records WebM and Ogg Opus, and Safari records MP4 (AAC and H.264). The specification marks isTypeSupported as deprecated, since it cannot say whether a profile or resolution is efficient; for video use navigator.mediaCapabilities.encodingInfo() when the choice matters.

Chunks are not files

With a timeslice, the concatenation of all chunks is a valid file, but a single chunk usually is not playable by itself: the first holds the container header and later ones are continuation data. The consequences for an uploader are that chunks must be stored in order, that a lost chunk corrupts the rest, and that playback of a chunk in isolation fails. Browsers also write WebM without a duration or a seek index, so a recording that is played from the file cannot be seeked; remux it once on the server (ffmpeg -i in.webm -c copy out.webm) to add them.

Voice notes with chunked upload

The Chatterbox voice note requests the microphone, records with a one-second timeslice, uploads every chunk with its sequence number as it arrives, and sends a final request to mark the note complete. Uploading while recording means a long note is not lost if the tab closes, and nothing large stays in memory.

// Chatterbox voice notes: record the microphone and upload each chunk while recording.
const CANDIDATES = [
  'audio/webm;codecs=opus', // Chrome, Edge, Firefox
  'audio/mp4;codecs=mp4a.40.2', // Safari (and recent Chrome)
  'audio/mp4',
];

export function pickAudioType() {
  return CANDIDATES.find((type) => MediaRecorder.isTypeSupported(type)) ?? '';
}

export async function recordVoiceNote({ uploadUrl, maxMs = 60_000, timeslice = 1000 }) {
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: { echoCancellation: true, noiseSuppression: true },
  });
  const mimeType = pickAudioType();
  const recorder = new MediaRecorder(stream, mimeType ? { mimeType, audioBitsPerSecond: 32_000 } : {});
  const noteId = crypto.randomUUID();

  let seq = 0;
  let uploads = Promise.resolve(); // chunks must reach the server in order
  const send = (body, query) => fetch(`${uploadUrl}/${noteId}?${query}`, {
    method: 'POST',
    headers: { 'Content-Type': recorder.mimeType },
    body,
  }).then((res) => { if (!res.ok) throw new Error(`upload failed: ${res.status}`); });

  recorder.addEventListener('dataavailable', (event) => {
    if (event.data.size === 0) return;
    const query = `seq=${seq++}`;
    uploads = uploads.then(() => send(event.data, query));
  });

  const stopped = new Promise((resolve, reject) => {
    recorder.addEventListener('error', (event) => reject(event.error));
    recorder.addEventListener('stop', resolve, { once: true });
  });

  recorder.start(timeslice);
  const timer = setTimeout(() => recorder.state === 'recording' && recorder.stop(), maxMs);

  return {
    noteId,
    mimeType: recorder.mimeType,
    async stop() {
      clearTimeout(timer);
      if (recorder.state !== 'inactive') recorder.stop();
      await stopped;
      stream.getTracks().forEach((track) => track.stop()); // release the microphone
      await uploads;
      await send(new Blob(), 'done=1');
      return { noteId, chunks: seq };
    },
  };
}

Source: MDN: MediaRecorder.start()

On the server any HTTP stack that appends the chunks of one note id in seq order will do. A minimal Node.js receiver used in the test stores each body in an array indexed by seq; after done=1 the array concatenated is the file. In the test (with a 500 ms timeslice) a 2.6 second note produced five chunks of 8,135 bytes in total, starting with the EBML magic bytes 1a45dfa3 of WebM. The endpoint must authenticate the user, cap the size of a note and of the number of chunks, and check that the content type is an audio type, because it accepts arbitrary bytes from a browser. See Authentication and Authorization and Security.

Pitfalls

  • Echo and feedback. Recording a stream that is also played through the speakers records the speakers. Mute the preview element and do not connect a recording graph to AudioContext.destination.

  • Formats at the receiving end. A note recorded as MP4 on Safari and WebM on Chrome must be playable by every client, so store the Content-Type and let the player use canPlayType, or transcode on the server.

  • Permission and consent. Recording other people is regulated; show a visible indicator and announce it through the signalling channel, as Huddle does with a recording room event.

  • Memory. Keeping all chunks of a long meeting in an array holds the whole recording in memory; upload and discard as in the voice note.

Composing with canvas captureStream

canvas.captureStream(frameRate) returns a MediaStream with one CanvasCaptureMediaStreamTrack that emits a frame when the canvas changes, at most frameRate times a second. With rate 0 no frames are produced until you call the track’s requestFrame(), which is the way to avoid capturing a half-drawn frame. A canvas that has drawn cross-origin media without permission is tainted and captureStream throws SecurityError; serve the media from your origin or with CORS.

That is enough to record a meeting grid, a screen share with an overlay, or a game. Audio does not come from the canvas: mix the audio sources with Web Audio, create a MediaStreamAudioDestinationNode, and add its track to the canvas stream before creating the recorder. The Huddle recorder does both.

// Huddle: record the visible grid of video tiles plus the mixed audio into one file.
export class MeetingRecorder {
  constructor({ width = 1280, height = 720, fps = 30 } = {}) {
    this.canvas = Object.assign(document.createElement('canvas'), { width, height });
    this.ctx = this.canvas.getContext('2d');
    this.fps = fps;
    this.videos = new Map(); // participant id -> <video>
    this.audio = new AudioContext();
    this.mix = this.audio.createMediaStreamDestination();
    this.sources = new Map();
    this.chunks = [];
    this.recorder = null;
    this.raf = 0;
  }

  addParticipant(id, stream) {
    const video = Object.assign(document.createElement('video'), { srcObject: stream, muted: true, playsInline: true });
    video.play();
    this.videos.set(id, video);
    if (stream.getAudioTracks().length) {
      const source = this.audio.createMediaStreamSource(stream);
      source.connect(this.mix); // not connected to audio.destination: no echo for the recorder
      this.sources.set(id, source);
    }
  }

  removeParticipant(id) {
    this.videos.delete(id);
    this.sources.get(id)?.disconnect();
    this.sources.delete(id);
  }

  draw = () => {
    const { ctx, canvas } = this;
    const n = Math.max(this.videos.size, 1);
    const cols = Math.ceil(Math.sqrt(n));
    const rows = Math.ceil(n / cols);
    const w = canvas.width / cols;
    const h = canvas.height / rows;
    ctx.fillStyle = '#101820';
    ctx.fillRect(0, 0, canvas.width, canvas.height);
    [...this.videos.values()].forEach((video, i) => {
      if (video.readyState >= 2) ctx.drawImage(video, (i % cols) * w, Math.floor(i / cols) * h, w, h);
    });
    this.raf = requestAnimationFrame(this.draw);
  };

  start() {
    const output = this.canvas.captureStream(this.fps);
    this.mix.stream.getAudioTracks().forEach((track) => output.addTrack(track));
    const mimeType = ['video/webm;codecs=vp9,opus', 'video/webm;codecs=vp8,opus', 'video/mp4']
      .find((t) => MediaRecorder.isTypeSupported(t));
    this.recorder = new MediaRecorder(output, { mimeType, videoBitsPerSecond: 2_500_000 });
    this.recorder.ondataavailable = (e) => e.data.size && this.chunks.push(e.data);
    this.audio.resume();
    this.draw();
    this.recorder.start(2000);
  }

  stop() {
    return new Promise((resolve) => {
      this.recorder.addEventListener('stop', () => {
        cancelAnimationFrame(this.raf);
        this.audio.close();
        resolve(new Blob(this.chunks, { type: this.recorder.mimeType }));
      }, { once: true });
      this.recorder.stop();
    });
  }
}

Source: MDN: HTMLCanvasElement.captureStream()

Two details matter in production. First, requestAnimationFrame stops in a hidden tab, so a recording that depends on it freezes when the host switches to another window; drive the drawing from a timer, or better from an OffscreenCanvas in a Worker, and warn the user. Second, the recorder only captures what the browser has decoded, so a tile that is not being received (for example because the SFU paused a layer) is a still frame in the recording. Mixing in the browser also means the recording exists only on the machine of the person who pressed the button, which is the reason for the next section’s alternative.

Encoded transforms in a Worker

An encoded transform sits between the encoder and the packetiser on the sender and between the depacketiser and the decoder on the receiver. Your code gets each encoded frame as an RTCEncodedVideoFrame or RTCEncodedAudioFrame, can change its data, and passes it on. The legacy Chrome API createEncodedStreams() was replaced by the standard RTCRtpScriptTransform, available since Safari 15.4, Firefox 117 and Chrome 141.

The pattern has two halves:

  1. On the main thread create a Worker and assign sender.transform = new RTCRtpScriptTransform(worker, options) (or receiver.transform). The options object is structured-cloned into the worker.

  2. In the worker handle the rtctransform event. event.transformer has a readable stream of encoded frames, a writable stream to put frames back, and the options. Pipe through a TransformStream, or read and write by hand. The streams have no backpressure.

Every frame that is read must be written back or the track stalls. The transformer also offers generateKeyFrame(rid) on the send side and sendKeyFrameRequest() on the receive side, plus a keyframerequest event fired when a PLI or FIR arrives. The same object type with the string "sframe" selects the browser’s built-in SFrame transform where available; the dedicated RTCRtpSFrameEncryptor and RTCRtpSFrameDecryptor classes of the draft were not present in Chrome 155 (the probe returned undefined), so feature-detect them. End-to-end encryption with SFrame (RFC 9605) is the main use, covered in End-to-End Encryption.

The example appends a four-byte sequence number to every encoded frame on the sender, and strips it on the receiver while counting gaps. It does not encrypt anything; it shows the plumbing, and is a useful probe for frame loss between the two ends. Each transformer keeps its own counters, because one worker serves the audio and the video sender.

// Runs in a dedicated Worker. Each sender or receiver gets its own transformer and its own counters.
const TRAILER = 4; // bytes appended to every encoded frame: a big-endian sequence number

onrtctransform = (event) => {
  const { readable, writable, options } = event.transformer;
  const state = { next: 0, expected: 0, lost: 0 }; // per track: audio and video count separately
  const transform = options.operation === 'send' ? tag : untag;
  readable
    .pipeThrough(new TransformStream({ transform: (frame, controller) => transform(frame, controller, state) }))
    .pipeTo(writable);
};

function tag(frame, controller, state) {
  const input = new Uint8Array(frame.data);
  const output = new Uint8Array(input.byteLength + TRAILER);
  output.set(input);
  new DataView(output.buffer).setUint32(input.byteLength, state.next++);
  frame.data = output.buffer;
  controller.enqueue(frame);
}

function untag(frame, controller, state) {
  if (frame.data.byteLength <= TRAILER) return; // drop a frame that cannot carry the trailer
  const body = frame.data.byteLength - TRAILER;
  const seq = new DataView(frame.data).getUint32(body);
  if (seq > state.expected) state.lost += seq - state.expected;
  state.expected = seq + 1;
  frame.data = frame.data.slice(0, body);
  controller.enqueue(frame);
  if (seq % 60 === 0) {
    const kind = frame instanceof RTCEncodedVideoFrame ? 'video' : 'audio';
    postMessage({ kind, seq, lost: state.lost });
  }
}
// Main thread: attach the transform to a sender and to a receiver.
export function attachFrameTag(pc, onStats) {
  const worker = new Worker('frame-tag-worker.js');
  worker.onmessage = (e) => onStats?.(e.data);
  const supported = typeof RTCRtpScriptTransform !== 'undefined';
  return {
    supported,
    sender(rtpSender) {
      if (supported) rtpSender.transform = new RTCRtpScriptTransform(worker, { operation: 'send' });
    },
    receiver(rtpReceiver) {
      if (supported) rtpReceiver.transform = new RTCRtpScriptTransform(worker, { operation: 'receive' });
    },
  };
}

Source: MDN: RTCRtpScriptTransform

On a loopback call between two peer connections in headless Chrome the receiver decoded 58 video frames in three seconds and reported zero lost frames, so the trailer was removed correctly before decoding. Pitfalls:

  • A trailer is part of the payload that the SFU forwards. Servers that parse the payload (to detect keyframes, for example) cannot read an encrypted or modified one; that is why E2EE keeps some header bytes in clear.

  • Adding bytes enlarges frames and can push packets over the MTU, so budget a few bytes only.

  • The transform must be set before the first frame flows, so assign it right after addTrack or addTransceiver, and assign it to receivers in ontrack.

  • Worker code that throws stops the track silently. Log errors in the worker and test with a call to a real peer.

Raw media: track processor and generator

Raw processing works on decoded media: a VideoFrame per camera frame, or an AudioData per audio block. The specification (Media Capture Transform) defines two classes in a dedicated worker: MediaStreamTrackProcessor, which turns a track into a ReadableStream of frames, and VideoTrackGenerator, which exposes a WritableStream and a track whose content is whatever you write. Pipe the first through your filter into the second and you have a new track to send or record. MediaStreamTrackHandle lets a worker obtain a track from the main thread.

Browser status is uneven, and a page must feature-detect:

API Chrome Firefox Safari

MediaStreamTrackProcessor

94, on the main thread only (not exposed in workers in 155)

Not supported

18, in workers

MediaStreamTrackGenerator

94, non-standard, main thread

Not supported

Not supported

VideoTrackGenerator

Not in Chrome 155

Not supported

18, in workers

RTCRtpScriptTransform

141

117

15.4

The table follows the MDN browser-compatibility data (version 8.1.7) and the probe of headless Chrome 155. The practical recipe for Chrome is therefore to create the processor and the generator on the main thread, which the specification’s earlier proposal exposes, and transfer their streams to a Worker that does the pixel work. The streams are transferable, so the frames never touch the main thread. A page that only needs Safari can create everything inside the worker. Where neither exists, return the original track, or fall back to drawing the video on a canvas and capturing it, which costs main thread time.

The background blur below blurs the whole frame and then repaints a sharp ellipse in the middle. The ellipse stands in for the mask a segmentation model (for example MediaPipe Image Segmenter) would produce; replace the clipping with that mask and the rest of the pipeline stays the same. Two rules cannot be skipped: close every input VideoFrame after use, or the camera pipeline runs out of buffers and the video freezes, and always create the output frame with the input’s timestamp.

// Worker: receives a readable and a writable stream of VideoFrame and blurs the frame edges.
onmessage = ({ data: { readable, writable, radius = 8 } }) => {
  let canvas;
  let ctx;
  const transformer = new TransformStream({
    transform(frame, controller) {
      if (!canvas || canvas.width !== frame.displayWidth || canvas.height !== frame.displayHeight) {
        canvas = new OffscreenCanvas(frame.displayWidth, frame.displayHeight);
        ctx = canvas.getContext('2d');
      }
      const { width: w, height: h } = canvas;
      ctx.filter = `blur(${radius}px)`;
      ctx.drawImage(frame, 0, 0, w, h); // blurred background
      ctx.filter = 'none';
      ctx.save(); // sharp "person" area: an ellipse stands in for a segmentation mask
      ctx.beginPath();
      ctx.ellipse(w / 2, h / 2, w * 0.25, h * 0.4, 0, 0, Math.PI * 2);
      ctx.clip();
      ctx.drawImage(frame, 0, 0, w, h);
      ctx.restore();
      const out = new VideoFrame(canvas, { timestamp: frame.timestamp, duration: frame.duration ?? undefined });
      frame.close(); // always close input frames, or the camera stalls
      controller.enqueue(out);
    },
  });
  readable.pipeThrough(transformer).pipeTo(writable);
};
// Main thread: turn a camera track into a blurred track. Returns the original track when unsupported.
export function blurTrack(track, radius = 8) {
  const worker = new Worker('blur-worker.js');
  if (typeof MediaStreamTrackProcessor === 'undefined' || typeof MediaStreamTrackGenerator === 'undefined') {
    return { track, stop() { worker.terminate(); }, supported: false };
  }
  // Chromium exposes both classes on the main thread; the streams are transferable to the worker.
  const processor = new MediaStreamTrackProcessor({ track });
  const generator = new MediaStreamTrackGenerator({ kind: 'video' });
  worker.postMessage({ readable: processor.readable, writable: generator.writable, radius },
    [processor.readable, generator.writable]);
  return { track: generator, supported: true, stop() { generator.stop(); worker.terminate(); } };
}

Source: MDN: MediaStreamTrackProcessor

In the test the generated track played in a video element at 640 pixels wide. A processed track is an ordinary track, so it can be given to RTCRtpSender.replaceTrack(). Processing at 30 frames a second costs CPU and battery, so Huddle offers blur as an option, lowers the capture resolution first, and turns it off when getStats() reports qualityLimitationReason of cpu (see Stats, Quality and Bandwidth). Remember to call stop() on the generator when the filter is removed.

Audio levels with Web Audio

A speaking indicator and a microphone meter need the loudness of an audio track, not the audio itself. Create an AudioContext, wrap the stream with createMediaStreamSource, connect it to an AnalyserNode, and sample the time-domain data. Do not connect the analyser to the context’s destination, or the user hears themselves. The root-mean-square of the samples is a good level; a threshold turns it into "speaking". For remote participants the same code runs on the received stream, or use the audio-level RTP header extension (RFC 6464) that SFUs already forward, which costs nothing in the browser.

// Speaking indicator: root-mean-square level of an audio track, 0..1, sampled every 100 ms.
export function watchLevel(stream, onLevel, { threshold = 0.02 } = {}) {
  const ctx = new AudioContext();
  const analyser = ctx.createAnalyser();
  analyser.fftSize = 1024;
  ctx.createMediaStreamSource(stream).connect(analyser); // do not connect to ctx.destination
  const samples = new Float32Array(analyser.fftSize);
  const timer = setInterval(() => {
    analyser.getFloatTimeDomainData(samples);
    let sum = 0;
    for (const s of samples) sum += s * s;
    const level = Math.sqrt(sum / samples.length);
    onLevel({ level, speaking: level > threshold });
  }, 100);
  return () => { clearInterval(timer); ctx.close(); };
}

Source: MDN: AnalyserNode

The fake microphone in headless Chrome produced a maximum level of about 0.021 over 1.5 seconds, above the 0.02 threshold. Real voices vary, so calibrate the threshold or smooth the level. An AudioContext created without a user gesture starts suspended; call resume() from a click handler. Close the context when the participant leaves, because browsers limit the number of live contexts.

Server-side recording through the SFU

Recording in the browser records one person’s view, depends on that person’s laptop, battery and tab, and cannot capture participants who left before. Production systems record on the server, where the SFU already receives every stream:

  • An SFU such as mediasoup exposes plain RTP: a PlainTransport plus a consumer sends a participant’s RTP to a local port, and ffmpeg or GStreamer reads it through an SDP file and writes the file. See mediasoup.

  • LiveKit has Egress as a separate service: a room composite egress records the whole room, laid out by a headless Chrome page, and a track egress writes each track. It outputs MP4, HLS or a live RTMP stream. See LiveKit.

  • Jitsi’s recorder, Jibri, joins the conference as a hidden participant in a real browser and records the screen. See Janus and Jitsi.

The choice is between quality of layout and cost: a composite recording costs a browser instance per meeting, separate tracks are cheap but must be composed afterwards. Whichever is used, the server has no encoded-media decryption key in an end-to-end encrypted meeting, so recording and E2EE exclude each other unless the recorder joins as a participant that holds the key. Announce the recording to all participants, and store it under the retention rules of your jurisdiction. For broadcasting the output instead of storing it, see WHIP, WHEP and Live Streaming.

Common errors

Symptom Cause and fix

NotSupportedError from the constructor

The mimeType is not supported; test with isTypeSupported and fall back to the default.

InvalidStateError from start()

The recorder is already recording or paused; create a new recorder per take.

Last chunk missing

The upload ran before the final dataavailable; wait for the stop event, then the uploads, as the voice note does.

SecurityError from captureStream

The canvas is tainted by cross-origin media.

Recording freezes when switching tabs

requestAnimationFrame is paused in hidden tabs.

Video freezes after a few seconds in a processor

An input VideoFrame was not closed.

Remote track stalls after setting a transform

A frame was read but not written back, or the worker threw.

RTCRtpScriptTransform is undefined

An engine older than Chrome 141, Firefox 117 or Safari 15.4; feature-detect and send without the transform.

Related pages

References