Realtime APIs

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against OpenAI GPT-Live (gpt-live-1) and the Realtime API (gpt-realtime, gpt-realtime-1.5, gpt-realtime-2 and gpt-realtime-2.1) with the openai Python package 3.22.1, the Gemini Live API (gemini-3.8-live, gemini-3.1-flash-live-preview, gemini-2.5-flash-native-audio-preview-12-2025) with google-genai 2.26.0, Azure Voice Live, FastAPI 0.142.2 on Python 3.12, and Spring AI 2.0.1 on Spring Boot 4.1.x with Java 21 (see the version baseline). The request and event shapes below were read from the vendors' documentation on that date. No call was made to any of these services from this repository: they need paid accounts, a microphone and a browser, so everything here is documented from the official pages linked after each block. Model names, event names and preview status change often; check the models page and the changelog of each provider before copying an identifier.

A realtime API takes the caller’s audio and returns spoken audio from one model, which removes two network hops from the cascaded design described on Voice Architectures. The retrieval-augmented DocsAssistant core cannot sit between "STT" and "TTS" because there is no text boundary to hook; it is reached either by the model deciding to call a tool, or, with GPT-Live, by delegation to a backend.

flowchart LR C[Caller: browser or phone] <-->|audio and events| M["Realtime model (OpenAI, Gemini, Azure)"] M -- "function tool, remote MCP, or delegation" --> B["Your backend"] B --> CORE["DocsAssistant core: ask(conversationId, userId, text)"] CORE --> R[(Retriever, memory, ACL filter)] CORE -- "short text answer with citations as titles" --> B B -- "tool result or commentary" --> M

Two OpenAI voice APIs

OpenAI documents two APIs for spoken conversations and a guide to choosing between them and a chained pipeline (see OpenAI: Voice agents):

GPT-Live Realtime API

	GPT-Live	Realtime API
Model	`gpt-live-1`	`gpt-realtime`, `gpt-realtime-1.5`, `gpt-realtime-2`, `gpt-realtime-2.1` (the models page tells which is current)
Division of work	The live model runs the conversation (listening, speaking, full duplex, deciding when to ask for help). A backend does the reasoning and the tool work; sending work to it is called delegation.	One model hears, reasons, calls tools and speaks. You configure instructions, tools and turn detection on the session.
Start a session	`POST /v1/live/sessions` with a `session` object and a `transport` (for WebRTC, the SDP offer).	`POST /v1/realtime/calls` (the server combines the browser’s SDP offer with the session configuration), or mint an ephemeral client secret with `POST /v1/realtime/client_secrets` and let the browser connect itself.
Connections	WebRTC, WebSocket, telephony and SIP, plus a server-side "sideband" connection to watch or control a session.	WebRTC, WebSocket, SIP, plus a server-side controls connection.
How the DocsAssistant core is reached	Client delegation: the application receives a delegation event and answers it. Or Responses delegation, where OpenAI calls a Responses model with your tools.	A function tool the model calls, or a remote MCP server.

Model

gpt-live-1

gpt-realtime, gpt-realtime-1.5, gpt-realtime-2, gpt-realtime-2.1 (the models page tells which is current)

Division of work

The live model runs the conversation (listening, speaking, full duplex, deciding when to ask for help). A backend does the reasoning and the tool work; sending work to it is called delegation.

One model hears, reasons, calls tools and speaks. You configure instructions, tools and turn detection on the session.

Start a session

POST /v1/live/sessions with a session object and a transport (for WebRTC, the SDP offer).

POST /v1/realtime/calls (the server combines the browser’s SDP offer with the session configuration), or mint an ephemeral client secret with POST /v1/realtime/client_secrets and let the browser connect itself.

Connections

WebRTC, WebSocket, telephony and SIP, plus a server-side "sideband" connection to watch or control a session.

WebRTC, WebSocket, SIP, plus a server-side controls connection.

How the DocsAssistant core is reached

Client delegation: the application receives a delegation event and answers it. Or Responses delegation, where OpenAI calls a Responses model with your tools.

A function tool the model calls, or a remote MCP server.

For the DocsAssistant the GPT-Live client delegation fits best when you want OpenAI’s conversational layer and your own answers (the hybrid architecture), and the Realtime API with a search_docs function tool or the MCP server fits when you prefer the single-model design and need the established tool-call loop. Both are billed by duration or audio tokens; quote no prices, and read the pricing and cost-optimisation pages (see References).

The ephemeral secret flow (Realtime API, WebRTC)

A browser must never hold your API key. For the Realtime API there are two documented ways to connect a browser over WebRTC: the unified interface, where the browser posts its SDP offer to your server and your server forwards it to /v1/realtime/calls; and ephemeral keys, where your server mints a short-lived client secret and the browser connects to OpenAI directly. The second removes your server from the media path.

sequenceDiagram participant B as Browser participant S as Your server (FastAPI or Spring) participant O as OpenAI Realtime API B->>S: GET /token (user is authenticated) S->>O: POST /v1/realtime/client_secrets (API key, session config) O-->>S: ephemeral client secret S-->>B: client secret value B->>O: SDP offer with client secret over HTTPS O-->>B: SDP answer B<<->>O: audio tracks and the oai-events data channel O-->>B: tool call event (search_docs) B->>S: POST /api/docs/search (user token) S-->>B: passages B->>O: function_call_output, response.create

Because the model runs the tool through the browser here, the browser calls your retrieval endpoint with the user’s own credentials, and the access-control filter runs on your server. That keeps the ACL decision where it was in the text channel; see Access Control and Privacy.

token_endpoint.py: minting a Realtime client secret with FastAPI

import os

import httpx
from fastapi import Depends, FastAPI

from docsassistant.security import current_user, User  # your existing authentication dependency

app = FastAPI()

SESSION = {
    "session": {
        "type": "realtime",
        "model": "gpt-realtime-2.1",            # check the models page for the current identifier
        "audio": {"output": {"voice": "marin"}},
        "instructions": (
            "You are DocsAssistant, a voice assistant for the documentation site. Answer in one or two short "
            "sentences. Call search_docs before answering any factual question and say the document title, not a URL."
        ),
        "tools": [
            {
                "type": "function",
                "name": "search_docs",
                "description": "Search the documentation and return the most relevant short passages.",
                "parameters": {
                    "type": "object",
                    "properties": {"query": {"type": "string"}},
                    "required": ["query"],
                },
            }
        ],
    }
}


@app.get("/token")
async def token(user: User = Depends(current_user)) -> dict:
    async with httpx.AsyncClient(timeout=10) as http:
        response = await http.post(
            "https://api.openai.com/v1/realtime/client_secrets",
            headers={
                "Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}",
                "OpenAI-Safety-Identifier": user.hashed_id,   # stable, privacy-preserving end-user id
            },
            json=SESSION,
        )
    response.raise_for_status()
    return response.json()      # the browser reads "value" and uses it once to connect

Source: OpenAI: WebRTC (the page shows the token request and the browser connection) and OpenAI: Realtime client secrets reference. current_user stands for the authentication dependency that the Serving RAG as an API and Python FastAPI integration pages already build.

The Spring Boot equivalent uses the same endpoint through RestClient. The model identifier and session shape are configuration, not code.

RealtimeTokenController.java: minting a Realtime client secret with Spring Boot

import java.util.List;
import java.util.Map;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.MediaType;
import org.springframework.security.core.annotation.AuthenticationPrincipal;
import org.springframework.security.oauth2.jwt.Jwt;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestClient;

@RestController
class RealtimeTokenController {

    private final RestClient openAi;

    RealtimeTokenController(RestClient.Builder builder, @Value("${openai.api-key}") String apiKey) {
        this.openAi = builder.baseUrl("https://api.openai.com/v1")
                .defaultHeader("Authorization", "Bearer " + apiKey)
                .build();
    }

    @GetMapping(value = "/token", produces = MediaType.APPLICATION_JSON_VALUE)
    Map<?, ?> token(@AuthenticationPrincipal Jwt user) {
        Map<String, Object> session = Map.of("session", Map.of(
                "type", "realtime",
                "model", "gpt-realtime-2.1",
                "audio", Map.of("output", Map.of("voice", "marin")),
                "tools", List.of(Map.of(
                        "type", "function",
                        "name", "search_docs",
                        "description", "Search the documentation and return short passages.",
                        "parameters", Map.of(
                                "type", "object",
                                "properties", Map.of("query", Map.of("type", "string")),
                                "required", List.of("query"))))));
        return this.openAi.post().uri("/realtime/client_secrets")
                .header("OpenAI-Safety-Identifier", Integer.toHexString(user.getSubject().hashCode()))
                .body(session)
                .retrieve()
                .body(Map.class);
    }
}

Source: OpenAI: WebRTC for the request and Spring Framework: REST Clients for RestClient. Use a real keyed hash for the safety identifier in production, not hashCode(); the line above only keeps the example short. Spring AI 2.0.1 has no speech-to-speech abstraction, so the Realtime API is called directly; its TranscriptionModel and TextToSpeechModel serve the cascaded design instead (see Audio Transcription & Speech).

Session configuration, tools and truncation

The first thing the client sends over the data channel (or WebSocket) after the connection opens is a session.update, unless your server already fixed the configuration when it minted the secret, which is better because the instructions and tools then never reach the browser. A session carries:

instructions, the persona and the rules for using tools (keep them short for voice);
audio.output.voice and the turn-detection settings (server VAD or semantic detection; see Turn-Taking and Interruptions);
tools, which are either function tools that your client executes, or mcp tools that the Realtime API executes against a remote MCP server.

For a function tool, the model emits a function-call item with the arguments; your code runs it, adds a function_call_output conversation item with the result, and sends response.create so the model speaks. For the DocsAssistant, the result is the two or three top passages with their titles, short enough to read aloud.

When the caller interrupts, the server stops the audio, but the conversation still contains the full planned answer. The client reports how much was actually played by sending a conversation.item.truncate event with the item ID and the audio end offset in milliseconds, which deletes the unheard text and its transcript so that the next turn is consistent. This is the history truncation of the turn-taking page, delegated to the API.

Remote MCP to the DocsAssistant server

Instead of writing the function tool, point the session at the MCP server built on Exposing RAG through MCP. The Realtime API calls the server itself, so the browser never sees the tool traffic.

session-update.json: a remote MCP tool on a Realtime session

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "model": "gpt-realtime-2.1",
    "tools": [
      {
        "type": "mcp",
        "server_label": "docsassistant",
        "server_url": "https://mcp.example.com/mcp",
        "allowed_tools": ["search_docs", "retrieve_passage"],
        "require_approval": "never",
        "authorization": "<short-lived token for this user>"
      }
    ]
  }
}

Source: OpenAI: Realtime with tools. Names such as mcp.example.com, search_docs and retrieve_passage are placeholders for your own deployment. A remote MCP server must be reachable from OpenAI’s network over HTTPS and must authenticate each caller, as described in MCP authorization and Deploying remote servers; pass the end user’s own short-lived token so that the access-control filter sees the right identity. Use allowed_tools to expose only the read-only retrieval tools to the voice model.

SIP and the server WebSocket

For telephone calls OpenAI accepts SIP: a call arriving on a SIP trunk pointed at OpenAI triggers a webhook to your server, which accepts the call with a session configuration and can then monitor it over a WebSocket. See the Telephony and SIP guide linked in References and Telephony with Asterisk and SIP. A server-side WebSocket (wss://api.openai.com/v1/realtime?model=…) with your API key is the right connection for a backend that bridges audio from a telephony or PBX system; never use the key in the browser.

GPT-Live in practice

A GPT-Live WebRTC session follows the sequence documented by OpenAI: the browser creates a peer connection with the microphone track and a data channel named oai-events before the SDP offer, sends the offer to your server, and your server creates the session with POST /v1/live/sessions containing the session configuration and transport: { type: "webrtc", sdp: … }. The HTTP request itself starts the session; the client must not send a session.start event, and it waits for session.started on the data channel before sending application commands. Creating a WebRTC session bills 15 seconds of voice duration during initialisation, credited against the running charges.

The session configuration chooses the delegation mode. With client delegation your application is the backend:

live_session.py: creating a GPT-Live session with client delegation (FastAPI)

from fastapi import Depends, FastAPI
from openai import AsyncOpenAI
from pydantic import BaseModel

from docsassistant.security import current_user, User

app = FastAPI()
client = AsyncOpenAI(max_retries=0)   # async client: the call below does not block the event loop


class Offer(BaseModel):
    sdp: str


@app.post("/api/session", status_code=201)
async def create_session(offer: Offer, user: User = Depends(current_user)) -> dict:
    result = await client.live.create(
        session={
            "model": "gpt-live-1",
            "instructions": (
                "You are the voice of DocsAssistant. Keep replies short. For any question about the "
                "documentation, ask the backend."
            ),
            "delegation": {"type": "client"},
        },
        transport={"type": "webrtc", "sdp": offer.sdp},
    )
    return result.model_dump()      # session id and SDP answer, returned unchanged

Source: OpenAI: WebRTC (GPT-Live) and OpenAI: Delegation and tools. The OpenAI example is synchronous and local-only. The endpoint above adds the authentication dependency its authors tell you to add, and it uses AsyncOpenAI. A synchronous SDK call inside an async handler would hold the event loop for the whole round trip and stall every other request and WebSocket on the server.

When the live model decides it needs help, the data channel carries a session.delegation.created event whose delegation object has an id and a target of client. It does not contain the question: your application collects session.input_transcript.delta and session.output_transcript.delta events, which carry delta, start_ms and end_ms, and builds the question for the backend from that history. It then calls the DocsAssistant core and returns the answer with the delegation ID, which GPT-Live paraphrases aloud:

delegation_handler.py: answering a client delegation from the DocsAssistant core

async def on_delegation(connection, delegation_id: str, transcript: list[str], thread_id: str) -> None:
    question = " ".join(transcript[-6:])                 # rebuild the user's request from transcript fragments
    answer = await docsassistant.ask(thread_id, question)  # the same core as the text channel, collected to a string
    await connection.session.commentary.append(
        event_id=f"result-{delegation_id}",
        delegation_id=delegation_id,
        content=answer.short_spoken_text,                 # titles instead of URLs, no markdown
    )

Source: OpenAI: Delegation and tools (client delegation and the session.commentary.append update). connection is a server-side connection to the session, documented under Server-side controls; docsassistant.ask is the ask(conversationId, userId, text) operation of the core. OpenAI’s guidance is to keep the business rules and tool workflow in the backend and give the live model only the facts to say; use session.thinking.append for facts it should know but not read out. Because the backend work continues if the caller interrupts, the application decides whether to cancel it.

The other mode, Responses delegation, has OpenAI run a Responses model with the tools you list (for example hosted web search, as in OpenAI’s quickstart) and pass the context and results between it and GPT-Live. It needs no backend code but the DocsAssistant retrieval has to be exposed as a function or MCP tool of that Responses model, and the choice is made at session creation: to change it, start a new session.

Gemini Live

The Gemini Live API is a stateful WebSocket API for audio, video and text with native audio output. The models page lists gemini-3.8-live as the stable model, with gemini-3.1-flash-live-preview and gemini-2.5-flash-native-audio-preview-12-2025 as other options. Audio goes in as raw 16-bit PCM at 16 kHz (audio/pcm;rate=16000) and comes back as chunks of PCM. A server-to-server session in Python:

gemini_live.py: a Gemini Live session with a search tool

import asyncio

from google import genai
from google.genai import types

client = genai.Client()   # reads GEMINI_API_KEY

search_docs = types.FunctionDeclaration(
    name="search_docs",
    description="Search the documentation and return short passages.",
    parameters={"type": "OBJECT", "properties": {"query": {"type": "STRING"}}, "required": ["query"]},
)

config = types.LiveConnectConfig(
    response_modalities=["AUDIO"],
    system_instruction="You are DocsAssistant. Answer in one or two short sentences and call search_docs first.",
    tools=[types.Tool(function_declarations=[search_docs])],
)


async def run(mic_chunks, speaker):
    async with client.aio.live.connect(model="gemini-3.8-live", config=config) as session:
        async def send():
            async for chunk in mic_chunks:
                await session.send_realtime_input(
                    audio=types.Blob(data=chunk, mime_type="audio/pcm;rate=16000"))

        async def receive():
            async for message in session.receive():
                if message.tool_call:
                    responses = [
                        types.FunctionResponse(
                            id=call.id, name=call.name,
                            response={"result": await docsassistant.search(call.args["query"])})
                        for call in message.tool_call.function_calls
                    ]
                    await session.send_tool_response(function_responses=responses)
                elif message.server_content and message.server_content.model_turn:
                    for part in message.server_content.model_turn.parts:
                        if part.inline_data:
                            await speaker(part.inline_data.data)

        await asyncio.gather(send(), receive())

Source: Google: Gemini Live API and its linked pages for the SDK getting started guide and tool use (Live API tool use). For a browser that talks to Gemini directly, the backend mints an ephemeral token with client.auth_tokens.create; by default the client has one minute to start a session and thirty minutes of life, uses limits how often the token may start a session, and a token can be locked to a model and configuration with live_connect_constraints so that system instructions stay on the server. Ephemeral tokens are marked Preview, work only with the Live API and the v1beta version, and are described at Google: Ephemeral tokens. Sessions are long-lived, and session resumption is the documented way to reconnect after the connection ages out.

Azure Voice Live

The Azure Voice Live API combines speech recognition, a generative model and speech synthesis behind one WebSocket interface that is designed to be compatible with the Azure OpenAI Realtime API events, with additions that are optional and additive: noise suppression, echo cancellation, interruption detection, advanced end-of-turn detection, over 140 locales for recognition, standard and custom voices, avatars, and function calling. You choose the generative model (GPT-Realtime, GPT-5, GPT-4.1, Phi and others, subject to region). It is the realtime choice when the rest of your stack is on Azure, you need data residency in Azure regions, or you want a specific neural or custom voice. Retrieval follows the VoiceRAG pattern the documentation names: a function tool that calls your search, exactly as in the Realtime examples above. Authentication and the endpoint are tied to Azure AI Foundry resources, so read Microsoft Learn: Voice Live API and its how-to for the current request shape. For the Azure side of a data and AI estate, see Azure AI, data and IoT overview; Voice Live’s tiers, SDKs and a function-calling example are on Foundry Tools: Voice Live.

Comparison

OpenAI GPT-Live OpenAI Realtime Gemini Live Azure Voice Live

	OpenAI GPT-Live	OpenAI Realtime	Gemini Live	Azure Voice Live
Model identifiers (2026-10-01)	`gpt-live-1`	`gpt-realtime`, `-1.5`, `-2`, `-2.1`	`gemini-3.8-live` and two others	Choice of GPT-Realtime, GPT-5, GPT-4.1, Phi and more
Browser transport	WebRTC (server posts offer to `/v1/live/sessions`)	WebRTC (unified interface or ephemeral secret), WebSocket	WebSocket with an ephemeral token	WebSocket from a backend
Telephony	Telephony and SIP, partner integrations (LiveKit, Twilio, Telnyx, Daily/Pipecat)	SIP, plus partner integrations	Through a framework or your own bridge	Through your own bridge or Azure Communication Services
Reaching the RAG core	Delegation (client or Responses)	Function tool or remote MCP	Function declarations	Function tools (VoiceRAG)
Barge-in	Full duplex, handled by the model	Server VAD, `conversation.item.truncate`	Activity detection, interruption in the session	Built-in interruption detection
Data residency control	Per OpenAI project settings	Per OpenAI project settings	Per Google Cloud and API terms	Azure regions

Model identifiers (2026-10-01)

gpt-live-1

gpt-realtime, -1.5, -2, -2.1

gemini-3.8-live and two others

Choice of GPT-Realtime, GPT-5, GPT-4.1, Phi and more

Browser transport

WebRTC (server posts offer to /v1/live/sessions)

WebRTC (unified interface or ephemeral secret), WebSocket

WebSocket with an ephemeral token

WebSocket from a backend

Telephony

Telephony and SIP, partner integrations (LiveKit, Twilio, Telnyx, Daily/Pipecat)

SIP, plus partner integrations

Through a framework or your own bridge

Through your own bridge or Azure Communication Services

Reaching the RAG core

Delegation (client or Responses)

Function tool or remote MCP

Function declarations

Function tools (VoiceRAG)

Barge-in

Full duplex, handled by the model

Server VAD, conversation.item.truncate

Activity detection, interruption in the session

Built-in interruption detection

Data residency control

Per OpenAI project settings

Per Google Cloud and API terms

Azure regions

None of the four lets you run the model on your own hardware. When that is a requirement, use the cascaded design with Pipecat and local models, or the on-device options on the speech models page.

Security and cost checklist

Keep the API key on the server; hand browsers only ephemeral secrets or tokens, minted after you authenticate the user, with the shortest lifetime and a single use where the API allows it.
Authenticate and rate-limit the token or session endpoint; each call starts billable audio.
Pass the end user’s identity to retrieval (tool endpoint or MCP token) so access-control filters apply.
Cap session duration and idle time on your side; a forgotten tab is a billable session.
Log transcripts according to your retention policy and redact personal data before storage (see Production Voice Agents).
Treat tool results as untrusted text that the model will read aloud: strip instructions from retrieved passages (see Guardrails and Prompt Injection).