Cost, Latency & Throughput

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Examples on this page target Python 3.12 with the current anthropic Python SDK against model claude-sonnet-5 ($2 / MTok input, $10 / MTok output) and the current openai Python SDK against gpt-6-sol ($2 / MTok input, $10 / MTok output), plus Ollama 0.34 (Python ollama client 0.6.x) serving llama3.1:8b locally. Prices, endpoints and streaming shapes were checked against Anthropic’s pricing page, OpenAI’s pricing page and Ollama’s API docs as of 2026-09-28.

TTFT, TPOT and throughput

A single request’s latency, as experienced by a caller, is not one number but two, because a decoder-only model produces its response one token at a time rather than all at once:

  • Time to first token (TTFT) — the delay between sending a request and receiving the first token of the response. It is dominated by how long the model takes to process the entire input (the prompt, system prompt, tool definitions and prior turns) before it can emit anything, so TTFT grows with input length and with how busy the serving infrastructure is, largely independently of how long the response will be.

  • Time per output token (TPOT) — the average delay between one output token and the next, once generation has started. It is set mainly by the model’s size and the serving hardware’s raw decode throughput, and it stays roughly constant across a single response’s output tokens.

  • Throughput — how many output tokens per second a request (or a batch of concurrent requests) sustains once generation is under way; it is approximately the reciprocal of TPOT for a single stream, \(1 / \text{TPOT}\), and is the number that matters most for a long generated response, while TTFT dominates the perceived responsiveness of a short one.

Putting TTFT and TPOT together gives the total latency of a request that generates output tokens. The first token arrives at TTFT and each of the remaining tokens adds one TPOT, so

where the approximation drops a single TPOT and is accurate for any response longer than a few tokens.

This is why a short, chatty exchange (small ) is dominated by TTFT — cutting prompt size or using a faster model to first-token matters most — while a long generated report or an agent’s multi-step plan is dominated by the term, where the per-token rate matters more than the initial delay. The timeline below sketches where each quantity is measured on a single streamed response:

sequenceDiagram participant Client participant Server Client->>Server: request (prompt, n_out tokens requested) Note over Server: processes the whole prompt Server-->>Client: first token (TTFT elapsed) loop n_out - 1 more tokens Server-->>Client: next token (TPOT elapsed since previous) end Note over Client: latency = TTFT + TPOT x (n_out - 1), approx. TTFT + TPOT x n_out

TTFT and TPOT are not published constants — they vary by model, prompt length, current load on the provider’s infrastructure and, for a self-hosted model, the serving hardware and batching policy — so the only reliable way to know either number for a given workload is to measure it directly against a representative prompt, as the two examples below do.

Token pricing and batch APIs

Tokens & Tokenization already introduces the per-request cost formula, , for input tokens at price and output tokens at price ; this page reuses that formula rather than re-deriving it. What changes from one call to the next is which per-token prices apply, because a provider typically publishes several tiers for the same model:

Standard (synchronous) Batch API

claude-sonnet-5 (Anthropic)

$2 / MTok in, $10 / MTok out

$1 / MTok in, $5 / MTok out (50% off)

gpt-6-sol (OpenAI)

$2 / MTok in, $10 / MTok out

50% off both, per OpenAI’s Batch API discount

A batch API (Anthropic’s and OpenAI’s both work this way) accepts a file of many independent requests, runs them asynchronously — typically completing within 24 hours rather than synchronously — and returns each result once ready, at roughly half the price of the same request made through the synchronous endpoint. It trades latency (a batch request has no meaningful TTFT of its own; the caller polls or is notified once the whole batch finishes) for a lower per-token price, which makes it a good fit for large offline jobs — re-summarising a document corpus, running an evaluation suite, classifying a backlog — where no individual result is needed within seconds. See Anthropic — Batch processing and OpenAI — Batch API.

The cache_creation_input_tokens / cache_read_input_tokens fields and the cached-price tiers used by prompt caching (next) are a third pricing dimension layered on top of the standard/batch split above, not a replacement for it — both discounts can apply to the same request.

Prompt caching economics

Reusing a long, unchanged prefix (a system prompt, a reference document, tool definitions) across many requests is itself a lever on both cost and TTFT, since a cached prefix is billed at a fraction of the base input price and does not need to be reprocessed from scratch. This page does not restate that mechanism or its worked cost example — see Context Windows & Memory, which covers Anthropic’s explicit cache_control breakpoints, OpenAI’s automatic caching and a dated 10-turn cost comparison in depth.

Choosing a smaller model or a lower reasoning-effort level

Beyond caching and batching, the two remaining levers on cost and latency both amount to asking the model to do less work per request:

  • A smaller model in the same family — Anthropic’s Haiku tier or OpenAI’s gpt-6-luna-class models price their input and output tokens at a fraction of their larger siblings' rates and typically post a lower TPOT as well, at the cost of weaker performance on harder tasks. Routing easy requests to a small model and only escalating the ones it cannot handle to a larger one is a common way to keep average cost and latency low without giving up quality on the requests that need it. Choosing a Model covers this routing and cascade pattern, with a worked cascade router, alongside the selection workflow end to end.

  • A lower reasoning-effort level — a reasoning model that is allowed to think for longer produces more hidden reasoning tokens before its visible answer, which are billed as output tokens and add directly to both cost (via ) and latency (via ) even though the caller never sees them. Turning the effort level down trades some accuracy on hard problems for a cheaper, faster response on the easy majority of requests. See Reasoning Models for how Anthropic’s extended thinking and OpenAI’s reasoning_effort parameter each expose this trade-off.

A dated Python cost calculator

The following calculator applies the cost formula above to a token count and a small table of per-MTok prices. The prices are the ones quoted at the top of this page, checked as of 2026-09-28 — they are a snapshot, not a timeless fact, and must be replaced with whatever the provider’s pricing page currently lists before this calculator is used for a real budget:

# Prices as of 2026-09-28; replace with current prices from the provider's own
# pricing page before relying on this for a real budget.
PRICES_PER_MTOK = {
    "claude-sonnet-5": {"input": 2.00, "output": 10.00},
    "gpt-6-sol": {"input": 2.00, "output": 10.00},
}


def request_cost(model: str, input_tokens: int, output_tokens: int) -> float:
    """cost = n_in * p_in + n_out * p_out, prices given per million tokens (MTok)."""
    prices = PRICES_PER_MTOK[model]
    return (input_tokens * prices["input"] + output_tokens * prices["output"]) / 1_000_000


print(request_cost("claude-sonnet-5", input_tokens=6_000, output_tokens=300))
print(request_cost("gpt-6-sol", input_tokens=6_000, output_tokens=300))

See Anthropic — Pricing and OpenAI — Pricing for the tables this calculator’s prices were copied from.

Measuring TTFT and TPOT against a hosted streaming API

TTFT and TPOT are measured, not looked up, by timing a streamed response: record when the request was sent, record when the first output token arrives, and record the arrival time of every later token to average the gaps between them:

import time

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

start = time.perf_counter()
first_token_at = None
token_times = []

with client.messages.stream(
    model="claude-sonnet-5",
    max_tokens=300,
    messages=[{"role": "user", "content": "Explain TTFT and TPOT in three sentences."}],
) as stream:
    for event in stream:
        if event.type == "content_block_delta" and event.delta.type == "text_delta":
            now = time.perf_counter()
            if first_token_at is None:
                first_token_at = now
            token_times.append(now)

ttft = first_token_at - start
gaps = [b - a for a, b in zip(token_times, token_times[1:])]
tpot = sum(gaps) / len(gaps) if gaps else 0.0
print(f"TTFT: {ttft:.3f}s, TPOT: {tpot:.4f}s/token, tokens: {len(token_times)}")

Each content_block_delta event does not correspond exactly to one model token (a delta can bundle several sub-word pieces), so tpot above is an average delay per streamed chunk rather than a per-token figure to the last decimal place — close enough to compare prompts, models or providers against each other. See the official Anthropic — Streaming Messages guide for the full set of streamed event types this loop could branch on.

Measuring TTFT and TPOT against a local Ollama stream

The same timing technique applies unchanged to a locally served model through Ollama’s streaming chat endpoint, which yields one JSON object per generated chunk over HTTP instead of Anthropic’s typed SSE events:

import time

import ollama

start = time.perf_counter()
first_token_at = None
token_times = []

stream = ollama.chat(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Explain TTFT and TPOT in three sentences."}],
    stream=True,
)

for chunk in stream:
    now = time.perf_counter()
    if first_token_at is None:
        first_token_at = now
    token_times.append(now)

ttft = first_token_at - start
gaps = [b - a for a, b in zip(token_times, token_times[1:])]
tpot = sum(gaps) / len(gaps) if gaps else 0.0
print(f"TTFT: {ttft:.3f}s, TPOT: {tpot:.4f}s/token, chunks: {len(token_times)}")

Because Ollama runs on local hardware rather than shared hosted infrastructure, both numbers depend heavily on the GPU or CPU running it and on how many other requests it is serving concurrently, so a local TTFT/TPOT pair is only comparable to another measurement taken on the same machine and model, not to a hosted API’s numbers. See the official Ollama API docs for the /api/chat endpoint’s streaming response shape.

Related pages

References