Cost Management

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against LiteLLM 1.103.2 (proxy), Langfuse 4.49.0, Prometheus 3.15.0 and Python 3.12; the full list is the version baseline on the landing page. Every snippet was written from the official documentation linked after it and was not run. No vendor prices are quoted: prices change often and differ per contract, so every formula uses symbols and every example uses placeholder values that you replace from your own price list.

LLM applications have a cost profile unlike ordinary services: cost scales with tokens, not requests, and one user can generate arbitrarily many tokens. The practices below are paraphrased from the production chapters of Huyen’s AI Engineering and Aryan’s LLMOps (see the bibliography); the formulas and configuration come from current documentation.

A token-cost model

For one request with uncached input tokens, input tokens served from a prompt cache, and output tokens, the cost is

where , and are the per-token prices for fresh input, cached input and output. Output tokens are usually priced higher than input tokens and a cache read lower, which is why the three terms are kept apart. The monthly bill for the DocsAssistant with requests, a batch share priced at a fraction of the interactive price, and a semantic-cache hit rate that skips the model call entirely, is approximately

with covering embedding, vector store, observability and GPU reservations. The model is deliberately simple; its value is that each lever below maps to one symbol: caching changes and , batching changes and , routing changes the prices, and trimming changes .

Cost per request should be a first-class metric, computed from the token counts that the OpenTelemetry GenAI usage attributes already carry, and charted next to quality (see Metrics, Dashboards and Alerting). A cost reduction that is not paired with an evaluation run (see Evaluation in CI) is a quality regression waiting to be discovered.

Prompt caching

Providers and serving engines can reuse the computation for a prompt prefix that repeats between requests. Hosted providers expose this either automatically or through explicit cache markers, with provider-specific minimum prefix lengths, lifetimes and discounts documented on their pricing pages; vLLM and SGLang reuse the KV cache of identical prefixes (see Serving at Scale). The design consequence is the same everywhere: put the stable content first (system prompt, tool definitions, policy text) and the variable content last (retrieved passages, the user question). A prompt that interpolates a timestamp or user name at the top defeats the cache.

def build_messages(system_prompt: str, tools_preamble: str, passages: list[str], question: str) -> list[dict]:
    # Stable prefix first: identical across requests, so it can be cached.
    stable = system_prompt + "\n\n" + tools_preamble
    # Volatile suffix last: changes on every request.
    context = "\n\n".join(passages)
    return [
        {"role": "system", "content": stable},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
    ]

Source pattern: vLLM: Automatic Prefix Caching. Track the cached share of input tokens as a metric; a falling ratio after a prompt edit is the usual sign that a variable value crept into the prefix. The saved fraction is .

Batch APIs

Work that nobody is waiting for — nightly golden-set runs, bulk re-embedding, offline judging of sampled traffic — does not need an interactive latency. Several providers offer asynchronous batch endpoints that accept a file of requests and return results within a stated window at a reduced price; the discount, the window and the size limits are in each provider’s documentation. In the model above this is the and pair. For self-hosted engines the equivalent is offline batched inference, where throughput matters and latency does not. The DocsAssistant moves its CI evaluation, scheduled regression runs (see Drift and Regression Monitoring) and sampled online judging to batch paths and keeps the interactive path for answers.

With a hosted batch endpoint the job is a JSONL file of ordinary requests, each with a custom_id used to match results, uploaded and submitted in two calls. The example uses the OpenAI Python SDK; other providers' batch endpoints follow the same file-in, file-out shape with their own field names.

import json
import os

from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
MODEL = os.environ["BATCH_MODEL"]  # placeholder: your approved model id

with open("golden.jsonl", encoding="utf-8") as src, open("batch_input.jsonl", "w", encoding="utf-8") as dst:
    for line in src:
        item = json.loads(line)
        dst.write(json.dumps({
            "custom_id": item["id"],                       # results come back unordered: match on this
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": {"model": MODEL, "messages": [{"role": "user", "content": item["question"]}], "max_tokens": 800},
        }) + "\n")

batch_file = client.files.create(file=open("batch_input.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    metadata={"job": "docsassistant-nightly-eval"},
)
print(batch.id)  # poll client.batches.retrieve(batch.id); read client.files.content(<output_file_id>) when done

Source: OpenAI: Batch API guide. The self-hosted equivalent is vLLM’s offline LLM.generate over a list of prompts, described in vLLM: Offline Inference.

Routing and cascades

Not every question needs the largest model. Two patterns reduce cost:

Routing chooses one model per request from a cheap signal, such as a classifier on the question, the presence of tool calls, or the tenant’s plan.
Cascading tries a small model first and escalates only when a confidence check fails, for example when a judge or a rule (no citation, low retrieval score) rejects the draft.

The expected cost of a two-stage cascade is

where is the fraction escalated and , are the average costs of one call to each model. The alternative is to send everything to the large model at per request, so the cascade saves money only when , that is

Every request pays for the small attempt, but only the share that the small model answers acceptably avoids the large call; the small model’s cost must be covered by those avoided calls. A cheap small model with a high escalation rate can cost more than no cascade at all, and the margin shrinks to nothing as approaches 1. Measure and the quality of the accepted small-model answers on the golden set before enabling it; routing quality is itself something to evaluate (see Building Eval Sets). The routing and fallback mechanics of the gateway are in Deployment and Release Patterns, and the RAG-specific query routing is in Query Understanding and Routing.

A gateway router handles the model groups and the error fallbacks; the quality-based escalation of a cascade is application logic, because LiteLLM’s fallbacks fire on errors (rate limits, provider failures), not on a weak answer.

import os

from litellm import Router

router = Router(
    model_list=[
        {"model_name": "docs-small",
         "litellm_params": {"model": os.environ["SMALL_MODEL"], "api_key": os.environ["SMALL_MODEL_API_KEY"]}},
        {"model_name": "docs-large",
         "litellm_params": {"model": os.environ["LARGE_MODEL"], "api_key": os.environ["LARGE_MODEL_API_KEY"]}},
    ],
    fallbacks=[{"docs-small": ["docs-large"]}],   # on errors only
)


def acceptable(answer: str, retrieval_score: float) -> bool:
    """Cheap confidence check: cites a source and retrieval was confident. Tune on the golden set."""
    return "[source:" in answer and retrieval_score >= 0.5


def cascade(messages: list[dict], retrieval_score: float) -> tuple[str, str]:
    draft = router.completion(model="docs-small", messages=messages).choices[0].message.content
    if acceptable(draft, retrieval_score):
        return draft, "docs-small"
    final = router.completion(model="docs-large", messages=messages).choices[0].message.content
    return final, "docs-large"   # log which tier answered: the share of "docs-large" is q

Source: LiteLLM: Router and LiteLLM: Fallbacks. The model ids come from the environment, and the threshold in acceptable is a placeholder to tune on labelled data.

Context trimming

Input tokens dominate the bill for retrieval-augmented systems because every request carries retrieved passages and history. Levers, in the order they are usually applied:

Retrieve fewer, better passages. Rerank and cut to the smallest that holds the golden-set quality.
Summarise or window the conversation history rather than resending it whole.
Strip boilerplate from passages (navigation, repeated headers) at ingestion time.
Cap output length with a maximum-token setting sized to the longest legitimate answer.

A trimmer that enforces a token budget keeps the highest-ranked passages and the newest turns that fit, counting tokens with the tokenizer of the target model family (here tiktoken; use your model’s own tokenizer when it differs):

import tiktoken

ENC = tiktoken.get_encoding("o200k_base")   # pick the encoding that matches your model


def n_tokens(text: str) -> int:
    return len(ENC.encode(text))


def trim_context(passages: list[tuple[float, str]], history: list[dict], budget: int, k: int = 4) -> tuple[list[str], list[dict]]:
    """Top-k passages by rerank score, then as much recent history as the remaining budget allows."""
    kept_passages, used = [], 0
    for _, text in sorted(passages, key=lambda p: p[0], reverse=True)[:k]:
        cost = n_tokens(text)
        if used + cost > budget:
            break
        kept_passages.append(text)
        used += cost
    kept_history = []
    for turn in reversed(history):          # newest first
        cost = n_tokens(turn["content"])
        if used + cost > budget:
            break
        kept_history.insert(0, turn)
        used += cost
    return kept_passages, kept_history

Source: tiktoken README (get_encoding, encode). Each cut is a hypothesis about quality, so each goes through the evaluation gate. The retrieval and caching trade-offs are covered in more depth in Caching, Latency and Cost and in Production Considerations for Vector Search and RAG.

Semantic caching

An exact-match cache helps only when users type identical questions. A semantic cache embeds the question and returns a stored answer when a previous question is close enough, which sets in the model above. The risk is serving a wrong answer to a question that is similar but not equivalent ("How do I enable X in version 2?" versus version 3), so a semantic cache needs a similarity threshold tuned on labelled pairs, a time-to-live, invalidation when the corpus changes, and a partition key (tenant, product version, language) so answers never cross a boundary. Measure the false-hit rate with the judge on sampled hits, and keep tenant-specific or personal answers out of the cache.

The lookup below shows the three controls together: the partition key scopes every entry, the threshold decides a hit, and the time-to-live expires stale answers. embed is whatever embedding call the application already uses for retrieval; vectors are normalised so the dot product is the cosine similarity.

import time

import numpy as np

THRESHOLD = 0.92        # placeholder: tune on labelled equivalent / non-equivalent question pairs
TTL_SECONDS = 24 * 3600


class SemanticCache:
    def __init__(self, embed):
        self.embed = embed
        self.entries: dict[tuple, list[tuple[np.ndarray, str, float]]] = {}

    def _vector(self, text: str) -> np.ndarray:
        v = np.asarray(self.embed(text), dtype=np.float32)
        return v / np.linalg.norm(v)

    def get(self, question: str, tenant: str, docs_version: str, lang: str) -> str | None:
        key = (tenant, docs_version, lang)                  # never match across a partition
        now, q = time.time(), self._vector(question)
        live = [e for e in self.entries.get(key, []) if now - e[2] < TTL_SECONDS]
        self.entries[key] = live
        if not live:
            return None
        sims = np.array([float(vec @ q) for vec, _, _ in live])
        best = int(sims.argmax())
        return live[best][1] if sims[best] >= THRESHOLD else None

    def put(self, question: str, answer: str, tenant: str, docs_version: str, lang: str) -> None:
        self.entries.setdefault((tenant, docs_version, lang), []).append((self._vector(question), answer, time.time()))

    def invalidate(self, docs_version: str) -> None:
        """Call on a corpus rebuild: drop every partition for that documentation version."""
        self.entries = {k: v for k, v in self.entries.items() if k[1] != docs_version}

Source: NumPy: numpy.linalg.norm. In production the same lookup runs against a vector store; LiteLLM documents a redis-semantic cache type with similarity_threshold and ttl settings in LiteLLM: Caching. Check how that cache scopes its entries before relying on it for the tenant and version partition shown here.

Gateway budgets and quotas

The LiteLLM proxy issues virtual keys and can attach a spend budget and rate limits to each key or team; the application holds the virtual key, never the provider key.

# litellm_config.yaml
model_list:
  - model_name: docs-assistant
    litellm_params:
      model: openai/gpt-4o-mini        # placeholder: use your approved model
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Source: LiteLLM: Proxy configuration (the os.environ/ form reads a value from the environment).

# Create a key for one team with a monthly budget and rate limits
curl -X POST "${LITELLM_URL}/key/generate" \
  -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
        "team_id": "docs-assistant",
        "max_budget": 200,
        "budget_duration": "30d",
        "tpm_limit": 100000,
        "rpm_limit": 600
      }'

Source: LiteLLM: Virtual Keys and LiteLLM: Budgets and rate limits. The budget amount above is a placeholder in the gateway’s configured currency. When a key exhausts its budget the gateway rejects requests; decide in advance whether the application degrades (smaller model, cached answers only) or fails closed, and test that path.

Per-tenant attribution

A single total hides which tenant, feature or prompt version causes the spend. Attribute cost at the point where tokens are counted by attaching identifiers to every trace and request: tenant, feature, prompt version and model. Pseudonymise user identifiers (see Data Operations).

from langfuse import observe, get_client

@observe(name="docs-assistant-answer")
def answer(question: str, tenant_id: str, prompt_version: str) -> str:
    client = get_client()
    client.update_current_trace(
        user_id=tenant_id,                      # pseudonymous tenant id, not an email
        tags=["docs-assistant", f"prompt:{prompt_version}"],
        metadata={"feature": "qa"},
    )
    ...

Source: Langfuse: Python SDK instrumentation and Langfuse: Token and cost tracking. The monthly cost report groups spend by those dimensions and divides by a unit of value (answered questions, resolved tickets) to give cost per outcome, which is the number a product owner can act on.

Self-hosting break-even

Self-hosting an open-weight model trades a variable cost per token for a mostly fixed cost per GPU-hour. The method needs three measured inputs, not published numbers: the sustained throughput of your engine on your prompts (tokens per second at your latency target, from the load-test harness in Serving at Scale), the achievable utilisation of the fleet over a month, and the all-in monthly fleet cost (compute, storage, networking and the people-time to operate it). The effective self-hosted price per token is

with the seconds in the month. Self-hosting wins when is below the blended hosted price for the same quality, and the break-even monthly volume is tokens. Three corrections are usually larger than the headline difference: low utilisation (a fleet idle at night multiplies ), the quality gap between the open model and the hosted one on your golden set, and the operational burden. Compute the result with your own numbers and revisit it when either side changes its prices or you change engines. Local and on-premise options are covered in Running LLMs Locally.

Denial-of-wallet alerts

Because spend follows tokens, an attacker or a runaway loop can drain a budget without breaking availability: long prompts, agents that call tools in a cycle, or a leaked key. The defensive controls belong to Unbounded Consumption and Denial of Wallet in the AI Security & Responsible AI sub-section, but the operational side is monitoring: alert on the spend rate, not on the total.

groups:
  - name: llm-cost
    rules:
      - alert: LlmSpendRateHigh
        expr: sum(rate(llm_cost_dollars_total[15m])) * 3600 > 50
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "LLM spend rate is above the hourly ceiling"
      - alert: LlmTokensPerRequestAnomaly
        expr: |
          sum(rate(llm_tokens_total[15m])) / sum(rate(llm_requests_total[15m]))
            > 3 * (sum(rate(llm_tokens_total[1d])) / sum(rate(llm_requests_total[1d])))
        for: 15m
        labels:
          severity: ticket

Source: Prometheus: Alerting rules. The metric names are this guide’s own conventions and the thresholds are placeholders. Pair the alerts with hard limits at the gateway (budgets, request-size caps, maximum agent steps) so that an alert is a notification and not the only defence, and give each alert a runbook (see Operational Governance).