Deployment and Release Patterns

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against LiteLLM 1.103.2 (proxy server), Langfuse Python SDK 4.16.0, MLflow 3.16.1, Spring AI 2.0.1 on Spring Boot 4.1.x with Java 21 and Python 3.12; the full list is the version baseline on the landing page. Every snippet here was written from the official documentation linked after it and was not run: no gateway was started, no key was generated and no traffic was split by this repository. Option names such as router_settings, router_settings.fallbacks and the /key/generate body fields were read from the LiteLLM documentation, and the exact set of fields supported can differ between releases, so check the page for the version you deploy. Model ids shown as examples are illustrations of the naming pattern, not a recommendation, and they must be checked against the provider’s current model list.

A model-backed feature changes in more ways than a normal service. The code can stay identical while a prompt is reworded, a provider moves an alias to a new snapshot, or a gateway route points at a different backend. Each of those is a release, and each can change every answer the DocsAssistant gives. This page is about releasing those changes deliberately: a stable interface in front, separately versioned artefacts behind it, and a staged path to production with a way back. The evaluation side of the story is on Evaluation in CI; the practices on registries and approvals are on Prompt and Model Management; the capacity side is on Serving at Scale. The general release vocabulary follows Chip Huyen’s AI Engineering and the operational framing of LLMOps by Aryan; the configuration below is written from current tool documentation because the book tooling pre-dates it.

API-first design

Treat the DocsAssistant as a contract, not as a model call. Clients (the web widget, the IDE plugin, the Slack bot) depend on an HTTP interface: a request shape, a response shape with citations, an error vocabulary and a version. Everything behind that interface — the prompt, the model, the retriever, even whether the answer is generated by one model or a cascade — is an implementation detail that can change without a client release. This is the same argument made on Serving RAG as an API, and the process-level options (a service per pipeline, a sidecar, an embedded library) are compared on Deployment Patterns.

Three consequences matter for release engineering:

The contract carries a version, the behaviour does not. Version the path or a header (/v1/answers) for breaking changes to the schema. Do not add a new API version for every prompt edit; behaviour changes are tracked by separate identifiers returned with each answer, described below.
Every response says which artefacts produced it. Return, or at least log, a prompt version, a model id, a route name and an index snapshot id. Without those identifiers you cannot attribute a bad answer to a release, split metrics by variant or roll back precisely.
Errors and degradation are part of the contract. Timeouts, a quota rejection and a fallback to a smaller model should be visible to the client in a documented way (a status code and a field such as degraded: true), so a canary or a failover does not silently change what the caller receives.

A client that only knows an OpenAI-compatible chat endpoint can also be pointed at a gateway instead of at a provider. That is the second layer of indirection and the subject of the next section.

A gateway between the application and the models

A gateway gives the application one model name and one set of credentials while the platform team decides which backend serves it. For the DocsAssistant, the application asks for the logical name docs-assistant; the gateway maps it to a self-hosted model on the vLLM tier, with a hosted model as a fallback. The comparison of gateway products, and the way a local server is placed behind one, is covered on LLM Gateways; this page uses the LiteLLM proxy to show the release-relevant parts.

Routing and fallbacks in config.yaml

The proxy reads a config.yaml with a model_list, router settings and general settings. Several entries sharing the same model_name form a group the router chooses between; a fallbacks list under router_settings names the group to try when a group keeps failing. The LiteLLM reliability page also shows fallbacks under litellm_settings in an advanced example, but its main examples place it under router_settings, which is the placement used on every page of this sub-section.

# config.yaml -- LiteLLM proxy (sketch, not run)
model_list:
  - model_name: docs-assistant            # the logical name the application uses
    litellm_params:
      model: openai/docs-assistant-v7     # a vLLM server speaking the OpenAI API
      api_base: http://vllm.internal:8000/v1
      api_key: os.environ/VLLM_API_KEY
  - model_name: docs-assistant-backup     # hosted fallback group
    litellm_params:
      model: anthropic/claude-sonnet-4-5-20250929   # dated snapshot id, see pinning below
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  timeout: 30
  fallbacks:
    - docs-assistant: ["docs-assistant-backup"]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

The os.environ/NAME form tells the proxy to read the value from the environment, so no credential appears in the file that is committed. The master_key is the administrative secret that authorises key management; applications never receive it. A persistent database is required for virtual keys and spend tracking, which is why database_url is present. The fallback is a resilience feature first and a release tool second, but it deserves a test: a fallback that has never been exercised is a guess. Run the evaluation gate against the backup group too, because a fallback that answers in a different style or refuses less is a behaviour change users will meet during an incident.

Source: LiteLLM: Proxy config.yaml, LiteLLM: Fallbacks and reliability.

Virtual keys and budgets

Do not hand the master key to services. The proxy can mint virtual keys scoped to specific model names, with a spend limit and a reset period, and attributed to a team or user. One key per consumer (the web widget, the nightly evaluation job, the shadow runner) means a runaway loop in one of them exhausts its own budget and not everyone’s, and spend reports can be cut by consumer. Cost accounting and denial-of-wallet controls are the subject of the cost page; the release-relevant point is that the evaluation and shadow traffic get their own keys with their own caps, so mirrored requests cannot starve production.

# Create a key for the shadow runner (sketch, not run). The master key comes from the environment.
curl -X POST "${LITELLM_URL}/key/generate" \
  -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
        "key_alias": "docs-assistant-shadow",
        "models": ["docs-assistant-candidate"],
        "max_budget": 25,
        "budget_duration": "30d",
        "rpm_limit": 20,
        "metadata": {"team": "docs-assistant", "purpose": "shadow"}
      }'

The response contains the generated key, which is shown once and must go straight into the consumer’s secret store, not into a repository or a chat message. The budget is expressed in the gateway’s spend unit and is tracked from the model prices the proxy knows about; this page quotes no prices, so size the number from your own invoices.

Source: LiteLLM: Virtual keys, LiteLLM: Budgets and rate limits.

Version prompts and models separately from code

Three things change at different speeds and for different reasons, and a release process that welds them together makes every change as slow as the slowest:

Artefact Changes when Versioned in Released by

Artefact	Changes when	Versioned in	Released by
Application code	A feature or bug fix lands	Git, container image tag	The normal service pipeline
Prompt	Wording, examples or output format are tuned	A prompt registry with immutable versions and movable labels or aliases	Moving a label, after the evaluation gate
Model route	A backend, model id or parameter changes	Gateway `config.yaml` in Git, or the gateway database	Config deploy or a route weight change

Application code

A feature or bug fix lands

Git, container image tag

The normal service pipeline

Prompt

Wording, examples or output format are tuned

A prompt registry with immutable versions and movable labels or aliases

Moving a label, after the evaluation gate

Model route

A backend, model id or parameter changes

Gateway config.yaml in Git, or the gateway database

Config deploy or a route weight change

Keeping the prompt out of the image has a real benefit: a prompt fix does not need a rebuild and a rollout, and a rollback is a label move, not a revert-and-redeploy. It has a real cost as well. A prompt edited in a UI, outside review, can reach production without the checks code gets. The remedy is the same as for code: immutable versions, a promotion step that requires the evaluation gate to pass, and an audit record of who moved which label when. Registries from MLflow, LangSmith and Langfuse all offer the version-plus-alias model; their features are compared on Prompt and Model Management.

The release record for a single DocsAssistant answer is therefore a tuple, which should be attached to the trace and the response metadata:

{
  "app_version": "2.8.1",
  "prompt": {"name": "docs-assistant-system", "version": 14, "label": "production"},
  "route": "docs-assistant",
  "model": "openai/docs-assistant-v7",
  "index_snapshot": "docs-2026-09-30"
}

Source: this tuple is a convention of this guide, not an official schema; the JSON syntax is defined by RFC 8259.

With that tuple every metric can be grouped by the artefact that changed, which is what makes a canary comparison possible.

Release strategies

Four strategies are used for model-backed features. They differ in who sees the candidate’s output, which is the question that decides the risk.

Strategy	Who sees the candidate	What it tells you
Shadow	Nobody; the candidate receives a copy of live requests and its answers are only recorded	Latency, errors, cost and offline-judged quality on real traffic, with zero user impact
Canary	A small, growing share of users receives the candidate’s answers	Real behaviour and real user signals, with a bounded blast radius
A/B test	Two stable groups receive the old or new variant for long enough to measure a difference	Whether the change improves a business or satisfaction metric, with statistical support
Blue-green	All users, switched at once between two full environments	Fast cut-over and fast reversal, but no gradual exposure

Strategy

Who sees the candidate

What it tells you

Shadow

Nobody; the candidate receives a copy of live requests and its answers are only recorded

Latency, errors, cost and offline-judged quality on real traffic, with zero user impact

Canary

A small, growing share of users receives the candidate’s answers

Real behaviour and real user signals, with a bounded blast radius

A/B test

Two stable groups receive the old or new variant for long enough to measure a difference

Whether the change improves a business or satisfaction metric, with statistical support

Blue-green

All users, switched at once between two full environments

Fast cut-over and fast reversal, but no gradual exposure

sequenceDiagram autonumber participant U as User participant API as DocsAssistant API participant GW as Gateway participant S as Stable (prompt v13) participant C as Candidate (prompt v14) participant J as Judge and metrics Note over U,J: Shadow: the user only ever sees the stable answer U->>API: question API->>GW: request GW->>S: serve S-->>GW: answer GW-->>API: stable answer API-->>U: stable answer API--)GW: mirrored copy after the reply (sampled, candidate route) GW--)C: serve C--)GW: candidate answer GW--)API: candidate answer, never shown to the user API--)J: stable and candidate pair logged, judged offline Note over U,J: Canary: a small share of users sees the candidate U->>API: question API->>GW: request (sticky assignment by user id) alt 5 percent bucket GW->>C: serve C-->>GW: answer else remaining users GW->>S: serve S-->>GW: answer end GW-->>API: answer from the assigned variant API-->>U: answer GW--)J: metrics tagged with variant

The shadow path costs extra model calls for every mirrored request, so it needs its own budget (the virtual key above) and usually a sampling rate. It cannot measure anything that depends on the user reacting to the answer, such as feedback ratings or follow-up questions, because the user never sees it. The canary path exposes real users, so it needs automatic abort rules and a sticky assignment: if a user flips between variants on consecutive messages of the same conversation, both the experience and the measurement are corrupted.

Shadow releases

Run the candidate on a copy of production traffic, record its answers and compare them with the stable answers using the judge and the deterministic checks from the evaluation pages (see LLM-as-a-Judge and Online Evaluation and Feedback). Mirror at the application layer, after the stable answer has been returned, so that a slow or failing candidate cannot delay the user:

import asyncio
import random

SHADOW_SAMPLE_RATE = 0.2  # fraction of requests mirrored to the candidate

async def answer(question: str, ctx: RequestContext) -> Answer:
    stable = await call_route("docs-assistant", question, ctx)

    if random.random() < SHADOW_SAMPLE_RATE:
        # fire and forget: never await the candidate on the request path
        asyncio.create_task(run_shadow(question, ctx, stable))

    return stable

async def run_shadow(question, ctx, stable):
    try:
        candidate = await asyncio.wait_for(
            call_route("docs-assistant-candidate", question, ctx), timeout=60
        )
        await record_pair(ctx.trace_id, stable, candidate)   # judged later, offline
    except Exception:
        await record_shadow_failure(ctx.trace_id)             # failures are data too

Source: the mirroring uses the Python standard library, see asyncio: Coroutines and Tasks; the shadow pattern itself is a convention of this guide.

The candidate uses its own gateway route and its own virtual key, so the mirrored calls are visible and capped. A shadow period should be long enough to cover the weekly and monthly rhythm of the questions (release weeks, end of quarter), not just a quiet afternoon, and it should record candidate failures as carefully as successes: a model that times out on long inputs shows up here, before a user meets it.

Canary releases

For a canary, a subset of users receives the candidate, the subset grows while metrics stay healthy, and the rollout stops or reverses when they do not. With the LiteLLM router the share can be expressed as a weight on two deployments under one logical name; with the weighted strategy the router picks proportionally to the weights.

# config.yaml fragment -- 95/5 split behind one logical name (sketch, not run)
model_list:
  - model_name: docs-assistant
    litellm_params:
      model: openai/docs-assistant-v7
      api_base: http://vllm.internal:8000/v1
      api_key: os.environ/VLLM_API_KEY
      weight: 95
  - model_name: docs-assistant
    litellm_params:
      model: openai/docs-assistant-v8
      api_base: http://vllm-canary.internal:8000/v1
      api_key: os.environ/VLLM_API_KEY
      weight: 5

router_settings:
  routing_strategy: simple-shuffle   # weighted pick when weights are set

A per-request weighted pick is a split of requests, not of users. It is acceptable for stateless single-turn checks and wrong for a conversational assistant where the same user would see answers from both variants. For a sticky canary, decide the variant in the application from a hash of the user or conversation id and call two distinct logical names, as in the flag example below. Whichever mechanism is used, tag every request with the variant so the metrics pipeline can compare them, and define the abort rule before the rollout starts: for example, error rate, p95 time to first token and judged-quality score on the sampled canary traffic compared with the stable group over the same window. The statistical care needed for comparing two rates on small samples is the one described on Evaluation in CI; a canary at five percent of a low-traffic service may need many days to accumulate enough observations to see a modest regression, which is a reason to rely on the shadow stage for quality and on the canary mainly for operational health.

Source: LiteLLM: Router and load balancing.

A/B tests

A canary asks "is the candidate safe?". An A/B test asks "is it better?". The difference is design: fixed assignment for the test duration, a pre-registered primary metric (for the DocsAssistant, a thumbs-up rate or the share of conversations that end without a re-ask), a sample size computed in advance, and no peeking and stopping at the first favourable reading. For a rate metric, the sample size per arm needed to detect an absolute difference around a baseline rate at significance and power is approximately:

The formula makes the practical point visible: halving the detectable difference quadruples the required traffic. Adaptive allocation, which shifts traffic toward the better variant as evidence accumulates, is the multi-armed bandit approach covered on Online Evaluation and Feedback; it trades some statistical clarity for less exposure to the worse variant.

Prompt feature flags

A feature flag decides at request time which variant a request uses, so a rollout, a kill switch and a per-segment experiment can be driven without a deploy. For prompts it combines two pieces: the flag (who gets what) and the registry label (what that variant resolves to). The flag below is evaluated against a hash of the user id so that a user stays in one variant:

import hashlib
from langfuse import get_client

langfuse = get_client()   # reads LANGFUSE_* settings from the environment

CANARY_PERCENT = 5        # read from a flag service or config in a real deployment

def bucket(user_id: str) -> int:
    return int(hashlib.sha256(user_id.encode()).hexdigest(), 16) % 100

def system_prompt_for(user_id: str):
    label = "candidate" if bucket(user_id) < CANARY_PERCENT else "production"
    prompt = langfuse.get_prompt("docs-assistant-system", label=label)
    return prompt, label

prompt, label = system_prompt_for(request.user_id)
messages = prompt.compile(product="irurueta docs")
# attach `label` and `prompt.version` to the trace so metrics can be split by variant

Moving the candidate label to a new version and the production label to the previous one is all that promotion and rollback require; the application code does not change. Langfuse caches fetched prompts on the client by default, so a label move reaches running instances after the cache interval and not instantly, and the SDK can fall back to a cached copy when the registry is unreachable. State the expected propagation delay in the runbook rather than assuming immediacy. MLflow’s prompt registry offers the same shape through aliases, and a dedicated flag service such as one implementing the OpenFeature specification can replace the hand-written bucket function when flags are shared across many features. A flag is also a piece of production configuration that decays: remove it once the rollout is complete, and keep at least one kill switch that is independent of the registry, so that a registry outage cannot also prevent a rollback.

Source: Langfuse: Prompt management, MLflow: Prompt Registry, OpenFeature: Introduction.

Rollback

A rollback is only as fast as the thing that gets rolled back is separable from the rest. With the artefacts above, the rollback ladder from fastest to slowest is:

Flag off. Set the canary percentage to zero or flip the kill switch. Seconds, no deploy.
Move the prompt label back. Point production at the previous version. Seconds to the cache interval.
Shift gateway weight or route. Set the candidate weight to zero or point the logical name at the previous deployment, then reload the proxy configuration. Minutes.
Redeploy the previous application image. The usual service rollback. Minutes, and the only one that also undoes code.
Restore the previous index snapshot. For a retrieval regression caused by a bad ingestion run; the snapshot ids in the release tuple make this a lookup instead of a forensic exercise.

Three habits make these steps real. Keep the previous version deployable for a defined period instead of deleting it when the candidate is promoted, because a rollback to a model server that no longer exists is not a rollback. Test the rollback in a rehearsal, including the case where the registry or the gateway database is the failing component. And decide the triggers in advance, wired to alerts (see Drift and Regression Monitoring): an automated abort on error rate or latency is reasonable, while a quality rollback is usually a human decision informed by sampled judged answers, because the metrics are noisier.

Pin vendor model versions

A hosted model name can mean two different things. A dated snapshot id refers to a specific model version that the provider commits to keeping stable until its announced retirement; an alias is a convenience name that the provider moves to a newer snapshot over time. Provider documentation describes both patterns, and their conventions differ, so read the model overview page of each vendor you use. Examples of the naming pattern are claude-sonnet-4-5-20250929 (dated) versus claude-sonnet-4-5 (alias), and gpt-4o-2024-08-06 (dated) versus gpt-4o (alias); treat these only as an illustration of the shape.

For production the rule is simple: pin the dated id in the gateway config and change it only through the same gate as any other release. An alias in production means a provider-side release you did not stage, which can change refusals, formatting and tool-call behaviour overnight, with no entry in your own history. The cost of pinning is that snapshots are eventually deprecated: subscribe to the provider’s deprecation notices, record the retirement date next to each pinned id, and schedule the migration as a normal canary well before that date. Aliases are fine in development and in a nightly job whose purpose is to detect that a newer snapshot has appeared, which feeds the regression runs described on Drift and Regression Monitoring. Open-weight models served from your own tier avoid provider-side moves, but the same discipline applies to the weights: pin the repository revision or digest, not a branch name such as main.

Source: Anthropic: Models overview, OpenAI: Models.

Putting it together for DocsAssistant

A prompt change is committed to the registry as a new immutable version. The evaluation gate from Evaluation in CI runs against it and records its result on the version.
The version is attached to the candidate label. A shadow job mirrors a sample of production traffic through the docs-assistant-candidate route using a capped virtual key, and the paired judge scores are compared with stable.
If the shadow comparison is clean, the flag moves the canary percentage from 0 to 5, then to larger shares, tagging every request with the variant and watching the abort rules.
The production label is moved to the new version. The previous version stays deployable for the agreed period.
If a rule fires at any step, the ladder above is followed from the top: flag off, label back, weight to zero.

Pitfalls

Aliases in production. Provider-side model moves arrive as unexplained quality changes. Pin dated ids.
Non-sticky canaries. Request-level random assignment in a multi-turn assistant mixes variants inside one conversation.
Shadow traffic without a budget. Mirrored calls double the model spend unless sampled and capped by their own key.
Unversioned prompts edited in place. If a version can be overwritten, a rollback target does not exist.
Labels with no audit trail. Record who moved a label and which evaluation run justified it.
A fallback that was never tested. The backup route should pass the same gate as the primary.
Leaking the master key. Applications get virtual keys with a model list and a budget; the master key stays with the platform team and in the environment of the proxy only.
Treating canary metrics as proof of quality. A small, short canary detects operational problems; quality claims need the judged samples and a sample size calculation.
Flags that never get removed. Retire the flag and the unused branch once the rollout is done.

Adversarial inputs and the security of the gateway itself, such as key exposure and prompt injection through mirrored traffic, belong to AI Security & Responsible AI, a separate sub-section.