Drift and Regression Monitoring

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against Python 3.12 with NumPy, GitHub Actions workflow syntax current on that date and Prometheus 3.15.0 alerting rules; the full list is the version baseline on the landing page. Every snippet on this page was written from the official documentation linked after it and was not run: no workflow was executed on GitHub, no Prometheus server evaluated the rule, and the Python functions are a self-contained NumPy sketch that this repository did not execute. The numbers in the text are illustrative thresholds to be tuned on your own traffic, not measurements, and no vendor prices or benchmark figures are quoted.

A release gate (see Evaluation in CI) protects the system against changes you make. Production also changes without you: the provider updates the model behind an alias, users start asking about a product that shipped last week, a re-indexing job shifts what retrieval returns. None of these produces a pull request, so none of them trips a gate. Drift and regression monitoring is the set of scheduled checks that notice these changes. The framing of production behaviour as a moving distribution that must be monitored, rather than a one-off acceptance test, is paraphrased from Huyen’s AI Engineering and from Aryan’s LLMOps (see the bibliography); the formulas and workflows below are written from current documentation.

Four sources of silent change

Source What changes Which signal sees it first

Vendor model update

The hosted model behind a name such as a "latest" alias is replaced, or an old snapshot is retired.

A scheduled run of the golden set against the live endpoint; output-shape statistics.

Input drift

The questions users ask: topics, length, language, the share of vague or multi-part questions.

Histogram distances between a reference window and the current window.

Retrieval or data drift

The corpus, the chunking or the index changes, so the same question retrieves different context.

Retrieval-score distribution, citation rate, the judge’s context-relevance score.

Output drift

Answer length, refusal rate, citation rate, format compliance move even though no one touched the prompt.

Per-day output statistics from traces.

Detection works in three layers of increasing cost: cheap statistics computed from every trace (input and output shape), a sampled judge on live traffic (see Online Evaluation and Feedback), and a scheduled replay of the golden set (see Building Eval Sets). The pipeline for the DocsAssistant is:

flowchart LR T[(Traces)] --> S[Shape stats
length, refusal, citations] T --> H[Input histograms] H --> D[Distance vs reference window] S --> M[Metrics endpoint] D --> M C[Scheduled cron] --> G[Golden set replay] G --> J[Judge scores] O[Sampled online judge] --> J J --> M M --> P[Prometheus rules] P -->|fires| R[Runbook: re-validate] R -->|pin or roll back| V[Model version]

Vendor model updates

A hosted model is a dependency you do not build. Providers typically offer two kinds of identifier: a dated snapshot that is meant to stay the same until it is retired, and an alias that points at the newest snapshot and moves when the provider publishes one. Each provider documents which of its identifiers are aliases and publishes a deprecation schedule; read yours rather than assuming. Three consequences follow for the DocsAssistant:

  • An alias is a deployment you did not approve. If the application calls an alias, a model release is a production release with no evaluation. Treat the alias as acceptable only for experiments.

  • A snapshot is not forever. Retirement dates mean that pinning postpones the work, it does not remove it; the calendar of retirements belongs in the same place as certificate expiry dates.

  • Even a pinned name can behave differently through changes outside the model, such as a gateway, a safety layer or a tokenizer change, so pinning does not replace the scheduled regression run below.

Self-hosted models avoid the surprise but have their own version axis: the weights, the quantisation, the serving engine and the chat template. Record all four as the model version, and log them on every trace, as described in Observability with OpenTelemetry. How a new model version reaches users, through shadow traffic, canary and rollback, is covered in Deployment and Release Patterns; this page covers how you find out that one arrived.

Input drift

Input drift means the distribution of questions in the current window differs from a reference window in which the system was known to be healthy. You cannot compare raw text, so reduce each question to a low-dimensional description and compare histograms. Two descriptions are cheap and useful:

  • Length histogram: the number of tokens in the question, in fixed bins. It catches pasted logs and stack traces, or a new integration that sends templated queries.

  • Topic histogram: embed each question, assign it to one of a fixed set of clusters (for example the k-means centroids fitted once on the reference window and then frozen), and count questions per cluster. It catches new subject areas.

Keep the bins and centroids fixed for the lifetime of a reference window. Refitting them on current data hides exactly the change you want to see.

Distances between histograms

Let and be the reference and current histograms, normalised to probabilities over bins. The population stability index sums, over bins, the difference of the proportions weighted by the log ratio:

It is symmetric and non-negative, and it is zero only when the histograms match. It is undefined when a bin is empty in either histogram, so apply a small floor to every proportion first. Credit-scoring practice popularised rule-of-thumb bands for it, but those bands come from a different domain; calibrate your own threshold by comparing windows that you know to be similar, such as consecutive weeks, and alert on a multiple of the largest value observed there.

The Jensen-Shannon divergence compares each histogram to their mixture , so it stays finite even when a bin is empty in one of them:

With the natural logarithm it is bounded above by , and its square root is a metric. Prefer it when histograms are sparse; prefer PSI when you want a number whose per-bin contributions can be listed to explain the shift.

drift.py: PSI and Jensen-Shannon divergence between two histograms
import numpy as np

EPS = 1e-6  # floor applied to every proportion before taking logarithms


def to_proba(counts) -> np.ndarray:
    """Normalise raw counts to proportions, flooring empty bins."""
    counts = np.asarray(counts, dtype=float)
    p = counts / counts.sum()
    p = np.clip(p, EPS, None)
    return p / p.sum()


def psi(ref_counts, cur_counts) -> float:
    p, q = to_proba(ref_counts), to_proba(cur_counts)
    return float(np.sum((q - p) * np.log(q / p)))


def jsd(ref_counts, cur_counts) -> float:
    """Jensen-Shannon divergence in nats, bounded by ln 2."""
    p, q = to_proba(ref_counts), to_proba(cur_counts)
    m = 0.5 * (p + q)
    kl = lambda a, b: float(np.sum(a * np.log(a / b)))
    return 0.5 * kl(p, m) + 0.5 * kl(q, m)


def length_histogram(token_counts, edges=(0, 8, 16, 32, 64, 128, 256, 10**9)) -> np.ndarray:
    hist, _ = np.histogram(token_counts, bins=edges)
    return hist


# Example: reference week vs. current week, token lengths of user questions.
reference = length_histogram(np.random.default_rng(0).gamma(4.0, 8.0, 5000))
current = length_histogram(np.random.default_rng(1).gamma(4.0, 12.0, 5000))
print(f"PSI={psi(reference, current):.3f}  JSD={jsd(reference, current):.3f}")

SciPy ships a jensenshannon function that returns the square root of this quantity (the distance, not the divergence), so compare like with like if you switch implementations. The same two functions work unchanged on the topic histogram: only the construction of the counts differs.

Derived from the NumPy: histogram and SciPy: jensenshannon documentation.

A distance only says that something moved. The follow-up is to list the bins with the largest contribution to the distance, pull a sample of questions from those bins, and decide whether the shift is a new legitimate use (add cases to the golden set) or noise to ignore. Sampling those questions into the review queue is part of the flywheel in Online Evaluation and Feedback.

Output drift

Output drift is a change in the shape of answers. Four statistics computed from every trace need no judge and no labels:

Statistic What a shift usually means

Answer length (tokens)

A model update changed verbosity, or a prompt edit leaked into an unrelated path. Track the median and a high percentile.

Refusal rate

A safety layer or a model update became more (or less) cautious; or retrieval failed so the assistant says it cannot answer.

Citation rate

The share of answers that cite at least one source. A drop often means retrieval returned nothing usable or the model stopped following the citation instruction.

Format compliance

The share of answers that parse as the expected structure (for example valid JSON or a required heading). A drop is an immediate user-visible failure.

Define each one as a counter pair so that Prometheus can compute rates over any window: for example a docsassistant_answers_total counter and a docsassistant_answers_refused_total counter incremented by the application when it classifies an answer. Naming and label rules (keep the label set small, never put a user or question id in a label) are in Metrics, Dashboards and Alerting. These shape statistics also double as a cheap early warning for a vendor update: a shift in median length and refusal rate on the same day, with no deployment of yours, is the typical fingerprint.

Scheduled regression runs

Replay the golden set on a timer against the same endpoint users hit. The result tells you whether the system as deployed still meets the bar, independently of any pull request. Run it at least daily for hosted models that you call through an alias, and at a lower frequency for pinned or self-hosted ones.

regression-schedule.yml: scheduled golden-set replay (GitHub Actions)
name: scheduled-regression
on:
  schedule:
    - cron: "17 3 * * *"   # daily; minute 17 avoids the top-of-hour load spike
  workflow_dispatch: {}      # lets you run it by hand after an incident

permissions:
  contents: read

concurrency:
  group: scheduled-regression
  cancel-in-progress: false

jobs:
  replay:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r evals/requirements.txt
      - name: Replay golden set against production endpoint
        env:
          DOCSASSISTANT_URL: ${{ vars.DOCSASSISTANT_URL }}
          DOCSASSISTANT_API_KEY: ${{ secrets.DOCSASSISTANT_API_KEY }}
          JUDGE_API_KEY: ${{ secrets.JUDGE_API_KEY }}
        run: python evals/replay.py --set golden-v7 --out results.json
      - name: Publish scores to the metrics gateway
        env:
          PUSHGATEWAY_URL: ${{ vars.PUSHGATEWAY_URL }}
        run: python evals/push_scores.py results.json
      - uses: actions/upload-artifact@v7
        with:
          name: regression-results
          path: results.json

Scheduled workflows run from the latest commit on the default branch, the shortest allowed interval is five minutes, and the time is UTC; the documentation also notes that scheduled runs can be delayed under load and that a schedule in a public repository is disabled after a period without repository activity, so a monitoring job should itself be monitored (see the "absent" rule below). Credentials are read from repository secrets and passed as environment variables, never written into the workflow file.

Derived from the GitHub Actions documentation on the schedule event and on using secrets.

Two design choices keep the replay trustworthy. Hold the judge model and judge prompt fixed (the judge is also a dependency that can drift, so version it as described in LLM-as-a-Judge), and keep a small subset of cases whose answers are scored by deterministic checks only, so that a moving judge cannot hide a moving model.

Alerting on score drops

A judge score is a noisy measurement, so alerting on a single low value produces pages for nothing. Two standard devices make the rule statistical rather than hopeful.

A Wilson-bound rule

Let of replayed or sampled answers pass the judge, with observed proportion . The Wilson score interval for the true pass rate, with critical value , is

Raise an alert when the upper bound of this interval falls below the agreed target: the data then say, with the confidence implied by , that the pass rate is below target, not merely that this sample came out low. Unlike the normal approximation used for sample-size planning, the Wilson interval behaves sensibly at small and near 0 or 1, which is where a healthy assistant lives. The same bound is used for the CI threshold in Evaluation in CI.

A control-chart rule

A control chart (Shewhart) draws the metric over time with a centre line and limits. Estimate the centre line and the standard deviation of the daily pass rate from a baseline period in which the system was healthy, then signal when

The first clause catches a sudden drop and the second a smaller sustained one; both use only the current and past points, so the rule can be evaluated on the day itself. For slow drift that never crosses a limit on any single day, a cumulative-sum (CUSUM) or exponentially weighted chart is the standard tool; the engineering handbook of the national standards body listed in the references describes all three. The window length and the multipliers are tuning parameters: check them by replaying the rule over a history that includes a known incident and counting how many days it takes to fire.

A Prometheus rule

This page does not define a second quality metric. The online judge publishes the docsassistant_judge_score histogram described in Metrics, Dashboards and Alerting, and that page already records rubric:docsassistant_judge_good_ratio:rate1h and alerts on it with DocsAssistantJudgeScoreDegraded (see its alert rules). The rule below is a longer-window companion on the same series: it computes the same "share of sampled answers scoring above 0.7" over seven days, which smooths day-to-day noise and catches slow drift that the one-hour rule never sees, and it adds a guard for too-small samples.

docsassistant-quality.rules.yml: seven-day companion to the judge-score alert
groups:
  - name: docsassistant-quality-7d
    rules:
      # Same histogram and same 0.7 pass mark as
      # rubric:docsassistant_judge_good_ratio:rate1h, over seven days.
      - record: rubric:docsassistant_judge_good_ratio:rate7d
        expr: |
          (
            sum by (rubric) (increase(docsassistant_judge_score_bucket{le="+Inf"}[7d]))
              - sum by (rubric) (increase(docsassistant_judge_score_bucket{le="0.7"}[7d]))
          )
            / clamp_min(sum by (rubric) (increase(docsassistant_judge_score_bucket{le="+Inf"}[7d])), 1e-9)

      - alert: DocsAssistantJudgeScoreDegraded7d
        expr: |
          rubric:docsassistant_judge_good_ratio:rate7d{rubric="faithfulness"} < 0.85
          and
          sum by (rubric) (increase(docsassistant_judge_score_count{rubric="faithfulness"}[7d])) > 400
        for: 6h
        labels:
          severity: ticket
          slo: quality
        annotations:
          summary: "Share of well-scored sampled answers below target over 7 days"
          description: "Good-answer ratio is {{ $value | humanizePercentage }}. Follow the re-validation runbook."

      - alert: DocsAssistantRegressionRunMissing
        expr: absent(docsassistant_regression_last_success_timestamp_seconds) or
              time() - docsassistant_regression_last_success_timestamp_seconds > 2 * 86400
        for: 1h
        labels:
          severity: ticket
        annotations:
          summary: "Scheduled regression replay has not succeeded for two days"

The target (0.85) and the pass mark (0.7) are the same placeholders as on the metrics page, so the two alerts disagree only about the window: the one-hour alert reports a recent drop, the seven-day one a sustained one. Change both together. The > 400 guard is the cheap version of the Wilson rule: it refuses to fire before the sample is large enough for the target to be distinguishable from noise. Where you need the exact interval, compute it in the replay job and export its upper bound as a gauge, then alert on that gauge directly. The for clause adds a persistence requirement, and the last alert watches the watcher, since a silently disabled schedule looks identical to a healthy system.

Derived from the Prometheus documentation on alerting rules and recording rules.

Pinning and re-validation

When a signal fires, or a retirement notice arrives, follow the same procedure each time. Writing it down is what turns an incident into a routine.

  1. Pin. Switch the application from the alias to the last snapshot known to pass, if the provider still serves it, or roll the gateway route back to the previous model version. This stops further damage before diagnosis.

  2. Confirm. Replay the golden set against both the pinned and the suspect version, with the same judge, and compare them with a paired test on per-case results (see Evaluation in CI). A difference inside the noise band means the alert was a false alarm and the pin can be lifted.

  3. Localise. Split the failing cases by slice (topic, language, tool use). Look at which input bins moved, with the histogram distances above, to separate "the model changed" from "the questions changed".

  4. Fix or adapt. Edit the prompt, adjust retrieval, or accept the new behaviour and update the thresholds, through the normal pull request and gate.

  5. Re-validate and release. Run the full gate, then canary the candidate (see Deployment and Release Patterns) and watch the online scores.

  6. Record. Add the failing production cases to the next golden-set version, and record the model version, judge version and threshold in the change log so the next comparison has a baseline.

For planned changes, the same procedure runs ahead of time: when a provider announces a new snapshot or a retirement date, schedule the re-validation before the date, not after it. The set of cases and slices the procedure depends on comes from Building Eval Sets, and the dashboards that show the pinned and candidate versions side by side are the subject of Metrics, Dashboards and Alerting.

References