Metrics, Dashboards and Alerting

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against Prometheus 3.15.0, Grafana 13.2.3, vLLM 0.30.0, Spring AI 2.0.1 on Spring Boot 4.1.x with Micrometer, and the OpenTelemetry GenAI semantic conventions at Development stability (registry schema gen-ai-dev/1.42.0-dev); the full list is the version baseline on the landing page. Every snippet here was written from the official documentation linked after it and was not run: no Prometheus, Grafana or vLLM instance was started by this repository, no rule was evaluated and the dashboard JSON was not imported into Grafana (it was only checked to be well-formed JSON). Metric names are the ones documented by the OpenTelemetry conventions, vLLM and Micrometer on that date and they differ across versions and exporters: the OpenTelemetry-to-Prometheus translation adds unit suffixes, and vLLM has renamed metrics between releases. Names marked as application-defined (docsassistant*) are invented for this example. Always list what your own /metrics endpoint exposes before copying a query.

Traces explain one bad answer; metrics tell you that answers are going bad, how many, and since when. This page covers the metric side of the stack described in Observability with OpenTelemetry: what to measure for an LLM application, how to turn the measurements into Prometheus rules, how to show them in Grafana, and how to commit to targets (SLOs) that the alerts defend. Prometheus and Grafana mechanics are not repeated here; they live in the recording and alerting rules, Alertmanager, Grafana dashboards and Grafana alerting pages. The Spring Boot side is in Metrics and Observability. The vocabulary of "golden signals" comes from the Google SRE book chapter on monitoring distributed systems; the LLM-specific additions are this guide’s own adaptation, informed by the monitoring chapters of Huyen’s AI Engineering and Aryan’s LLMOps (paraphrased, not quoted).

Golden signals, adapted

The four classic signals are latency, traffic, errors and saturation. They still apply to DocsAssistant, but each needs an LLM reading, and seven more signals have no counterpart in an ordinary web service.

Signal What it means for DocsAssistant Where the metric comes from

Signal	What it means for DocsAssistant	Where the metric comes from
Latency	End-to-end request time, but also time to first token (TTFT, how long the user stares at a blank box) and time per output token (TPOT, how smoothly text streams). A long answer is slow by nature, so total duration alone hides regressions.	Client side: the OpenTelemetry `gen_ai.client.operation.duration` histogram and the streaming metrics `gen_ai.server.time_to_first_token` and `gen_ai.server.time_per_output_token`. Server side: vLLM’s own TTFT, TPOT and end-to-end histograms.
Traffic	Requests per second, and tokens per second in and out; token throughput predicts load better than request count.	Request counts from the duration histogram’s `_count` series; `gen_ai.client.token.usage` split by token type; vLLM prompt and generation token counters.
Errors	HTTP failures and provider errors, but also answers that are valid HTTP and wrong: empty output, truncation by `max_tokens`, refusals, malformed structured output, tool-call failures.	The `error.type` attribute on the duration histogram; a finish-reason label if your gateway exposes one; application counters for refusals and schema failures.
Saturation	Queue depth and KV-cache occupancy on the serving side, rate-limit rejections from a provider, gateway concurrency.	vLLM running and waiting request gauges and cache usage gauge; the gateway’s own metrics.
Token usage	Input and output tokens per request and per tenant; the input to cost, and an early signal for prompt bloat or an agent loop.	`gen_ai.client.token.usage` histogram.
Cost	Currency per request, per tenant, per day. Derived, not measured: tokens multiplied by a price table that you own.	A recording rule over token usage (see below); prices are configuration, not part of this page.
Quality	A judge score or pass rate on sampled live traffic, plus thumbs and regeneration rates from the feedback widget. Delayed and noisy, so it is smoothed over windows and treated as a trend, not a pager.	Application metrics emitted by the online evaluation job (see Online Evaluation and Feedback).
Cache hit rate	Share of requests served from a semantic cache, a provider prompt cache, or vLLM’s prefix cache. A drop usually precedes a latency and cost increase.	vLLM prefix-cache query and hit counters; provider cache-read token counts if reported; application counter for the semantic cache.
Empty retrieval	Share of questions for which the retriever returned no chunk at all (or none above the relevance cut-off). The model then answers from its own weights or refuses; a rise usually means an index build failed, a filter is wrong, or users started asking about content that is not indexed.	Application counter `docsassistant_retrieval_requests_total` with label `result` set to `empty` or `found`, incremented once per retrieval call.
Refusal rate	Share of answers in which the assistant declined ("I cannot find this in the documentation", a safety refusal). Some refusals are correct; a sudden change in either direction follows a prompt, model or guardrail change.	Application counter `docsassistant_answers_total` with label `outcome` set to `answered` or `refused`, set by the application’s refusal detector or the guardrail result.
Feedback rate	Share of answers that received a thumbs vote at all, and the share of votes that are thumbs-down. A falling vote rate makes the thumbs-down ratio less trustworthy; a rising thumbs-down ratio is the cheapest human quality signal available.	Application counter `docsassistant_feedback_total` with label `vote` set to `up` or `down`, incremented by the feedback endpoint, divided by `docsassistant_answers_total`.

Latency

End-to-end request time, but also time to first token (TTFT, how long the user stares at a blank box) and time per output token (TPOT, how smoothly text streams). A long answer is slow by nature, so total duration alone hides regressions.

Client side: the OpenTelemetry gen_ai.client.operation.duration histogram and the streaming metrics gen_ai.server.time_to_first_token and gen_ai.server.time_per_output_token. Server side: vLLM’s own TTFT, TPOT and end-to-end histograms.

Traffic

Requests per second, and tokens per second in and out; token throughput predicts load better than request count.

Request counts from the duration histogram’s _count series; gen_ai.client.token.usage split by token type; vLLM prompt and generation token counters.

Errors

HTTP failures and provider errors, but also answers that are valid HTTP and wrong: empty output, truncation by max_tokens, refusals, malformed structured output, tool-call failures.

The error.type attribute on the duration histogram; a finish-reason label if your gateway exposes one; application counters for refusals and schema failures.

Saturation

Queue depth and KV-cache occupancy on the serving side, rate-limit rejections from a provider, gateway concurrency.

vLLM running and waiting request gauges and cache usage gauge; the gateway’s own metrics.

Token usage

Input and output tokens per request and per tenant; the input to cost, and an early signal for prompt bloat or an agent loop.

gen_ai.client.token.usage histogram.

Cost

Currency per request, per tenant, per day. Derived, not measured: tokens multiplied by a price table that you own.

A recording rule over token usage (see below); prices are configuration, not part of this page.

Quality

A judge score or pass rate on sampled live traffic, plus thumbs and regeneration rates from the feedback widget. Delayed and noisy, so it is smoothed over windows and treated as a trend, not a pager.

Application metrics emitted by the online evaluation job (see Online Evaluation and Feedback).

Cache hit rate

Share of requests served from a semantic cache, a provider prompt cache, or vLLM’s prefix cache. A drop usually precedes a latency and cost increase.

vLLM prefix-cache query and hit counters; provider cache-read token counts if reported; application counter for the semantic cache.

Empty retrieval

Share of questions for which the retriever returned no chunk at all (or none above the relevance cut-off). The model then answers from its own weights or refuses; a rise usually means an index build failed, a filter is wrong, or users started asking about content that is not indexed.

Application counter docsassistant_retrieval_requests_total with label result set to empty or found, incremented once per retrieval call.

Refusal rate

Share of answers in which the assistant declined ("I cannot find this in the documentation", a safety refusal). Some refusals are correct; a sudden change in either direction follows a prompt, model or guardrail change.

Application counter docsassistant_answers_total with label outcome set to answered or refused, set by the application’s refusal detector or the guardrail result.

Feedback rate

Share of answers that received a thumbs vote at all, and the share of votes that are thumbs-down. A falling vote rate makes the thumbs-down ratio less trustworthy; a rising thumbs-down ratio is the cheapest human quality signal available.

Application counter docsassistant_feedback_total with label vote set to up or down, incremented by the feedback endpoint, divided by docsassistant_answers_total.

Two design points follow. First, label discipline: model, route, tenant tier and outcome are good labels; user id, session id, prompt text and trace id are not, because each unique value creates a new time series and Prometheus memory grows with cardinality. Per-user detail belongs in traces and logs. Second, latency percentiles are histograms, not averages: a mean hides the slow tail that users notice. Both points are expanded in Instrumenting Applications.

Producing the metrics

Three producers feed Prometheus in the DocsAssistant stack.

The Spring AI application. Spring AI emits Micrometer observations for model calls, which the Micrometer Prometheus registry (or an OpenTelemetry bridge) exposes. Exposition names depend on the registry and its naming convention, so the rules below assume the OpenTelemetry names after translation to Prometheus (dots become underscores, a unit suffix such as _seconds is appended). If your application exposes Micrometer timer names instead, substitute them.
The serving layer. vLLM exposes a Prometheus endpoint with vllm:-prefixed metrics. Serving is described in Serving at Scale. The names used below (vllm:time_to_first_token_seconds, vllm:time_per_output_token_seconds, vllm:e2e_request_latency_seconds, vllm:num_requests_waiting, vllm:kv_cache_usage_perc, vllm:prefix_cache_queries, vllm:prefix_cache_hits) were read from the vLLM metrics documentation; several have been renamed in past releases, and the name of the cache-usage gauge in particular differs between the older and newer engines. Check /metrics on your version.
The online evaluation job. A small worker samples finished conversations, asks a judge for a score and publishes it. The metric names (docsassistant_judge_score, docsassistant_cache_requests_total) are application-defined. The request path of the application publishes the retrieval, refusal and feedback counters from the golden-signals table in the same way; here is the Java side using the Micrometer API.

import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.DistributionSummary;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.stereotype.Component;

@Component
class QualityMetrics {

    private final DistributionSummary judgeScore;
    private final MeterRegistry registry;

    QualityMetrics(MeterRegistry registry) {
        this.registry = registry;
        // Judge scores are normalised to 0..1; buckets let Prometheus compute
        // "share of sampled answers scoring at least 0.7" without raw samples.
        this.judgeScore = DistributionSummary.builder("docsassistant.judge.score")
                .description("LLM-as-a-judge score on sampled production answers")
                .tag("rubric", "faithfulness")
                .serviceLevelObjectives(0.3, 0.5, 0.7, 0.9)
                .register(registry);
    }

    void recordScore(double score) {
        judgeScore.record(score);
    }

    void recordCache(String layer, boolean hit) {
        Counter.builder("docsassistant.cache.requests")
                .tag("layer", layer)                 // "semantic" | "prefix" | "provider"
                .tag("result", hit ? "hit" : "miss")
                .register(registry)
                .increment();
    }

    // Golden signals "Empty retrieval", "Refusal rate" and "Feedback rate".
    void recordRetrieval(int chunksReturned) {
        Counter.builder("docsassistant.retrieval.requests")
                .tag("result", chunksReturned == 0 ? "empty" : "found")
                .register(registry)
                .increment();
    }

    void recordAnswer(boolean refused) {
        Counter.builder("docsassistant.answers")
                .tag("outcome", refused ? "refused" : "answered")
                .register(registry)
                .increment();
    }

    void recordFeedback(boolean thumbsUp) {
        Counter.builder("docsassistant.feedback")
                .tag("vote", thumbsUp ? "up" : "down")
                .register(registry)
                .increment();
    }
}

Derived from the Micrometer documentation on distribution summaries and counters. Micrometer’s Prometheus registry renders docsassistant.judge.score as docsassistant_judge_score_bucket, _sum and _count, and appends _total to counters, so docsassistant.retrieval.requests becomes docsassistant_retrieval_requests_total. The code was not compiled here.

Prometheus recording rules

Dashboards and alerts that run histogram_quantile over raw buckets at every refresh are expensive and slow. A recording rule pre-computes the expression on the evaluation interval and stores the result as a new series, following the level:metric:operations naming convention. The rule file below records the latency percentiles, traffic, the error ratio over the four windows the burn-rate alerts need (5 minutes, 30 minutes, 1 hour, 6 hours), token throughput, cost, cache hit rate, quality ratio, and the empty-retrieval, refusal and feedback ratios for DocsAssistant. Rule-group syntax and evaluation behaviour are explained in Recording and Alerting Rules; the functions used are described in PromQL Functions.

# rules/docsassistant-recording.yml
groups:
  - name: docsassistant-latency
    interval: 30s
    rules:
      # p95 end-to-end latency of model calls made by the application
      - record: route_model:gen_ai_client_operation_duration_seconds:p95_5m
        expr: |
          histogram_quantile(
            0.95,
            sum by (le, gen_ai_request_model) (
              rate(gen_ai_client_operation_duration_seconds_bucket{service_name="docsassistant"}[5m])
            )
          )

      # p95 time to first token as measured by vLLM
      - record: model:vllm_ttft_seconds:p95_5m
        expr: |
          histogram_quantile(
            0.95,
            sum by (le, model_name) (rate(vllm:time_to_first_token_seconds_bucket[5m]))
          )

      # p95 time per output token as measured by vLLM
      - record: model:vllm_tpot_seconds:p95_5m
        expr: |
          histogram_quantile(
            0.95,
            sum by (le, model_name) (rate(vllm:time_per_output_token_seconds_bucket[5m]))
          )

  - name: docsassistant-traffic-errors
    interval: 30s
    rules:
      - record: service:docsassistant_requests:rate5m
        expr: sum(rate(gen_ai_client_operation_duration_seconds_count{service_name="docsassistant"}[5m]))

      - record: service:docsassistant_errors:rate5m
        expr: |
          sum(rate(gen_ai_client_operation_duration_seconds_count{service_name="docsassistant", error_type!=""}[5m]))

      # Error ratio over the four windows used by the burn-rate alerts.
      - record: service:docsassistant_error_ratio:rate5m
        expr: |
          service:docsassistant_errors:rate5m
            / clamp_min(service:docsassistant_requests:rate5m, 1e-9)

      - record: service:docsassistant_error_ratio:rate30m
        expr: |
          sum(rate(gen_ai_client_operation_duration_seconds_count{service_name="docsassistant", error_type!=""}[30m]))
            / clamp_min(sum(rate(gen_ai_client_operation_duration_seconds_count{service_name="docsassistant"}[30m])), 1e-9)

      - record: service:docsassistant_error_ratio:rate1h
        expr: |
          sum(rate(gen_ai_client_operation_duration_seconds_count{service_name="docsassistant", error_type!=""}[1h]))
            / clamp_min(sum(rate(gen_ai_client_operation_duration_seconds_count{service_name="docsassistant"}[1h])), 1e-9)

      - record: service:docsassistant_error_ratio:rate6h
        expr: |
          sum(rate(gen_ai_client_operation_duration_seconds_count{service_name="docsassistant", error_type!=""}[6h]))
            / clamp_min(sum(rate(gen_ai_client_operation_duration_seconds_count{service_name="docsassistant"}[6h])), 1e-9)

  - name: docsassistant-tokens-cost-cache-quality
    interval: 1m
    rules:
      - record: model_type:gen_ai_client_token_usage:rate5m
        expr: |
          sum by (gen_ai_request_model, gen_ai_token_type) (
            rate(gen_ai_client_token_usage_sum{service_name="docsassistant"}[5m])
          )

      # Cost per second = tokens per second * unit price. The price series is a
      # gauge published by the application from its own price table (no prices
      # are given on this page): docsassistant_token_price_per_token{model, token_type}.
      - record: model:docsassistant_cost_per_second:rate5m
        expr: |
          sum by (gen_ai_request_model) (
            model_type:gen_ai_client_token_usage:rate5m
              * on (gen_ai_request_model, gen_ai_token_type) group_left()
                max by (gen_ai_request_model, gen_ai_token_type) (docsassistant_token_price_per_token)
          )

      - record: layer:docsassistant_cache_hit_ratio:rate15m
        expr: |
          sum by (layer) (rate(docsassistant_cache_requests_total{result="hit"}[15m]))
            / sum by (layer) (rate(docsassistant_cache_requests_total[15m]))

      # share of sampled answers whose judge score is above 0.7; each side is
      # aggregated by rubric before subtracting, so several instances still
      # produce exactly one series per rubric on each side
      - record: rubric:docsassistant_judge_good_ratio:rate1h
        expr: |
          (
            sum by (rubric) (rate(docsassistant_judge_score_bucket{le="+Inf"}[1h]))
              - sum by (rubric) (rate(docsassistant_judge_score_bucket{le="0.7"}[1h]))
          )
            / clamp_min(sum by (rubric) (rate(docsassistant_judge_score_bucket{le="+Inf"}[1h])), 1e-9)

  - name: docsassistant-retrieval-refusal-feedback
    interval: 1m
    rules:
      # share of retrieval calls that returned no chunk
      - record: service:docsassistant_empty_retrieval_ratio:rate15m
        expr: |
          sum(rate(docsassistant_retrieval_requests_total{result="empty"}[15m]))
            / clamp_min(sum(rate(docsassistant_retrieval_requests_total[15m])), 1e-9)

      # share of answers in which the assistant refused
      - record: service:docsassistant_refusal_ratio:rate15m
        expr: |
          sum(rate(docsassistant_answers_total{outcome="refused"}[15m]))
            / clamp_min(sum(rate(docsassistant_answers_total[15m])), 1e-9)

      # share of answers that received any vote (feedback rate)
      - record: service:docsassistant_feedback_ratio:rate1h
        expr: |
          sum(rate(docsassistant_feedback_total[1h]))
            / clamp_min(sum(rate(docsassistant_answers_total[1h])), 1e-9)

      # share of votes that are thumbs-down
      - record: service:docsassistant_thumbs_down_ratio:rate1h
        expr: |
          sum(rate(docsassistant_feedback_total{vote="down"}[1h]))
            / clamp_min(sum(rate(docsassistant_feedback_total[1h])), 1e-9)

Derived from the Prometheus documentation on recording rules and the recording-rule naming practices, and from the OpenTelemetry GenAI metrics and vLLM metrics design pages. Not run. The label names after translation (gen_ai_request_model, gen_ai_token_type, error_type, service_name) depend on the OpenTelemetry-to-Prometheus mapping in use: with the Prometheus OTLP receiver, service.name becomes a resource attribute that is promoted to a label only if configured. The judge-ratio rule treats histogram buckets as cumulative counts, as Prometheus does: observations above 0.7 are the total minus the le="0.7" bucket. Each side is summed by rubric before the subtraction; subtracting raw bucket series would need exactly one series per rubric on each side and fails with duplicate-series errors as soon as two instances export the histogram. clamp_min keeps an idle window from dividing by zero.

Alert rules and burn rates

An alert should say "a user-visible promise is at risk", not "a number moved". Two kinds are useful here: symptom alerts on latency, errors and quality, which page someone, and cause alerts on saturation and cache behaviour, which open a ticket or appear on the dashboard. The most robust symptom alerts for an SLO are burn-rate alerts, from the Google SRE Workbook chapter on alerting on SLOs: they fire when the error budget (defined in the next section) is being consumed too quickly, using two windows so that a short spike does not page and a slow leak is not missed.

# rules/docsassistant-alerts.yml
groups:
  - name: docsassistant-slo-burn
    rules:
      # Availability SLO 99.5% -> error budget 0.5%. Burn rate 14.4 consumes
      # 2% of a 30-day budget in one hour. Both windows must agree.
      - alert: DocsAssistantErrorBudgetFastBurn
        expr: |
          service:docsassistant_error_ratio:rate1h > (14.4 * 0.005)
          and
          service:docsassistant_error_ratio:rate5m > (14.4 * 0.005)
        for: 2m
        labels:
          severity: page
          slo: availability
        annotations:
          summary: "DocsAssistant is burning its availability error budget quickly"
          runbook_url: "https://docs.example.internal/runbooks/docsassistant-availability"

      # Slow burn: burn rate 6 for 6 hours consumes 5% of a 30-day budget.
      - alert: DocsAssistantErrorBudgetSlowBurn
        expr: |
          service:docsassistant_error_ratio:rate6h > (6 * 0.005)
          and
          service:docsassistant_error_ratio:rate30m > (6 * 0.005)
        for: 15m
        labels:
          severity: ticket
          slo: availability
        annotations:
          summary: "DocsAssistant is slowly burning its availability error budget"

  - name: docsassistant-symptoms
    rules:
      - alert: DocsAssistantTTFTHigh
        expr: model:vllm_ttft_seconds:p95_5m > 2
        for: 10m
        labels:
          severity: page
          slo: latency
        annotations:
          summary: "p95 time to first token above 2 s on {{ $labels.model_name }}"
          description: "Check queue depth and KV-cache usage on the dashboard before restarting anything."

      - alert: DocsAssistantJudgeScoreDegraded
        expr: rubric:docsassistant_judge_good_ratio:rate1h{rubric="faithfulness"} < 0.85
        for: 2h
        labels:
          severity: ticket
          slo: quality
        annotations:
          summary: "Share of well-scored sampled answers fell below target"
          description: "Compare with the last prompt, model and index changes before blaming the judge."

  - name: docsassistant-causes
    rules:
      - alert: VllmQueueBuilding
        expr: avg_over_time(vllm:num_requests_waiting[5m]) > 10
        for: 10m
        labels:
          severity: ticket
        annotations:
          summary: "Requests are queuing on {{ $labels.instance }}"

      - alert: DocsAssistantCacheHitRatioDropped
        expr: |
          layer:docsassistant_cache_hit_ratio:rate15m
            < 0.5 * (layer:docsassistant_cache_hit_ratio:rate15m offset 1d)
        for: 30m
        labels:
          severity: ticket
        annotations:
          summary: "Cache hit ratio for layer {{ $labels.layer }} halved against yesterday"

Derived from the Prometheus alerting rules documentation and the Google SRE Workbook chapter Alerting on SLOs. Not run. The burn-rate alerts read the four error-ratio recording rules defined in the previous section (service:docsassistant_error_ratio:rate5m, :rate30m, :rate1h, :rate6h), and the burn-rate factors 14.4 (1 hour, 2% of the budget) and 6 (6 hours, 5% of the budget) are the page and ticket tiers of the SRE Workbook’s multiwindow, multi-burn-rate recommendation. The thresholds (2 seconds, 10 queued requests, 0.85, the halved cache ratio) are placeholders: derive real ones from your own SLO and baseline, and see Drift and Regression Monitoring for alerting on score drops against a moving reference rather than a fixed number.

Prometheus only evaluates the rules; deduplicating, grouping, routing, silencing and notifying belong to Alertmanager, covered in Alertmanager. A routing tree that sends severity: page to the on-call rotation and severity: ticket to a queue is enough to start. Grafana can evaluate alert rules too, which is convenient when the data source is not Prometheus; the trade-offs are in Grafana Alerting. Keep one evaluation engine per alert, otherwise the same condition notifies twice.

A compact Grafana dashboard

One screen should answer "is DocsAssistant healthy right now, and if not, which layer?". The dashboard below has six panels: request rate and error ratio, p95 latency, TTFT and TPOT, token throughput by type with cost, cache hit ratio with the quality ratio, and the empty-retrieval, refusal, feedback and thumbs-down ratios. It reads only the recording rules defined above, which keeps refreshes cheap. The JSON is the dashboard model that Grafana imports through Dashboards > New > Import, trimmed to the fields needed; Grafana fills defaults for the rest.

{
  "title": "DocsAssistant - LLM health",
  "uid": "docsassistant-llm-health",
  "schemaVersion": 39,
  "version": 1,
  "refresh": "1m",
  "time": { "from": "now-6h", "to": "now" },
  "tags": ["llmops", "docsassistant"],
  "templating": {
    "list": [
      {
        "name": "datasource",
        "type": "datasource",
        "query": "prometheus"
      }
    ]
  },
  "panels": [
    {
      "id": 1,
      "type": "timeseries",
      "title": "Traffic and error ratio",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "${datasource}" },
      "targets": [
        { "refId": "A", "legendFormat": "requests/s", "expr": "service:docsassistant_requests:rate5m" },
        { "refId": "B", "legendFormat": "error ratio", "expr": "service:docsassistant_error_ratio:rate5m" }
      ],
      "fieldConfig": {
        "defaults": { "unit": "short" },
        "overrides": [
          {
            "matcher": { "id": "byName", "options": "error ratio" },
            "properties": [
              { "id": "unit", "value": "percentunit" },
              { "id": "custom.axisPlacement", "value": "right" }
            ]
          }
        ]
      }
    },
    {
      "id": 2,
      "type": "timeseries",
      "title": "p95 model-call latency",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "${datasource}" },
      "targets": [
        { "refId": "A", "legendFormat": "{{gen_ai_request_model}}", "expr": "route_model:gen_ai_client_operation_duration_seconds:p95_5m" }
      ],
      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] }
    },
    {
      "id": 3,
      "type": "timeseries",
      "title": "TTFT and TPOT (p95, vLLM)",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
      "datasource": { "type": "prometheus", "uid": "${datasource}" },
      "targets": [
        { "refId": "A", "legendFormat": "TTFT {{model_name}}", "expr": "model:vllm_ttft_seconds:p95_5m" },
        { "refId": "B", "legendFormat": "TPOT {{model_name}}", "expr": "model:vllm_tpot_seconds:p95_5m" }
      ],
      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] }
    },
    {
      "id": 4,
      "type": "timeseries",
      "title": "Token throughput and cost rate",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
      "datasource": { "type": "prometheus", "uid": "${datasource}" },
      "targets": [
        { "refId": "A", "legendFormat": "{{gen_ai_request_model}} {{gen_ai_token_type}}", "expr": "model_type:gen_ai_client_token_usage:rate5m" },
        { "refId": "B", "legendFormat": "cost/s {{gen_ai_request_model}}", "expr": "model:docsassistant_cost_per_second:rate5m" }
      ],
      "fieldConfig": {
        "defaults": { "unit": "short" },
        "overrides": [
          {
            "matcher": { "id": "byFrameRefID", "options": "B" },
            "properties": [ { "id": "custom.axisPlacement", "value": "right" } ]
          }
        ]
      }
    },
    {
      "id": 5,
      "type": "stat",
      "title": "Cache hit ratio and good-answer ratio",
      "gridPos": { "h": 6, "w": 24, "x": 0, "y": 16 },
      "datasource": { "type": "prometheus", "uid": "${datasource}" },
      "targets": [
        { "refId": "A", "legendFormat": "cache {{layer}}", "expr": "layer:docsassistant_cache_hit_ratio:rate15m" },
        { "refId": "B", "legendFormat": "good answers {{rubric}}", "expr": "rubric:docsassistant_judge_good_ratio:rate1h" }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit",
          "min": 0,
          "max": 1,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "red", "value": null },
              { "color": "yellow", "value": 0.7 },
              { "color": "green", "value": 0.85 }
            ]
          }
        },
        "overrides": []
      }
    },
    {
      "id": 6,
      "type": "timeseries",
      "title": "Empty retrieval, refusals and feedback",
      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 22 },
      "datasource": { "type": "prometheus", "uid": "${datasource}" },
      "targets": [
        { "refId": "A", "legendFormat": "empty retrieval", "expr": "service:docsassistant_empty_retrieval_ratio:rate15m" },
        { "refId": "B", "legendFormat": "refusals", "expr": "service:docsassistant_refusal_ratio:rate15m" },
        { "refId": "C", "legendFormat": "answers with feedback", "expr": "service:docsassistant_feedback_ratio:rate1h" },
        { "refId": "D", "legendFormat": "thumbs-down share of votes", "expr": "service:docsassistant_thumbs_down_ratio:rate1h" }
      ],
      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1 }, "overrides": [] }
    }
  ]
}

Derived from the Grafana documentation on the dashboard JSON model and on importing dashboards. The JSON is syntactically valid but was not imported into Grafana 13.2.3, and the schema of the dashboard model has been evolving (Grafana also supports a newer resource-style format); if the import dialog rejects a field, export a simple dashboard from your own instance and compare. Panel design, variables, thresholds and provisioning dashboards from files are covered in Grafana Dashboards and Visualization.

Design choices worth keeping when you extend it: put the same time range on every panel so incidents can be correlated by eye; show the error budget remaining as a stat panel once the SLO rules exist; give the saturation panels (queue depth, KV-cache usage) their own row below the symptoms, because the order on the screen is the order of questions during an incident; and link panels to the trace backend with a data link carrying the model and time range, so that a spike leads straight to example traces.

SLOs for LLM features

A service level indicator (SLI) is a measured ratio of good events to valid events. A service level objective (SLO) is the target for that ratio over a window. The error budget is what the target allows to go wrong. For a target \(T\) (for example \(0.995\)) over a window of \(N\) events, the budget and the burn rate are

\[ B = (1 - T)\,N, \qquad \text{burn rate} = \frac{\text{observed error ratio}}{1 - T} \]

A burn rate of 1 spends exactly the whole budget by the end of the window; a burn rate of \(r\) exhausts it in \(W/r\) for a window of length \(W\). The fraction of the budget spent by an alert that fires after a time \(t\) at burn rate \(r\) is

\[ \frac{r \cdot t}{ W } \]

which is where the numbers in the alert rules come from: with \(W = 30\) days (720 hours), a burn rate of 14.4 for one hour spends \(14.4/720 = 2\%\) of the budget, and a burn rate of 6 for six hours spends \(6 \cdot 6/720 = 5\%\) of it; the SRE Workbook recommends picking the page and ticket thresholds from the fraction of budget you are willing to lose before someone is told. For an availability target of \(T = 0.995\), \(1 - T = 0.005\), which is the constant used in the YAML above.

DocsAssistant has three kinds of objective, and the third is the one that makes this an LLM page.

SLO	Good event	Valid event	Example target
Availability	A request that returned a non-error, non-truncated answer.	Any user request that reached the application (exclude requests rejected for bad input).	\(T = 0.995\) over 30 days.
Latency	A request whose TTFT is below a threshold and whose streaming rate is above a floor, or whose total time is below a threshold for non-streamed calls.	Streamed or non-streamed user requests.	95% of requests under the TTFT threshold over 30 days.
Quality	A sampled answer whose judge score is at least the pass mark on the faithfulness rubric.	Answers that were sampled and judged.	A pass ratio floor over 7 days.

SLO

Good event

Valid event

Example target

Availability

A request that returned a non-error, non-truncated answer.

Any user request that reached the application (exclude requests rejected for bad input).

\(T = 0.995\) over 30 days.

Latency

A request whose TTFT is below a threshold and whose streaming rate is above a floor, or whose total time is below a threshold for non-streamed calls.

Streamed or non-streamed user requests.

95% of requests under the TTFT threshold over 30 days.

Quality

A sampled answer whose judge score is at least the pass mark on the faithfulness rubric.

Answers that were sampled and judged.

A pass ratio floor over 7 days.

For latency, a histogram bucket boundary must coincide with the threshold, because the SLI is read as "observations in the bucket \(\le\) threshold, divided by all observations"; pick the threshold first and then configure the histogram buckets in the serving layer or the Micrometer registry to include it. This is why the example application above declares explicit serviceLevelObjectives boundaries on the score histogram.

The quality SLO deserves a longer comment. A judge score is an estimate, so the SLI inherits the judge’s bias and the sampling error. Quality SLIs should therefore use a window long enough to contain a few hundred judged answers, be calibrated against human labels periodically (see LLM-as-a-Judge), and page only through the slow-burn route: a quality ticket the next morning is the right urgency for most degradations that are not accompanied by errors. Because the judge itself can drift when its model is updated, compare the quality SLI with the thumbs-down rate before concluding that quality changed. When the budget is spent, the response is policy, not heroics: freeze prompt and model changes except fixes, as described in Deployment and Release Patterns.

Operating the dashboards and alerts

Start from symptoms. Every page-severity alert should map to an SLO. If an alert has no SLO behind it, make it a ticket or remove it.
Write a runbook before enabling a page. The runbook_url annotation above is mandatory in practice: the first questions (which model, which route, did a prompt or index change ship, is the queue growing, is the provider degraded) should be answered there. The runbook structure is part of Operational Governance.
Keep rules in version control and test them. promtool check rules validates syntax and promtool test rules runs unit tests against synthetic series; both are described in the Prometheus unit-testing documentation. Neither was run for this page.
Watch cardinality of what you add. A new label such as tenant multiplies series by the number of tenants; measure with count({name=~"gen_ai_.*"}) before and after.
Retention. Raw 15-second samples are not needed for 30-day SLO windows when recording rules exist; the recording rules are what an SLO review will read.
Metrics and traces together. An alert tells you the window; the trace backend tells you the requests. Make sure every alert annotation carries a link template to the trace search for that model and time range.
Cost alerts belong here and in cost management. A sudden rise of the cost rate can be an attack or a loop. The budget and attribution side is covered in Cost Management; abuse scenarios are part of LLMSecOps in the AI Security & Responsible AI sub-section.