Recording & alerting rules

This section documents the current Prometheus 3.x line (latest 3.14.0) as published at the Prometheus documentation and the current Grafana 13.x line (latest 13.2.1) as published at the Grafana documentation, which are the references these pages are written and verified against. No specific patch version is pinned. Some capabilities (Grafana Enterprise-only features, Grafana Cloud-hosted services, and long-term-storage systems like Thanos/Cortex/Mimir) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, since both projects iterate quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

A rule is a named PromQL expression, evaluated on evaluation_interval and grouped into rule groups inside YAML files referenced from prometheus.yml’s `rule_files. Prometheus supports two kinds: recording rules, which save an expression’s result as a new time series, and alerting rules, which turn an expression’s result into a firing or resolved alert. This page covers the naming convention and use cases for recording rules, the for/labels/annotations fields and templating syntax for alerting rules, and validating both with promtool. A first alerting rule already appeared in Getting Started; this page covers writing rules well, not just writing one.

Rule groups

Both rule types live in the same file structure: one or more named groups, each holding an ordered list of rules evaluated sequentially on that group’s own interval (defaulting to the global evaluation_interval if unset). Rules within a group can see the results of earlier rules in the same evaluation cycle, which is what lets an alerting rule reference a recording rule defined above it in the same group.

groups:
  - name: orders-api.rules
    interval: 30s          # optional: overrides evaluation_interval for this group only
    rules:
      - record: ...        # a recording rule
      - alert: ...          # an alerting rule
# https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/

Rules in different groups run concurrently and independently, which is why a rule that depends on another’s output must sit in the same group, after it. Split unrelated rules into separate groups instead of one large one, so a slow group’s evaluation time does not delay unrelated rules.

Recording rules

A recording rule precomputes an expression and stores it as a new metric, on every evaluation, just like a regular scrape would:

groups:
  - name: orders-api.rules
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
# https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/

job:http_requests:rate5m is itself queryable, exactly like any scraped metric — job:http_requests:rate5m{job="orders-api"} returns the precomputed value instantly instead of re-running the sum(rate(…​)) aggregation on every query. See Recording rules for the full rule-group syntax, including record, expr, and the optional labels a recording rule may attach to every series it produces.

The level:metric:operations naming convention

Recording rule names should follow level:metric:operations, so a glance at the name tells you what aggregation level it is at and what was done to reach it, without reading the expression:

  • level — the aggregation labels remaining after the rule runs (job, instance, job_instance, or a colon-free base metric name for no aggregation).

  • metric — the original metric name, with its _total/unit suffix generally kept.

  • operations — the PromQL operations applied, outermost first, separated by underscores (for example rate5m for a five-minute rate(), or irate for irate()).

groups:
  - name: orders-api.rules
    rules:
      # level: job -- aggregated down to just the job label
      # metric: http_requests_total
      # operations: rate applied over a 5m window, then summed
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      # level: job_status -- job and status both survive the aggregation
      - record: job_status:http_requests:rate5m
        expr: sum by (job, status) (rate(http_requests_total[5m]))

      # level: instance -- no aggregation, just a renamed rate() over one series per instance
      - record: instance:node_cpu:rate5m
        expr: rate(node_cpu_seconds_total{mode!="idle"}[5m])

See the naming convention in Recording rules for the authoritative definition of each segment.

Use cases

Recording rules exist to trade evaluation cost, paid once per evaluation_interval, against query cost, paid on every dashboard load or every downstream rule evaluation:

  • Dashboards queried often. A Grafana panel re-runs its query every time someone opens the dashboard or it auto-refreshes; if the underlying expression is an expensive sum(rate(…​)) over high-cardinality series, precomputing it once as a recording rule turns every panel load into a cheap lookup instead of a repeated aggregation. See Grafana: Dashboards & Visualization for wiring recorded series into panels.

  • Expressions reused by other rules. When several alerting rules threshold-check the same aggregation (error rate by job, saturation by instance), computing it once as a recording rule and referencing it from every alert avoids repeating — and re-evaluating — the same subquery multiple times per cycle.

  • Long-window or high-cardinality aggregations. A rate() over a long range, or a sum by that collapses a very high-cardinality label, can be slow enough at query time to time out an ad-hoc request; the same computation as a recording rule pays that cost once on the server’s own schedule instead of once per caller.

Recording rules do not reduce raw sample storage or cardinality on their own — the source series are still scraped and retained — they only avoid re-deriving an aggregate from them repeatedly. Reducing retained cardinality itself is covered in Storage, Federation & Remote Write.

Alerting rules

An alerting rule evaluates a boolean-producing expression on every cycle; whenever the expression returns any series, an alert instance is created per returned series and moves through inactive → pending → firing (or back to inactive once the expression stops matching):

groups:
  - name: orders-api.alerts
    rules:
      - alert: HighErrorRate
        expr: job_status:http_requests:rate5m{status=~"5.."} / on (job) job:http_requests:rate5m > 0.05
        for: 10m
        labels:
          severity: page
          team: orders
        annotations:
          summary: "{{ $labels.job }} error rate above 5%"
          description: >-
            {{ $labels.job }} has a {{ $value | humanizePercentage }} error rate over the
            last 5 minutes (threshold 5%).
# https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/

This example also shows recording rules feeding an alert directly, instead of re-running both rate() aggregations inline — exactly the "reused by other rules" case above.

for: avoiding flapping

for: 10m requires the expression to keep matching a given series continuously across every evaluation for that whole duration before the alert leaves pending and becomes firing; a single evaluation where the condition briefly stops holding resets that series back to inactive. This is what keeps a noisy metric that crosses a threshold for one scrape from paging anyone — without for (or with it set to 0s), the alert fires the instant the expression matches, which is appropriate only for conditions that must never wait, such as up == 0 on a critical target. Longer for durations trade faster detection for fewer transient false pages; Status > Alerts in the expression browser shows each alert’s current state and how long it has been pending. See Alerting rules for the full pending/firing/inactive state machine.

labels: routing metadata

labels on an alerting rule are attached to every alert instance it produces, merged with the labels already on the series the expression returned. They exist to be matched on downstream, not to be read by a human — severity, team, and similar dimensions are what Alertmanager's routing tree and inhibition rules match against to decide who gets paged and through which receiver. A labels value can itself use the {{ }} templating described below, though this is far less common than templating annotations.

annotations: human-readable text

annotations, by contrast, carry the free-text a human actually reads — conventionally summary (a one-line description) and description (more detail) — and are never used for Alertmanager routing. Because annotation values are templated per alert instance, they can embed the value and labels that triggered that specific instance rather than a generic message.

Templating: {{ $value }} and {{ $labels.foo }}

Annotation (and label) values are Go templates evaluated once per alert instance, with two variables in scope: $value, the numeric result of expr for that series, and $labels, a map of that series’s label set ($labels.instance, $labels.job, and so on). Template functions like humanize, humanizePercentage, humanizeDuration, and printf format $value for a human instead of dumping a raw float:

annotations:
  summary: "High memory usage on {{ $labels.instance }}"
  description: >-
    {{ $labels.instance }} is using {{ $value | humanizePercentage }} of available memory
    (job {{ $labels.job }}).

See Template reference for the complete set of available functions and variables, including $labels access via bracket notation ({{ index $labels "label.with.dots" }}) for label names that are not valid Go identifiers.

Testing rules with promtool

promtool, bundled alongside the prometheus binary, validates rule files two ways: a static syntax check, and a full unit test against synthetic input series.

promtool check rules

Checks a rules file parses, every expr is valid PromQL, and the file follows a handful of style rules (such as the recording-rule naming convention above), without evaluating anything against real data:

promtool check rules orders_api_rules.yml
# https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/

promtool test rules

Runs one or more rule files against a synthetic time series you define, at simulated points in time, and asserts the recorded series and firing alerts that should exist at each point — catching a wrong threshold, a missing for, or an inverted comparison before the rule ever reaches production. Given this rules file:

# orders_api_rules.yml
groups:
  - name: orders-api.alerts
    rules:
      - alert: HighErrorRate
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
              / on (job)
              sum by (job) (rate(http_requests_total[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} error rate above 5%"

a unit-test file feeds it synthetic input series and the alerts expected at given evaluation times:

# orders_api_rules_test.yml
rule_files:
  - orders_api_rules.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="orders-api", status="500"}'
        values: "0+6x20"    # 6 req/s of 500s, ramping for 20 minutes
      - series: 'http_requests_total{job="orders-api", status="200"}'
        values: "0+94x20"   # 94 req/s of 200s over the same window

    alert_rule_test:
      # Before 10 minutes of sustained breach, the alert must still be pending, not firing.
      - eval_time: 9m
        alertname: HighErrorRate
        exp_alerts: []

      # After 10 minutes of continuous breach, it must be firing with the expected labels/annotations.
      - eval_time: 15m
        alertname: HighErrorRate
        exp_alerts:
          - exp_labels:
              job: orders-api
              severity: page
            exp_annotations:
              summary: "orders-api error rate above 5%"
# https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/
promtool test rules orders_api_rules_test.yml

input_series.values uses promtool’s compact series-value syntax (`0+6x20 means "start at 0, add 6 at each of the next 20 steps of interval`"); a recording rule is asserted the same way via `promql_expr_test instead of alert_rule_test, comparing the recorded series' value at a given eval_time against an exp_samples list. See Unit testing rules for the full test-file schema, including multiple input_series blocks per group and testing several rule files together. Run both checks in CI before a rules file is deployed — check rules catches syntax errors immediately, test rules catches logic errors that only show up once real data flows through the expression.

Continue with Alertmanager for what happens to a firing alert once Prometheus has produced it — grouping, routing, silencing, and delivery to a receiver — or back up to PromQL: Functions for the aggregation and rate functions that recording-rule expressions like the ones above are built from.