Grafana alerting

This section documents the current Prometheus 3.x line (latest 3.14.0) as published at the Prometheus documentation and the current Grafana 13.x line (latest 13.2.1) as published at the Grafana documentation, which are the references these pages are written and verified against. No specific patch version is pinned. Some capabilities (Grafana Enterprise-only features, Grafana Cloud-hosted services, and long-term-storage systems like Thanos/Cortex/Mimir) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, since both projects iterate quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

Alerting exists in two places in this ecosystem and they are not the same feature wearing different skins. Prometheus/Alertmanager-native alerting — covered in Recording & Alerting Rules and Alertmanager — evaluates alert: rules inside Prometheus itself and routes the result through a standalone Alertmanager. Grafana-managed alerting evaluates its own alert rules inside Grafana, against any configured data source (Prometheus, Loki, an SQL database, CloudWatch, and more, mixed in the same rule if needed), and routes the result through Grafana’s own notification policies and contact points — which either deliver directly or forward to an external Alertmanager. This page assumes Grafana: Dashboards & Visualization for how Grafana is connected to a data source in the first place, and covers when to reach for each alerting model, Grafana’s query+condition rule shape, and its notification-policy/contact-point routing.

Two alerting models, one Alertmanager-shaped pipeline

Both models end at the same place conceptually — a firing alert grouped, routed, and delivered to a receiver — but they start evaluation in different processes:

flowchart LR subgraph Prometheus-native PR[Prometheus rule evaluator] -->|alert: rules, one data source| AM[Alertmanager] AM --> N1[Notification via route + receivers] end subgraph Grafana-managed GR[Grafana alert rule evaluator] -->|query + condition, any data source| GNP[Grafana notification policies] GNP -->|internal Alertmanager, default| N2[Notification via contact points] GNP -.->|or forward to an external Alertmanager| AM end

Grafana ships with its own embedded, Prometheus-compatible Alertmanager (part of Grafana’s unified alerting system) that notification policies route through by default; a data source of type Alertmanager can be pointed at that internal instance, at a standalone Prometheus Alertmanager, or at a Mimir/Cortex Alertmanager instead, and Grafana’s routing then targets whichever is selected as the default data source for alerting — so "Grafana-managed alerting" and "using Grafana’s own internal Alertmanager" are independently choosable, not the same decision. See Grafana Alerting for the full architecture.

When to reach for each

  • Stay Prometheus/Alertmanager-native when every alert condition is PromQL over Prometheus data, the team already owns rule_files and an Alertmanager deployment, and alerting needs to keep working even if Grafana itself is down — Prometheus evaluates and Alertmanager routes independently of any UI. promtool test rules (see Recording & Alerting Rules) also gives rule files a unit-testable, code-reviewable artifact that a Grafana-managed rule, edited through the UI or provisioned as YAML, does not get for free.

  • Reach for Grafana-managed alerting for a single pane of glass across heterogeneous data sources — a rule that alerts on a Loki log-count query, a CloudWatch metric, or a SQL row count needs Grafana, since none of those has a Prometheus-style rule evaluator of its own. It is also the natural choice when alert rules should live next to the dashboards that visualize the same data, or when the team has no separate Alertmanager to operate and wants routing configured entirely inside Grafana.

  • Mixed estates commonly do both: Prometheus keeps evaluating its own high-volume, PromQL-native rules against its own Alertmanager, while Grafana-managed rules cover the non-Prometheus data sources — and, if a single unified routing tree and on-call schedule is wanted, Grafana’s notification policies are pointed at the same external Alertmanager Prometheus already uses, rather than running two independent notification pipelines.

Grafana-managed alert rules: query + condition

A Grafana-managed alert rule is one or more queries (A, B, …​) against any data source, reduced by an expression, and compared against a threshold — the condition — on an evaluation interval set per evaluation group, analogous to a Prometheus rule group’s interval:

# provisioning/alerting/rules.yaml -- file-based provisioning of Grafana-managed alert rules
apiVersion: 1

groups:
  - orgId: 1
    name: orders-api-alerts
    folder: Orders API
    interval: 1m
    rules:
      - uid: orders-api-high-error-rate
        title: High error rate (orders-api)
        condition: C
        data:
          - refId: A
            datasourceUid: prometheus-uid
            relativeTimeRange: { from: 300, to: 0 }
            model:
              expr: job_status:http_requests:rate5m{job="orders-api",status=~"5.."}
              instant: true
          - refId: B
            datasourceUid: prometheus-uid
            relativeTimeRange: { from: 300, to: 0 }
            model:
              expr: job:http_requests:rate5m{job="orders-api"}
              instant: true
          - refId: C
            datasourceUid: __expr__
            model:
              type: math
              expression: "$A / $B > 0.05"
        for: 10m
        labels:
          severity: page
          team: orders
        annotations:
          summary: "orders-api error rate above 5%"
# https://grafana.com/docs/grafana/latest/alerting/[Grafana Alerting]

data mirrors what the alert-rule editor’s Query and condition step builds: one or more data-source queries (A, B), plus a built-in expr "expression" data source (math, reduce, resample, or threshold) that combines their results into the boolean condition refId. for, labels, and annotations mean the same thing here as on a Prometheus alerting rule — the pending-before-firing delay, routing metadata matched by notification policies below, and human-readable text — which is deliberate: a team moving a rule from Prometheus to Grafana-managed keeps the same mental model, only the query layer changes. See Grafana Alerting for the rule editor, the expression types, and the full provisioning-file schema.

Notification policies: Grafana’s routing tree

A notification policy tree is Grafana’s equivalent of Alertmanager’s route block from Alertmanager: a default policy plus nested child policies, each matching on label selectors and, on a match, either handling the alert with its own contact point and grouping/timing settings or continuing to evaluate its children.

# provisioning/alerting/notification-policies.yaml
apiVersion: 1

policies:
  - orgId: 1
    receiver: default-slack           # default contact point, used when nothing below matches
    group_by: ["alertname"]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: orders-team-pagerduty
        object_matchers:
          - ["team", "=", "orders"]
          - ["severity", "=", "page"]
        continue: false                # stop here; do not also evaluate sibling/child routes
        group_by: ["alertname", "job"]
        repeat_interval: 1h
# https://grafana.com/docs/grafana/latest/alerting/[Grafana Alerting]

object_matchers, group_by, group_wait/group_interval/repeat_interval, and continue are the same grouping and timing knobs Alertmanager’s own route block uses — see Alertmanager for how each behaves — because Grafana’s internal Alertmanager implementation is an Alertmanager. The difference is only where the tree is authored and stored: a Grafana-managed alert’s labels (from the rule above) are what a notification policy’s object_matchers match against, exactly as an Alertmanager route matches a Prometheus alerting rule’s labels.

Contact points: Grafana’s receivers

A contact point is Grafana’s equivalent of an Alertmanager receiver: a named delivery target — email, Slack, PagerDuty, webhook, OpsGenie, and dozens more integrations — that a notification policy points to by name. A contact point can hold several integrations, all notified when it fires:

# provisioning/alerting/contact-points.yaml
apiVersion: 1

contactPoints:
  - orgId: 1
    name: orders-team-pagerduty
    receivers:
      - uid: orders-pagerduty
        type: pagerduty
        settings:
          integrationKey: ${PAGERDUTY_INTEGRATION_KEY}
          severity: critical
      - uid: orders-slack-backup
        type: slack
        settings:
          url: ${SLACK_WEBHOOK_URL}
          title: '{{ template "default.title" . }}'
# https://grafana.com/docs/grafana/latest/alerting/[Grafana Alerting]

Contact-point notification templates use the same underlying Go templating as Alertmanager’s templates, so {{ .Labels.job }}/{{ .Annotations.summary }}-style expressions carry over from an Alertmanager configuration with only minor adjustment. Provisioning both files above (via file-based provisioning, Terraform, or the HTTP alerting-provisioning API) keeps notification policies and contact points in version control the same way alertmanager.yml is, rather than left as UI-only edits.

Forwarding to an external Alertmanager instead

Because Grafana’s routing is itself Alertmanager-shaped, switching where it delivers is a data-source change, not a rule or policy rewrite: add an Alertmanager-type data source pointed at the external instance, set it as the alerting default, and Grafana’s notification policies and contact points stop applying — the external Alertmanager’s own route/receivers configuration takes over, exactly as if the alerts had come from Prometheus directly. This is the common shape for a mixed estate: Grafana keeps evaluating the data sources Prometheus cannot, but every alert — Grafana-managed or Prometheus-native — is routed and silenced in one place. See Grafana Alerting for configuring the Alertmanager choice per organization, and Alertmanager for the receiver/route configuration an external instance still expects.

Continue with Security & Production Operations, or back up to Grafana: Dashboards & Visualization for the panels these alert rules typically sit alongside.