Grafana alerting
|
This section documents the current Prometheus 3.x line (latest 3.14.0) as published at the Prometheus documentation and the current Grafana 13.x line (latest 13.2.1) as published at the Grafana documentation, which are the references these pages are written and verified against. No specific patch version is pinned. Some capabilities (Grafana Enterprise-only features, Grafana Cloud-hosted services, and long-term-storage systems like Thanos/Cortex/Mimir) are linked, not documented in depth. This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, since both projects iterate quickly. This section’s bibliography lists the reference material consulted while preparing these pages. |
Alerting exists in two places in this ecosystem and they are not the same feature wearing different
skins. Prometheus/Alertmanager-native alerting — covered in
Recording & Alerting Rules and
Alertmanager — evaluates alert: rules inside Prometheus
itself and routes the result through a standalone Alertmanager. Grafana-managed alerting evaluates
its own alert rules inside Grafana, against any configured data source (Prometheus, Loki, an SQL
database, CloudWatch, and more, mixed in the same rule if needed), and routes the result through
Grafana’s own notification policies and contact points — which either deliver directly or forward to
an external Alertmanager. This page assumes
Grafana: Dashboards & Visualization
for how Grafana is connected to a data source in the first place, and covers when to reach for each
alerting model, Grafana’s query+condition rule shape, and its notification-policy/contact-point
routing.
Two alerting models, one Alertmanager-shaped pipeline
Both models end at the same place conceptually — a firing alert grouped, routed, and delivered to a receiver — but they start evaluation in different processes:
Grafana ships with its own embedded, Prometheus-compatible Alertmanager (part of Grafana’s unified
alerting system) that notification policies route through by default; a data source of type
Alertmanager can be pointed at that internal instance, at a standalone Prometheus Alertmanager, or at
a Mimir/Cortex Alertmanager instead, and Grafana’s routing then targets whichever is selected as the
default data source for alerting — so "Grafana-managed alerting" and "using Grafana’s own internal
Alertmanager" are independently choosable, not the same decision. See
Grafana Alerting for the full architecture.
When to reach for each
-
Stay Prometheus/Alertmanager-native when every alert condition is PromQL over Prometheus data, the team already owns
rule_filesand an Alertmanager deployment, and alerting needs to keep working even if Grafana itself is down — Prometheus evaluates and Alertmanager routes independently of any UI.promtool test rules(see Recording & Alerting Rules) also gives rule files a unit-testable, code-reviewable artifact that a Grafana-managed rule, edited through the UI or provisioned as YAML, does not get for free. -
Reach for Grafana-managed alerting for a single pane of glass across heterogeneous data sources — a rule that alerts on a Loki log-count query, a CloudWatch metric, or a SQL row count needs Grafana, since none of those has a Prometheus-style rule evaluator of its own. It is also the natural choice when alert rules should live next to the dashboards that visualize the same data, or when the team has no separate Alertmanager to operate and wants routing configured entirely inside Grafana.
-
Mixed estates commonly do both: Prometheus keeps evaluating its own high-volume, PromQL-native rules against its own Alertmanager, while Grafana-managed rules cover the non-Prometheus data sources — and, if a single unified routing tree and on-call schedule is wanted, Grafana’s notification policies are pointed at the same external Alertmanager Prometheus already uses, rather than running two independent notification pipelines.
Grafana-managed alert rules: query + condition
A Grafana-managed alert rule is one or more queries (A, B, …) against any data source, reduced
by an expression, and compared against a threshold — the condition — on an evaluation interval set
per evaluation group, analogous to a Prometheus rule group’s interval:
# provisioning/alerting/rules.yaml -- file-based provisioning of Grafana-managed alert rules
apiVersion: 1
groups:
- orgId: 1
name: orders-api-alerts
folder: Orders API
interval: 1m
rules:
- uid: orders-api-high-error-rate
title: High error rate (orders-api)
condition: C
data:
- refId: A
datasourceUid: prometheus-uid
relativeTimeRange: { from: 300, to: 0 }
model:
expr: job_status:http_requests:rate5m{job="orders-api",status=~"5.."}
instant: true
- refId: B
datasourceUid: prometheus-uid
relativeTimeRange: { from: 300, to: 0 }
model:
expr: job:http_requests:rate5m{job="orders-api"}
instant: true
- refId: C
datasourceUid: __expr__
model:
type: math
expression: "$A / $B > 0.05"
for: 10m
labels:
severity: page
team: orders
annotations:
summary: "orders-api error rate above 5%"
# https://grafana.com/docs/grafana/latest/alerting/[Grafana Alerting]
data mirrors what the alert-rule editor’s Query and condition step builds: one or more data-source
queries (A, B), plus a built-in expr "expression" data source (math, reduce, resample,
or threshold) that combines their results into the boolean condition refId. for, labels, and
annotations mean the same thing here as on a Prometheus alerting rule — the pending-before-firing
delay, routing metadata matched by notification policies below, and human-readable text — which is
deliberate: a team moving a rule from Prometheus to Grafana-managed keeps the same mental model, only
the query layer changes. See
Grafana Alerting for the rule editor, the
expression types, and the full provisioning-file schema.
Notification policies: Grafana’s routing tree
A notification policy tree is Grafana’s equivalent of Alertmanager’s route block from
Alertmanager: a default policy plus nested child policies,
each matching on label selectors and, on a match, either handling the alert with its own contact point
and grouping/timing settings or continuing to evaluate its children.
# provisioning/alerting/notification-policies.yaml
apiVersion: 1
policies:
- orgId: 1
receiver: default-slack # default contact point, used when nothing below matches
group_by: ["alertname"]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- receiver: orders-team-pagerduty
object_matchers:
- ["team", "=", "orders"]
- ["severity", "=", "page"]
continue: false # stop here; do not also evaluate sibling/child routes
group_by: ["alertname", "job"]
repeat_interval: 1h
# https://grafana.com/docs/grafana/latest/alerting/[Grafana Alerting]
object_matchers, group_by, group_wait/group_interval/repeat_interval, and continue are the
same grouping and timing knobs Alertmanager’s own route block uses — see
Alertmanager for how each behaves — because Grafana’s
internal Alertmanager implementation is an Alertmanager. The difference is only where the tree is
authored and stored: a Grafana-managed alert’s labels (from the rule above) are what a notification
policy’s object_matchers match against, exactly as an Alertmanager route matches a Prometheus
alerting rule’s labels.
Contact points: Grafana’s receivers
A contact point is Grafana’s equivalent of an Alertmanager receiver: a named delivery target — email, Slack, PagerDuty, webhook, OpsGenie, and dozens more integrations — that a notification policy
points to by name. A contact point can hold several integrations, all notified when it fires:
# provisioning/alerting/contact-points.yaml
apiVersion: 1
contactPoints:
- orgId: 1
name: orders-team-pagerduty
receivers:
- uid: orders-pagerduty
type: pagerduty
settings:
integrationKey: ${PAGERDUTY_INTEGRATION_KEY}
severity: critical
- uid: orders-slack-backup
type: slack
settings:
url: ${SLACK_WEBHOOK_URL}
title: '{{ template "default.title" . }}'
# https://grafana.com/docs/grafana/latest/alerting/[Grafana Alerting]
Contact-point notification templates use the same underlying Go templating as Alertmanager’s
templates, so {{ .Labels.job }}/{{ .Annotations.summary }}-style expressions carry over from an
Alertmanager configuration with only minor adjustment. Provisioning both files above (via file-based
provisioning, Terraform, or the HTTP alerting-provisioning API) keeps notification policies and contact
points in version control the same way alertmanager.yml is, rather than left as UI-only edits.
Forwarding to an external Alertmanager instead
Because Grafana’s routing is itself Alertmanager-shaped, switching where it delivers is a data-source
change, not a rule or policy rewrite: add an Alertmanager-type data source pointed at the external
instance, set it as the alerting default, and Grafana’s notification policies and contact points stop
applying — the external Alertmanager’s own route/receivers configuration takes over, exactly as if
the alerts had come from Prometheus directly. This is the common shape for a mixed estate: Grafana
keeps evaluating the data sources Prometheus cannot, but every alert — Grafana-managed or
Prometheus-native — is routed and silenced in one place. See
Grafana Alerting for configuring the Alertmanager
choice per organization, and Alertmanager for the
receiver/route configuration an external instance still expects.
Continue with Security & Production Operations, or back up to Grafana: Dashboards & Visualization for the panels these alert rules typically sit alongside.