Prometheus & Monitoring

This section documents the current Prometheus 3.x line (latest 3.14.0) as published at the Prometheus documentation and the current Grafana 13.x line (latest 13.2.1) as published at the Grafana documentation, which are the references these pages are written and verified against. No specific patch version is pinned. Some capabilities (Grafana Enterprise-only features, Grafana Cloud-hosted services, and long-term-storage systems like Thanos/Cortex/Mimir) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, since both projects iterate quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

Prometheus is an open-source, metrics-based monitoring and alerting toolkit: it pulls numeric time series from instrumented targets on a schedule, stores them in its own local time-series database, evaluates PromQL expressions against that data for dashboards and alerting rules, and hands firing alerts to the Alertmanager for deduplication, grouping and routing. That pull model, and its focus on numeric metrics rather than logs, distributed traces or profiles, is what sets it apart from adjacent observability signals: metrics answer "how much/how many, over time" cheaply at scale, where logs and traces answer "what exactly happened, in this one request". Grafana complements it as the visualization and dashboarding layer, and can also evaluate its own alerting rules against the same data.

If you are new to this section, read Getting Started first to get a Prometheus instance scraping real targets, then Architecture & Ecosystem for how the pieces fit together, and Data Model & Metric Types for the vocabulary (time series, labels, the four metric types) every later page assumes. Everything after that builds on those foundations.

For the application-side view of instrumenting a Spring Boot service and wiring it to Prometheus and Grafana, see Metrics & Observability; this section is the deeper, tool-focused reference that page links out to. For dashboards and alerts on LLM applications (token usage, latency, cost, quality scores), see LLMOps: Metrics, Dashboards & Alerting.

What’s covered

Getting started

Getting Started — installing and running Prometheus (binary tarball, Docker), a first prometheus.yml, the expression browser, scraping node_exporter, and writing a first alerting rule.
Architecture & Ecosystem — the server, client libraries, exporters, service discovery, the Pushgateway, Alertmanager, remote storage, and why Prometheus pulls rather than pushes.

Data model & instrumentation

Data Model & Metric Types — time series, labels, jobs and instances, the four metric types (Counter, Gauge, Histogram, Summary), and native histograms.
Instrumenting Applications — client libraries, the exposition format and OpenMetrics, content negotiation, and metric/label naming conventions.
Exporters & the Pushgateway — node_exporter collectors, blackbox_exporter probing, writing a custom exporter, and the Pushgateway (and when not to use it).

Service discovery & Kubernetes

Service Discovery & Relabeling — static, file, HTTP, Consul, EC2 and Kubernetes service discovery, relabel_configs vs. metric_relabel_configs, and honor_labels.
Containers & Kubernetes Monitoring — cAdvisor container metrics, Kubernetes service-discovery roles, and kube-state-metrics.

PromQL

PromQL: Basics & Selectors — instant vs. range vectors, label matchers, subqueries, offset, the @ modifier, and the HTTP query API.
PromQL: Operators & Aggregation — arithmetic/comparison/logical operators, vector matching (one-to-one, many-to-one, group_left/ group_right), and aggregation operators.
PromQL: Functions — rate/irate/increase, histogram_quantile, predict_linear, label_replace/label_join, and time/date functions.

Rules & alerting

Recording & Alerting Rules — recording rule naming and use cases, alerting rule for/labels/annotations/templates, and testing rules with promtool.
Alertmanager — the notification pipeline, the routing tree, grouping and throttling, receivers and templates, inhibition, silences, and HA clustering.

Storage & operations

Storage, Federation & Remote Write — the local TSDB, federation, the remote_read/remote_write APIs, and long-term storage systems (Thanos, Cortex, Mimir).
Security & Production Operations — server-side TLS and basic auth, capacity planning, sharding, HA, meta-monitoring, promtool, and the management API.

Grafana

Grafana: Dashboards & Visualization — installing Grafana, adding the Prometheus data source, dashboards and panel types, and template variables.
Grafana Alerting — Grafana-managed alerting vs. Prometheus/Alertmanager alerting, and notification policies and contact points.

Cheat sheet

Cheat Sheet (PDF) — a single-page, printable summary of everything in this section, with a downloadable PDF.

Bibliography

Overview and Overview — Architecture, and First steps — Prometheus & Monitoring, Installing & running Prometheus, Prometheus architecture & ecosystem.
Getting started and Installation, the downloads page — Installing & running Prometheus.
Data model, Metric types, Jobs & instances and Native histograms — Data model & metric types.
Client libraries, Writing client libraries, Exposition formats, Content negotiation, Metric & label naming, Instrumentation and Histograms & summaries, plus OpenMetrics — Instrumenting applications.
Exporters & integrations, Writing exporters and Pushing metrics — Exporters & the Pushgateway.
Configuration and HTTP-based service discovery — Service discovery & relabeling.
Monitoring Docker container metrics using cAdvisor — Containers & Kubernetes monitoring.
Querying basics and the HTTP API — PromQL: Basics & selectors.
Operators — PromQL: Operators & Aggregation.
Functions and Query examples — PromQL: Functions.
Recording rules, Alerting rules, Template reference and Unit testing rules — Recording & alerting rules.
Alerting overview, Configuration, Alertmanager and High availability — Alertmanager.
Storage, Federation and Remote read API, plus Thanos, Cortex and Mimir (linked, not documented in depth) — Storage, federation & remote write.
Security, TLS encryption, Management API and promtool — Security & production operations.
node_exporter and blackbox_exporter (linked, not documented in depth) — Exporters & the Pushgateway.
kube-state-metrics (linked, not documented in depth) — Containers & Kubernetes monitoring.
Install Grafana, Prometheus data source, Dashboards, Panels & visualizations and Variables, plus Grafana’s public dashboard library — Grafana: Dashboards & visualization.
Grafana Alerting — Grafana alerting.
Pivotto, Julien; Brazil, Brian. Prometheus: Up & Running, 2nd ed. O’Reilly Media, 2023. ISBN 978-1-098-13108-3 — official page: https://oreil.ly/prometheus-up-running-2e. Consulted as part of the bibliography for this section; not the primary or main reference — the official Prometheus and Grafana documentation above is authoritative on any discrepancy.