Prometheus architecture & ecosystem

This section documents the current Prometheus 3.x line (latest 3.14.0) as published at the Prometheus documentation and the current Grafana 13.x line (latest 13.2.1) as published at the Grafana documentation, which are the references these pages are written and verified against. No specific patch version is pinned. Some capabilities (Grafana Enterprise-only features, Grafana Cloud-hosted services, and long-term-storage systems like Thanos/Cortex/Mimir) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, since both projects iterate quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

Prometheus is not one process but a small ecosystem built around one central idea: the Prometheus server periodically pulls (scrapes) metrics over HTTP from the things it monitors, stores them in its own time-series database, evaluates rules against them, and hands the results to whoever asks — Grafana, amtool, or a raw curl against its HTTP API. This page maps every piece of that ecosystem — the server, client libraries, exporters, service discovery, the Pushgateway, Alertmanager and remote storage — and explains why the pull model is the default and where the Pushgateway’s push model is the deliberate exception. See Overview — Architecture for the canonical diagram this page expands on.

Scrape targets — instrumented apps

The Prometheus server

The server is the hub of the whole system and, unlike most monitoring backends, it is a single self-contained binary with no required external dependencies. Internally it is built from a small number of cooperating pieces:

  • A scraper that reads the scrape configuration (targets, either static or produced by service discovery) and pulls a /metrics page from each target on its configured interval.

  • The TSDB (time-series database), an embedded, append-only, on-disk storage engine tuned for short samples with a small set of labels each, chunked into two-hour blocks and compacted in the background. It is covered in depth in Storage, Federation & Remote Write.

  • A rule evaluation engine that runs recording rules (precompute an expensive PromQL expression into a new series) and alerting rules (evaluate a PromQL expression and fire an alert when it is true) on a fixed interval, covered in Recording & Alerting Rules.

  • An HTTP API and query engine that serves PromQL queries — instant and range — to Grafana, amtool, and the built-in expression browser, plus the /metrics endpoint the server exposes about itself.

All four run inside one process against one local TSDB; there is no separate ingestion tier, message queue, or external index to operate for a single-server deployment. That simplicity is also the main scaling boundary the ecosystem’s remote-storage integrations exist to work around.

The pull model, and why it is the default

Prometheus scrapes targets rather than waiting for them to send data. A target does nothing but expose its current state at a stable HTTP path (/metrics by default); the server decides when to collect it, from where, and what to do if the request fails.

# prometheus.yml -- minimal scrape configuration
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "my-service"
    static_configs:
      - targets: ["my-service:8080"]
# https://prometheus.io/docs/introduction/overview/#architecture

This design choice pays off in several concrete ways:

  • Up detection is automatic. A failed scrape immediately produces up == 0 for that target — there is no separate heartbeat protocol to build, and no ambiguity between "the target has nothing to report" and "the target is down."

  • One place controls collection load. The server, not thousands of independently-configured agents, owns the scrape interval and timeout, so a fleet-wide change (or a temporary slowdown to protect an overloaded target) is one config edit instead of a redeploy of every instrumented process.

  • Targets stay simple. An instrumented process only has to serve a page of current values on request; it never needs to know a metrics backend’s address, retry logic, or batching strategy.

  • Ad hoc scraping needs no fleet-wide change. Because any HTTP endpoint that speaks the exposition format is scrapable, a developer can point a local Prometheus at one target — or curl its /metrics page directly — without touching production configuration.

The trade-off is the mirror image of each of those points: the server must be able to reach every target over the network (so a target behind restrictive egress-only firewalls or a NAT it cannot be dialled into is awkward), and anything that does not live long enough to be scraped — a batch job, a cron task, a CI step — has nothing to be scraped from. That second case is exactly what the Pushgateway exists to solve, and is the one place in the ecosystem where Prometheus’s data flow is deliberately push-based.

Client libraries: instrumenting a process

A client library is what turns application code into a scrape target: it maintains counters, gauges, histograms and summaries in memory and serves them, on request, as a plain-text page in the Prometheus exposition format. Official libraries exist for Go, Java/JVM, Python, Ruby, Rust and several other languages; they are the subject of Instrumenting Applications and the metric types themselves are covered in Data Model & Metric Types. A scrape of an instrumented process looks like this:

curl -s http://localhost:8080/metrics | head
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 128301

Exporters: metrics for things you cannot instrument

An exporter is a small standalone process that speaks the exposition format on behalf of a system you cannot (or would not) add a Prometheus client library to directly — a database, a hardware device, a hosted third-party service. It periodically queries that system through its own protocol and re-publishes the result as a /metrics page the Prometheus server scrapes like any other target. node_exporter (host-level CPU, memory, disk and network metrics) and the many database- and service-specific exporters in the exporters and integrations list are the most common examples; running them alongside containers and Kubernetes workloads is covered in Containers & Kubernetes Monitoring, and exporter conventions and behavior are covered in depth in Exporters & the Pushgateway.

Service discovery: finding what to scrape

Static target lists do not survive autoscaling, rolling deploys, or an orchestrator that reassigns IP addresses on every restart. Service discovery (SD) lets the scrape configuration instead describe where to look — a Kubernetes API server, a Consul catalog, a cloud provider’s API, DNS — and Prometheus resolves that into a current target list on its own, re-resolving periodically so scaling events are picked up without a config reload. Discovered targets carry metadata as labels (pod name, namespace, EC2 tags, and so on) that relabeling rules can then use to keep, drop, or rewrite targets before they are ever scraped. The full mechanics — SD mechanisms, the metadata each one exposes, and the relabeling pipeline — are in Service Discovery & Relabeling.

The Pushgateway: the push exception

The Pushgateway is a separate, small service that sits between short-lived jobs and the Prometheus server. A batch job that starts, does work, and exits before any scrape interval could catch it pushes its final metric values to the Pushgateway instead; the Pushgateway holds those values and exposes them at its own /metrics endpoint, which the Prometheus server then scrapes normally, on its usual pull schedule, like any other target.

echo 'batch_job_last_success_seconds 1' | curl --data-binary @- \
  http://pushgateway:9091/metrics/job/nightly_backup
# https://prometheus.io/docs/instrumenting/pushing/

This keeps the pull model intact end to end — the Pushgateway is a target, not an ingestion endpoint on the server itself — while covering the one case genuine pull scraping cannot: a process that no longer exists by the time a scrape would run. It is not a general-purpose push gateway for long-running services, and using it that way loses the automatic up/down detection a real scrape target gets for free, since a value pushed once keeps being served (and looking fresh) until it is explicitly deleted. Pushgateway operation, metric grouping, and its PUT/DELETE semantics are covered in Exporters & the Pushgateway.

Alertmanager: turning firing alerts into notifications

Alerting rules inside the Prometheus server only decide that something is wrong and produce a firing alert; the server then pushes that alert to Alertmanager, a separate process responsible for what happens next — deduplicating alerts coming from multiple Prometheus servers, grouping related ones into a single notification, routing each group to the right receiver by label, silencing known issues, and inhibiting lower-priority alerts once a related higher-priority one is already firing. Alertmanager then delivers the result over email, Slack, PagerDuty, a generic webhook, or any other configured integration. Separating this from the server means several Prometheus servers — as in a federated or sharded setup — can share one alert-routing brain instead of each needing its own notification logic. The rule side of this pipeline is in Recording & Alerting Rules; Alertmanager itself, including its routing tree and grouping configuration, is in Alertmanager.

Remote storage: beyond one server’s local TSDB

A single server’s local TSDB is deliberately not built for indefinite retention or cross-cluster querying. The remote_write API lets a server continuously ship every scraped sample to an external, Prometheus-API-compatible system — Thanos, Cortex, Mimir, or a cloud-hosted equivalent — for long-term storage, global querying across many servers, and durability beyond one node’s disk. The complementary remote_read API lets PromQL queries against the local server transparently pull in data that only exists in that remote system. Because remote_write is a push from the server’s own point of view, it does not contradict the pull model described above: pulling from targets and pushing accumulated samples onward to long-term storage are two independent, orthogonal data paths. This is covered in full in Storage, Federation & Remote Write.

How the pieces fit together

Reading the diagram above left to right: instrumented apps and exporters are scraped directly; short-lived jobs push to the Pushgateway, which is itself then scraped — so every path into Prometheus is still a pull from the server’s perspective. Inside the server, the TSDB stores what was scraped and rule evaluation reacts to it. From there, firing alerts are pushed to Alertmanager for routing and notification, samples are pushed onward via remote_write for long-term storage, and Grafana (or any other PromQL client) pulls results back out through the HTTP query API. From here: