Exporters & the Pushgateway

node_exporter exposes kernel and hardware metrics for the machine it runs on — CPU, memory, filesystems, disks, network interfaces — by reading /proc and /sys on Linux (with narrower support for other Unix-likes). Each metric family comes from an independent collector, and most collectors are on by default:

Collectors are toggled with --collector.<name> / --no-collector.<name> flags at startup; a handful (for example textfile, covered below) default to off instead:

The textfile collector is the escape hatch for anything node_exporter does not natively read: a cron job writes plain exposition-format lines to a .prom file in the watched directory, and node_exporter folds them into its own /metrics output on the next scrape — no separate HTTP endpoint needed for that data. The full collector list, their default on/off state, and each metric’s name and labels are documented in the project’s own reference, not duplicated here: node_exporter.

Where node_exporter reports what a host observes about itself, blackbox_exporter reports what an external probe observes about a target — it is the exporter to reach for when the question is "is this endpoint reachable and responding correctly," not "what is this host’s own CPU doing." It supports HTTP(S), TCP, ICMP, and DNS probes, each defined as a named module in its own config file:

Prometheus does not scrape blackbox_exporter directly for the target’s own metrics; it scrapes blackbox_exporter’s `/probe endpoint, passing the real target and the module to use as query parameters. The exporter performs the probe on the fly and returns metrics such as probe_success (1/0) and probe_duration_seconds for that request:

The relabel_configs block above is the standard pattern for every blackbox_exporter job: the real target from static_configs is rewritten into a param_target query parameter, copied into the instance label so it (not the exporter’s own address) identifies the series, and address__ is then overwritten to point the scrape itself at blackbox_exporter. Relabeling in general is covered in Service Discovery & Relabeling. ICMP probing needs raw-socket privileges (CAP_NET_RAW or root) on the host running the exporter.

When neither an existing exporter nor direct instrumentation (covered in Instrumenting Applications) fits — typically because the system you need metrics from is a third party you cannot add a client library to — the fix is a small standalone process that follows the same shape every official exporter uses:

The live-query-per-scrape pattern is simplest and keeps metrics as fresh as the last scrape, but it means a slow upstream call directly slows (or times out) the scrape; a background collector decouples the two at the cost of metrics that lag their own collection interval. Both are legitimate, and the choice is a per-exporter judgment call, not a Prometheus rule. The canonical guidance on this decision, naming conventions, and metric-type selection is Writing exporters.

Every mechanism above is pull-based: Prometheus initiates the scrape, on its own schedule, and can tell a target apart from a target that has simply gone quiet. The Pushgateway is the one place this model is deliberately inverted: it is a small HTTP service that accepts pushed metrics and holds them in memory so a normal Prometheus scrape of the Pushgateway itself can still pull them out. It exists for exactly one case: short-lived batch or cron jobs that exit before any scrape could ever reach them.

Prometheus then scrapes the Pushgateway like any other target:

honor_labels: true is not optional in practice here: without it, Prometheus’s own relabeling would overwrite the pushed job/instance labels with the Pushgateway’s own, collapsing every pushed job under one identity.