Storage, federation & remote write

This section documents the current Prometheus 3.x line (latest 3.14.0) as published at the Prometheus documentation and the current Grafana 13.x line (latest 13.2.1) as published at the Grafana documentation, which are the references these pages are written and verified against. No specific patch version is pinned. Some capabilities (Grafana Enterprise-only features, Grafana Cloud-hosted services, and long-term-storage systems like Thanos/Cortex/Mimir) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, since both projects iterate quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

A single Prometheus server is deliberately self-contained: it scrapes, stores, and queries its own samples on local disk, with no clustering and no built-in replication. That design is simple and fast, but it caps how much history one server can hold and rules it out as a durable archive. This page covers the on-disk storage engine and its retention flags, the /federate endpoint that lets one server scrape a filtered subset of another’s series, the remote_write/remote_read APIs that forward samples to or read them back from an external system, and the long-term storage systems — Thanos, Cortex, and Mimir — that build on those APIs to remove the single-server ceiling.

Local TSDB storage

On-disk layout, conceptually

Prometheus’s storage engine (the TSDB) keeps recent samples in an in-memory head block, backed by a write-ahead log (WAL) on disk so an unexpected restart replays and loses nothing. Every two hours the head is flushed into an immutable, self-contained on-disk block — its own directory of chunk files, an index, and metadata — covering that time window. A background compactor periodically merges adjacent blocks into larger ones, reducing per-block index/metadata overhead and reclaiming space from series that have since disappeared. None of this needs to be managed by hand; it is described here only so the retention flags below make sense. For the full picture, read Storage.

Retention flags

Retention is bounded by whichever of two flags is hit first — time or disk size — both passed on the command line:

prometheus \
  --config.file=prometheus.yml \
  --storage.tsdb.path=/prometheus \
  --storage.tsdb.retention.time=15d \
  --storage.tsdb.retention.size=512GB
# https://prometheus.io/docs/prometheus/latest/storage/

--storage.tsdb.retention.time (default 15d) deletes whole blocks once their samples fall outside the window; --storage.tsdb.retention.size caps the total size of the storage directory (WAL excluded) and, once reached, drops the oldest blocks to stay under it. Setting the size limit lets an operator bound disk usage on a host with a fixed volume without having to reason about ingestion rate and cardinality to derive a time-based equivalent.

Why local storage is not for long-term retention

Local TSDB storage is single-node and non-replicated: there is no built-in replication, no cross-server durability, and no support for horizontal scale-out of storage capacity. Losing the disk loses the data; growing beyond one host’s disk and memory means either shortening retention or routing samples elsewhere. It is well suited to the days-to-weeks window operational dashboards and alerting need, and poorly suited to compliance-grade or multi-year retention, multi-tenant querying, or globally deduplicated views across many Prometheus servers. That gap is exactly what federation (read-only, and only over a filtered subset) and the long-term storage systems described below exist to fill.

Federation

The /federate endpoint

Every Prometheus server exposes /federate, which returns the current value of whatever time series match one or more match[] selectors, in the same exposition format a scrape target would return. A second Prometheus server scrapes that endpoint like any other target, pulling in only the series it asked for — typically pre-aggregated recording-rule outputs from several lower-level servers into one higher-level, global-view server:

scrape_configs:
  - job_name: "federate"
    scrape_interval: 15s
    honor_labels: true
    metrics_path: "/federate"
    params:
      "match[]":
        - '{job="node"}'
        - '{__name__=~"job:.*"}'          # recording-rule outputs, by convention
    static_configs:
      - targets:
          - "prometheus-region-a:9090"
          - "prometheus-region-b:9090"
# https://prometheus.io/docs/prometheus/latest/federation/

honor_labels: true is required so the source server’s own instance/job labels are preserved instead of being overwritten by the scrape. The same query run directly against the endpoint:

curl -G 'http://prometheus-region-a:9090/federate' \
  --data-urlencode 'match[]={job="node"}' \
  --data-urlencode 'match[]={__name__=~"job:.*"}'
# https://prometheus.io/docs/prometheus/latest/federation/

Scaling limits of federation

Federation is intentionally narrow: it hands over only the series matched by match[], at scrape time, with no history beyond what the pull captures — it is not a way to replicate a full dataset or back up historical data. Each additional federating server adds another full evaluation of those selectors against the source’s local storage, so pulling many or high-cardinality series, or chaining several layers of federation, pushes load back onto the servers being federated from. The documented, supported use is aggregating a small number of higher-level, already-aggregated series (job- or cluster-level recording-rule outputs) into a global view, or occasionally pulling a handful of related series across the boundary between related services — not shipping raw per-instance metrics or building durable long-term storage on top of it. See Federation for the full scoping guidance.

The remote_write/remote_read APIs

Where federation pulls a filtered snapshot, remote_write and remote_read move samples wholesale between a Prometheus server and an external system, over a standardized HTTP protocol.

remote_write

A remote_write block makes Prometheus forward every scraped sample — after any configured relabeling — to a remote endpoint, asynchronously and with its own retrying queue, independent of local retention:

remote_write:
  - url: "https://remote-storage.example.com/api/v1/write"
    queue_config:
      max_samples_per_send: 5000
      max_shards: 30
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_.*"
        action: drop

The receiving system only needs to implement the write side of the protocol — this is exactly the integration point Thanos receive, Cortex, and Mimir (below) expose.

remote_read

A remote_read block does the reverse: PromQL queries evaluated against this server can transparently pull in matching series from the remote system for data outside local retention, merged with what is stored locally:

remote_read:
  - url: "https://remote-storage.example.com/api/v1/read"
    read_recent: false   # only fetch from remote what isn't already available locally

The wire format and query semantics are specified in Remote read API.

Long-term storage systems

remote_write/remote_read (plus, for some of these, a sidecar reading blocks directly off disk) are the integration points several purpose-built systems use to give Prometheus durable, horizontally scalable, long-term storage beyond a single server’s disk. They are linked here, not documented in depth:

  • Thanos — a sidecar process attached to each Prometheus server uploads its TSDB blocks to object storage (S3, GCS, Azure Blob, etc.) and serves them back through a Store API that a central Querier component fans queries out to, giving a globally deduplicated view across many Prometheus servers with effectively unlimited retention.

  • Cortex — a horizontally scalable, multi-tenant TSDB that Prometheus servers write into over remote_write, distributing ingestion and storage across a cluster instead of relying on object-storage sidecars per server.

  • Mimir — a newer, also horizontally scalable and remote_write- compatible long-term store from Grafana Labs, aimed at very high cardinality and scale, sharing much of its lineage with Cortex.

Continue with Security & Production Operations for retention and access-control concerns in production, or back to the Prometheus & Monitoring index.