Security & production operations

This section documents the current Prometheus 3.x line (latest 3.14.0) as published at the Prometheus documentation and the current Grafana 13.x line (latest 13.2.1) as published at the Grafana documentation, which are the references these pages are written and verified against. No specific patch version is pinned. Some capabilities (Grafana Enterprise-only features, Grafana Cloud-hosted services, and long-term-storage systems like Thanos/Cortex/Mimir) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, since both projects iterate quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

Prometheus has no built-in notion of users, roles, or fine-grained authorization — the server itself only knows how to terminate TLS and, optionally, require a single set of HTTP credentials in front of every endpoint. This page covers that web-config.yml layer; capacity-planning rules of thumb for memory and CPU; the two ways a single Prometheus server stops scaling for ingestion — horizontal sharding and remote-writing to a scalable backend — and the separate concern of read-path high availability; monitoring Prometheus with Prometheus; the promtool subcommands that validate config and rule files; and the small HTTP management API used to reload or drain a running server.

TLS and basic auth: web-config.yml

Since Prometheus 2.24, the server’s own HTTP endpoint — the expression browser, the scrape-facing /metrics, and the management API below — can terminate TLS and require HTTP basic auth, configured in a separate YAML file passed via --web.config.file. Nothing here is authorization: every successful login (or every client, with no auth configured at all) gets full access to every endpoint, including /-/quit if enabled.

tls_server_config:
  cert_file: prometheus.crt
  key_file: prometheus.key

basic_auth_users:
  # bcrypt hash, not the plaintext password -- generate with:
  # htpasswd -nBC 10 "" | tr -d ':\n'
  admin: "$2y$10$wUxNy9WgFcgWlkA2xfMBPeE9U0d18Sf9CDf37gDN9DR8pgvVfrOng"
# https://prometheus.io/docs/prometheus/latest/configuration/https/
./prometheus --config.file=prometheus.yml --web.config.file=web-config.yml

Reloading web-config.yml itself needs no SIGHUP or /-/reload call — Prometheus watches the file and picks up TLS certificate and credential changes automatically. Because there is no per-user or per-endpoint access control, and no way to expose read-only PromQL access to one team while keeping /-/quit restricted to another, real deployments put a reverse proxy (nginx, an API gateway, an Alertmanager-style OAuth2 proxy) in front and let it own authentication, authorization, and rate limiting; web-config.yml mainly protects a server that would otherwise sit on the network with scrape data — and its management API — completely open. See TLS encryption for the full tls_server_config/http_server_config option set (minimum TLS version, cipher suites, client-cert verification) and Security for the broader threat model this addresses — including the explicit statement that Prometheus’s HTTP API and its data model are not designed to be handed to untrusted users.

Capacity planning

Prometheus’s local TSDB is largely predictable per sample, but cardinality — the number of distinct label-value combinations a metric produces — dominates memory in a way sample count alone does not.

  • Disk: roughly 1—​2 bytes per sample once compressed, after the first two hours in the head block. A back-of-envelope estimate: ingested_samples_per_second * bytes_per_sample * retention_seconds.

  • Memory: dominated by the number of actively-scraped time series (unique metric-name + label-set combinations) held in the in-memory head block, not by the sample rate alone. A metric that adds a high-cardinality label — a raw user ID, a full URL path, a pod name that churns on every deploy — can multiply its series count by orders of magnitude and is the most common cause of an out-of-memory Prometheus server; this is why label design gets its own treatment in Data Model & Metric Types and Instrumenting Applications.

  • CPU: scales with scrape frequency, rule-evaluation frequency, and the cost of PromQL queries run against the head block — an expensive dashboard query re-run on every page load costs real CPU on every load, which is exactly what recording rules in Recording & Alerting Rules exist to amortize.

  • --query.max-samples bounds how many samples a single PromQL query may load into memory at once (default 50,000,000), guarding against one runaway query — an unbounded range on a high-cardinality series — exhausting server memory; a query that exceeds it fails outright rather than degrading the whole server.

These are rules of thumb for sizing a node, not a substitute for watching the server’s own resource usage under real load — see meta-monitoring below.

Horizontal sharding

A single Prometheus server is not natively distributed: it scrapes its configured targets, evaluates its rules, and stores the result on local disk, with no clustering or gossip between servers. When one server can no longer keep up with the number of targets or the ingestion rate, the standard pattern is horizontal sharding — running several Prometheus servers, each configured to scrape a disjoint subset of targets, typically selected by a hash of the target’s labels via relabeling:

scrape_configs:
  - job_name: "node"
    file_sd_configs:
      - files: ["targets/*.json"]
    relabel_configs:
      # keep only targets whose instance hashes to shard 0 of 3
      - source_labels: [__address__]
        modulus:        3
        target_label:   __tmp_hash
        action:         hashmod
      - source_labels: [__tmp_hash]
        regex:          "0"
        action:         keep
# https://prometheus.io/docs/prometheus/latest/getting_started/

Each shard is a fully independent Prometheus with its own storage; there is no query-time fan-out across shards built into the server, so dashboards and alerts that need a global view either query each shard separately or read from a downstream system that has already merged them. That downstream merge — and the alternative of not sharding scrape targets at all but instead remote_write-ing every sample from one or more Prometheus servers into a horizontally-scalable long-term-storage backend — is covered in Storage, Federation & Remote Write, along with federation, which pulls a limited, already-aggregated subset of series between servers rather than sharding raw ingestion.

High availability

Sharding solves capacity; it does nothing for a shard going down. The standard HA pattern is orthogonal: run two (or more) identical Prometheus replicas, each with the same scrape_configs and the same rule files, scraping the same targets independently. Neither replica knows about the other — there is no replication or consensus between them, only duplicated work — so each holds its own complete, independent copy of the data. A query layer in front (Thanos Query, Grafana Mimir, or Grafana’s own built-in data-source failover) reads from both and deduplicates identical series so a single replica restarting or missing a scrape doesn’t produce a visible gap. Because each replica scrapes independently, the two copies are rarely byte-identical — their scrape timestamps drift slightly — which is precisely what the read-side deduplication step accounts for. This document does not cover the deduplicating query layer itself in depth; see Storage, Federation & Remote Write for where it fits alongside remote-write-based long-term storage.

Meta-monitoring: watching Prometheus with Prometheus

Every Prometheus server exposes its own internal state — ingestion rate, head-block series count, rule-evaluation duration, TSDB compaction stats, web-config.yml reload failures — on its own /metrics endpoint, exactly like any other exporter. The getting-started skeleton already scrapes this by pointing a prometheus job at the server’s own address:

prometheus_tsdb_head_series
rate(prometheus_tsdb_head_samples_appended_total[5m])
prometheus_engine_query_duration_seconds

For an HA pair, each replica should scrape and alert on itself (and, where feasible, be scraped by a separate, independent monitoring server) so that a failure mode that takes down one replica’s own ingestion doesn’t also silence the alert about it. Typical meta-alerts cover scrape failures (up == 0 against the server’s own job), rule-evaluation lag (prometheus_rule_group_last_duration_seconds approaching the group’s interval), and TSDB out-of-order or write failures. Writing these as recording and alerting rules follows the same pattern as any other rule; see Recording & Alerting Rules.

Validating config and rules with promtool

promtool (bundled with every Prometheus release) has subcommands beyond the query/debug ones covered in Recording & Alerting Rules — notably check config and check rules, which parse a file exactly as the server would and catch typos, unknown fields, and invalid PromQL before a reload fails or, worse, silently loads a broken rule group:

promtool check config prometheus.yml
promtool check rules alert_rules.yml
# https://prometheus.io/docs/prometheus/latest/configuration/promtool/

Wiring either into CI on every change to prometheus.yml or a rules file catches a bad config before it ever reaches /-/reload. See promtool for the full subcommand list, including check web-config, check service-discovery, and check metrics (which lints scraped output against Prometheus’s own metric- and label-naming conventions).

The management API

With --web.enable-lifecycle set, a small set of unauthenticated-by-default endpoints let an operator reload or drain a running server without restarting it — exactly the endpoints the web-config.yml section above is protecting when TLS/basic auth or a reverse proxy is in front:

curl -X POST http://localhost:9090/-/reload   # re-read prometheus.yml and every rule file
curl -X POST http://localhost:9090/-/quit     # graceful shutdown (needs --web.enable-lifecycle,
                                               # same as /-/reload above)
curl http://localhost:9090/-/healthy          # 200 once the process is up -- liveness probe
curl http://localhost:9090/-/ready            # 200 once storage is initialized and queryable -- readiness probe
# https://prometheus.io/docs/prometheus/latest/management_api/

/-/reload is what turns a SIGHUP-free config change into effect — CI running promtool check config followed by a POST /-/reload is the standard deploy-time pattern for pushing a new prometheus.yml or rule file without downtime. /-/healthy and /-/ready are what a container orchestrator’s liveness and readiness probes should point at respectively: /-/healthy only confirms the process is running, while /-/ready additionally confirms the TSDB has finished startup replay and the server can actually serve queries — routing traffic on /-/healthy alone can send requests to a server that is up but not yet ready to answer them. See Management API for the complete endpoint list, including the admin-only snapshot and delete-series endpoints gated behind --web.enable-admin-api.

Continue with Storage, Federation & Remote Write for how sharded or HA-paired servers feed a scalable long-term-storage backend, and Alertmanager for routing the alerts these production servers fire.