Service discovery & relabeling

This section documents the current Prometheus 3.x line (latest 3.14.0) as published at the Prometheus documentation and the current Grafana 13.x line (latest 13.2.1) as published at the Grafana documentation, which are the references these pages are written and verified against. No specific patch version is pinned. Some capabilities (Grafana Enterprise-only features, Grafana Cloud-hosted services, and long-term-storage systems like Thanos/Cortex/Mimir) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, since both projects iterate quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

A Prometheus scrape_config names a job and, separately, how it finds the targets that job scrapes. Static lists work for small, stable setups, but anything that grows or churns — a fleet behind an inventory system, a Kubernetes cluster, an autoscaling group — needs Prometheus to ask something else for the current target list. This page covers the discovery mechanisms that feed a job (static_configs, file_sd_configs, http_sd_configs, and a brief survey of consul_sd_configs/ec2_sd_configs/ kubernetes_sd_configs), then the two relabeling stages that turn discovered targets and their raw metadata into the labels that actually reach storage: relabel_configs, which runs before a target is scraped, and metric_relabel_configs, which runs after, on the scraped samples themselves. The full scrape_config schema, including every discovery mechanism’s own options, is documented at Configuration. For the groundwork — what a scrape config is and how prometheus.yml is structured — see Getting Started.

Service discovery mechanisms

Every sd_configs block ends up producing the same shape: a list of targets, each with an address and a set of labels prefixed meta that describe where it came from. Those meta_* labels only exist during relabeling — they are discarded before scraping unless a relabel_configs rule copies their value into a label that survives, which is the whole point of the relabeling stage covered below.

static_configs: a fixed list

The simplest mechanism: a literal list of targets (host:port pairs) and optional labels applied to all of them, no external system involved.

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["10.0.1.11:9100", "10.0.1.12:9100"]
        labels:
          env: production
      - targets: ["10.0.2.11:9100"]
        labels:
          env: staging

file_sd_configs: targets from a file

Prometheus watches one or more files (JSON or YAML) and reloads the target list whenever they change, with no Prometheus restart or config reload needed. This is the common hand-off point for a configuration-management tool (Ansible, Puppet, Terraform) or a small custom script that writes the file on a cron: the external system owns the inventory, Prometheus just watches for changes.

scrape_configs:
  - job_name: node
    file_sd_configs:
      - files:
          - /etc/prometheus/targets/node/*.json
        refresh_interval: 5m

Each watched file holds a list of target groups:

[
  {
    "targets": ["10.0.1.11:9100", "10.0.1.12:9100"],
    "labels": { "env": "production", "rack": "a3" }
  }
]

http_sd_configs: targets from an HTTP endpoint

Prometheus polls a URL on refresh_interval and expects back the same target-group JSON shape as file_sd_configs — useful when the source of truth is a live API (a CMDB, an internal inventory service) rather than a file another process writes. The endpoint must set Content-Type: application/json and Prometheus retries with backoff on failure, keeping the last known good list rather than scraping nothing.

scrape_configs:
  - job_name: node
    http_sd_configs:
      - url: http://inventory.internal:8080/prometheus/sd/node
        refresh_interval: 1m

See HTTP-based service discovery for the exact response schema, polling and error-handling semantics, and how it differs from writing your own file_sd_configs output.

Consul, EC2, and Kubernetes SD

Beyond static/file/HTTP, Prometheus ships first-class discovery for a number of specific platforms; each populates its own set of _meta<mechanism>_* labels reflecting that platform’s metadata.

  • consul_sd_configs queries a Consul catalog for services matching a services filter (or all services when omitted) and exposes Consul tags, node metadata, and health-check state as _meta_consul* labels — a natural fit when services already register themselves in Consul.

  • ec2_sd_configs lists EC2 instances in a region via the AWS API, exposing instance ID, type, availability zone, VPC, subnet, security groups, and tags as _meta_ec2* labels — useful for scraping node/agent exporters across an unmanaged or autoscaled fleet without a separate inventory system.

  • kubernetes_sd_configs discovers node, pod, service, endpoints, endpointslice, or ingress objects directly from the Kubernetes API server, exposing pod/service/namespace metadata and annotations as _meta_kubernetes* labels.

scrape_configs:
  - job_name: consul-services
    consul_sd_configs:
      - server: consul.internal:8500
        services: ["web", "api"]

  - job_name: ec2-nodes
    ec2_sd_configs:
      - region: eu-west-1
        port: 9100

Kubernetes SD is the mechanism most readers of this section actually reach for, and it deserves more than a config snippet — the role you pick (pod vs endpoints vs service, and how each shapes the resulting _meta_kubernetes* labels) materially changes what gets scraped. That deep dive lives in Containers & Kubernetes Monitoring; this page only establishes that kubernetes_sd_configs exists alongside the other mechanisms above.

relabel_configs: rewriting targets before scraping

relabel_configs runs once per discovered target, before Prometheus scrapes it, against the target’s labels (the job’s own static labels, plus every meta_* label the discovery mechanism attached). It can drop a target entirely, keep only a matching subset, or rewrite/add labels — including building the final address__ used to reach the target. Each rule reads one or more source_labels, joins them with separator (default ;), matches the result against regex (default (.*), i.e. "match everything"), and applies action (default replace) using capture groups from that match.

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # keep: only pods that opted in via an annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        regex: "true"
        action: keep

      # drop: skip pods still in an init/terminating phase
      - source_labels: [__meta_kubernetes_pod_phase]
        regex: "Pending|Terminating"
        action: drop

      # replace: honor a custom scrape port from an annotation, falling back to 9100
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        separator: ";"
        regex: "([^:]+)(?::\\d+)?;(\\d+)"
        target_label: __address__
        replacement: "${1}:${2}"

      # labelmap: copy every pod label onto the target as a "pod_label_<name>" label
      - regex: __meta_kubernetes_pod_label_(.+)
        action: labelmap

      # replace: surface namespace and pod name as regular labels
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

The actions used most often:

Action Effect

keep

Drop the target unless the regex matches source_labels.

drop

Drop the target if the regex matches source_labels.

replace (default)

Set target_label to replacement, with $1/${1} etc. substituted from the regex’s capture groups.

labelmap

For every label name matching regex, add a new label named by its capture group, with the same value.

labeldrop / labelkeep

Remove labels matching regex (labeldrop), or remove every label that does not match (labelkeep).

hashmod

Set target_label to hash(source_labels) mod modulus — the basis for sharding targets across multiple Prometheus instances.

Because relabel_configs runs before scraping, it is the only place a target can be dropped without ever being scraped, and the only place address (and therefore the actual host:port Prometheus connects to) can be rewritten. Any label whose name starts with is stripped after relabeling finishes and never reaches storage — which is exactly why meta_* values must be copied into a plain label (as namespace/pod are above) if they need to survive.

metric_relabel_configs: rewriting samples after scraping

metric_relabel_configs uses the identical rule syntax — source_labels, separator, regex, action, target_label, replacement — but runs after the scrape completes, against the label set of each returned sample (the metric name is itself matched via the label name). It cannot change what gets scraped or how; it only decides what a successful scrape actually stores.

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["10.0.1.11:9100"]
    metric_relabel_configs:
      # drop: discard a noisy, low-value metric family entirely
      - source_labels: [__name__]
        regex: "node_scrape_collector_duration_seconds"
        action: drop

      # keep: for a job that exposes far more series than needed, keep only a
      # known-useful subset by metric name prefix
      - source_labels: [__name__]
        regex: "node_(cpu|memory|filesystem|network)_.*"
        action: keep

      # replace: normalize a device label's value before it is stored
      - source_labels: [device]
        regex: "veth.*"
        target_label: device
        replacement: "virtual"

The most common use for metric_relabel_configs is cardinality control: dropping metric families or high-cardinality label values (per-request-ID labels, ephemeral container IDs) that would otherwise bloat storage, without asking the target to stop exposing them. Because it runs per sample rather than per target, it is more expensive than relabel_configs at high scrape volume — prefer dropping a whole target earlier, in relabel_configs, when that is an option.

honor_labels: resolving label collisions

A scraped sample can carry a label whose name collides with one Prometheus would otherwise attach from the target (most often instance or job, but any target label can collide). honor_labels (a scrape-config-level boolean, default false) decides who wins:

  • honor_labels: false (the default) — the target’s label wins. A colliding label exposed by the scraped metric is renamed by prefixing it exported_<name>, and the target’s own value is kept under the original name.

  • honor_labels: true — the scraped metric’s label wins outright, overriding the target label of the same name with no renaming.

scrape_configs:
  - job_name: federate
    honor_labels: true
    metrics_path: /federate
    params:
      "match[]": ['{job="node"}']
    static_configs:
      - targets: ["prometheus-leaf:9090"]

honor_labels: true is the right default for federation and for any target scraped through a proxy that must preserve the original instance/job it is reporting on behalf of — see Storage, Federation & Remote Write. For an ordinary application target, leaving it false is almost always what you want: it stops a misbehaving exporter from silently overwriting the instance/job labels Prometheus uses to identify where a series came from. The full semantics, including how it interacts with honor_timestamps, are in Configuration.

Where to next