Containers & Kubernetes monitoring

This section documents the current Prometheus 3.x line (latest 3.14.0) as published at the Prometheus documentation and the current Grafana 13.x line (latest 13.2.1) as published at the Grafana documentation, which are the references these pages are written and verified against. No specific patch version is pinned. Some capabilities (Grafana Enterprise-only features, Grafana Cloud-hosted services, and long-term-storage systems like Thanos/Cortex/Mimir) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, since both projects iterate quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

Kubernetes is Prometheus’s best-supported deployment target: the kubelet already exposes per-container resource usage, the API server already knows every node, service, pod and endpoint, and Prometheus ships a dedicated kubernetes_sd_configs discovery mechanism that reads that API directly. This page covers the two metric sources you scrape — cAdvisor (resource usage) and kube-state-metrics (cluster state) — and the discovery roles that find their targets. It assumes the general relabel_configs mechanics from Service Discovery & Relabeling; this page focuses on what is specific to the Kubernetes roles. For container resource limits, logging drivers and the Docker daemon’s own Prometheus endpoint, see Resources, Logging and Monitoring. For Kubernetes observability in general, see Observability: Logging and Metrics.

cAdvisor: per-container resource metrics

cAdvisor (Container Advisor) is built into the kubelet itself — there is nothing separate to deploy. Every kubelet exposes it at /metrics/cadvisor, alongside the kubelet’s own /metrics endpoint, and both are normally scraped through the API server’s proxy rather than reached directly, since the kubelet’s HTTPS endpoint needs a bearer token and the cluster CA:

scrape_configs:
  - job_name: kubernetes-cadvisor
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
    # https://prometheus.io/docs/guides/cadvisor/

cAdvisor’s metrics are all resource-usage numbers scoped to a single container: container_cpu_usage_seconds_total, container_memory_working_set_bytes, container_fs_usage_bytes, container_network_receive_bytes_total, and so on, each labeled with pod, namespace, container and image. It answers how much a container is actually consuming, not whether the workload as a whole is healthy — that distinction matters below. See Monitoring Docker container metrics using cAdvisor for the full metric set and a walkthrough of this scrape config.

# Per-pod CPU usage over the last 5 minutes, summed across containers.
sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="default"}[5m]))

kubernetes_sd_configs: the discovery roles

A kubernetes_sd_configs entry talks to the Kubernetes API server (in-cluster config is autodetected from the service account mounted into the Prometheus pod) and re-runs on every change, so scrape targets track the cluster’s actual state without a restart. Each entry declares exactly one role, and the role determines both what gets listed as a target and which _meta_kubernetes* labels are attached for relabeling:

Role What it discovers and attaches

node

One target per cluster node, addressed at the kubelet’s port. Attaches meta_kubernetes_node_name, meta_kubernetes_node_label_* and _meta_kubernetes_node_annotation* — used above to reach cAdvisor via the API server proxy.

service

One target per Service port, addressed at the Service’s ClusterIP. Rarely scraped directly (a ClusterIP does not route to a specific pod’s metrics endpoint) but useful for blackbox probing of the Service as a whole. Attaches meta_kubernetes_service_name, meta_kubernetes_service_namespace and _meta_kubernetes_service_annotation*.

endpointslice

One target per address in every EndpointSlice, i.e. per actual pod backing a Service — the modern replacement for the older endpoints role, which read the deprecated Endpoints API and is still supported for compatibility. Attaches meta_kubernetes_endpointslice_* labels plus the backing pod’s own meta_kubernetes_pod_* labels, which is why it is the usual choice for scraping application pods through their Service.

pod

One target per pod, per declared container port, regardless of whether a Service fronts it. Attaches meta_kubernetes_pod_name, meta_kubernetes_pod_namespace, meta_kubernetes_pod_label_*, meta_kubernetes_pod_annotation_* and __meta_kubernetes_pod_container_name. This is the role the annotation-based pattern below is built on.

ingress

One target per Ingress host/path rule, not per backend pod — for blackbox-style monitoring of the routes an Ingress controller exposes, rather than of the workloads behind them. Attaches meta_kubernetes_ingress_name and meta_kubernetes_ingress_scheme.

Annotation-based scraping with the pod role

The common pattern is to scrape only pods that opt in via a prometheus.io/scrape: "true" annotation, using the pod’s own prometheus.io/port and prometheus.io/path annotations (when set) to build the final target address. This is plain relabel_configs work over the labels the pod role attaches — Service Discovery & Relabeling already shows the base version of this pattern (the vs meta_ prefix convention, the scrape/port-rewrite keep/replace steps, and promoting namespace/pod to labels). The one addition this pattern needs on top of that base is honoring an explicit prometheus.io/path annotation instead of always scraping the default /metrics:

relabel_configs:
  # ... the keep/port-rewrite/namespace/pod steps from the base pattern linked above, plus:

  # Honor an explicit prometheus.io/path annotation, default to /metrics.
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)

Swap role: pod for role: endpointslice and the same annotation checks work against _meta_kubernetes_endpointslice*-sourced labels instead, scraping only the pods that back an annotated Service. Container orchestration and the sidecar/DaemonSet layout that typically runs Prometheus and its exporters in-cluster are covered in Exporters & the Pushgateway; recording rules that pre-aggregate the resulting per-pod series are in Recording & Alerting Rules.

kube-state-metrics: cluster-state metrics

kube-state-metrics is a separate service (deployed once per cluster, not per node) that watches the Kubernetes API and exposes the declared and observed state of objects as plain Prometheus metrics — it does not talk to the kubelet and carries no resource-usage numbers at all. Where cAdvisor answers "how much CPU/memory is this container using", kube-state-metrics answers "how many replicas does this Deployment want versus have" or "what phase is this pod in":

# Pods stuck outside the Running phase for more than 10 minutes.
kube_pod_status_phase{phase!="Running"} == 1
and
(time() - kube_pod_created) > 600
# Deployments whose ready replica count has fallen behind the desired count.
kube_deployment_status_replicas_ready
  < on (namespace, deployment)
kube_deployment_spec_replicas

Deploy it as its own scrape target (a Deployment with a Service, discovered like any other workload via the endpointslice or pod role above) rather than trying to derive cluster state from cAdvisor’s per-container numbers, which have no concept of a Deployment’s desired replica count or a pod’s scheduling phase. See kube-state-metrics for the full metric catalog, deployment manifests, and versioning against Kubernetes releases.

Continue with Exporters & the Pushgateway for how exporters like kube-state-metrics and node_exporter fit into a cluster’s overall scrape topology, or Prometheus & Monitoring for the section overview.