Observability: Logging, Metrics and Tracing

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production; see the section bibliography for the reference material consulted while preparing these pages.

A cluster that runs the Bookshelf application produces three kinds of signal: logs (what a process said), metrics (numbers sampled over time) and traces (the path of one request across services). Kubernetes itself defines very little of this pipeline: it keeps container logs on the node, exposes a small resource-metrics API and emits Events, and expects you to add the collection and storage layers. This page is written against Kubernetes 1.37; see the section index for the full version baseline. It builds on Resource Management (requests and limits, which kubectl top and autoscalers read), Health Probes and Lifecycle (what to look at when a Pod restarts), and Autoscaling (the main consumer of the Metrics API). For the Compose and single-host view see Resources, Logging and Monitoring.

Container logs

Kubernetes assumes applications log to standard output and standard error. The container runtime (through the CRI) captures both streams and writes them to files on the node, conventionally under /var/log/pods/ with symlinks under /var/log/containers/. kubectl logs asks the kubelet for those files.

kubectl logs deploy/catalog-api -n bookshelf
kubectl logs catalog-api-7d9c6b5f4-x2k8p -c catalog-api -n bookshelf     # a named container
kubectl logs catalog-api-7d9c6b5f4-x2k8p --previous -n bookshelf         # the crashed instance
kubectl logs -l app=catalog-api --all-containers --prefix --tail=50 -n bookshelf
kubectl logs -f --since=10m --timestamps deploy/reviews-api -n bookshelf

--previous is the first thing to reach for on a CrashLoopBackOff Pod, because the current container has usually only just started. -l selects by label (by default it reads from a limited number of Pods; --max-log-requests raises it). A container’s logs are lost when the Pod is deleted or the node is lost, which is why a cluster-level pipeline is needed for anything you want to keep.

Source: Kubernetes Docs — Logging Architecture.

Rotation

The kubelet rotates container logs so one chatty container cannot fill the node disk. Two kubelet configuration fields control it: containerLogMaxSize (default 10Mi) and containerLogMaxFiles (default 5). Rotation applies to the files the runtime writes for containers; if you write your own log files inside a container, or mount a host directory, rotating them is your responsibility. kubectl logs only serves the latest file, so collectors that tail the files on disk see more history than the API does. The fields are part of the kubelet configuration file:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
containerLogMaxSize: 20Mi
containerLogMaxFiles: 5

Two further fields, containerLogMaxWorkers and containerLogMonitorInterval, tune how many rotations can run concurrently and how often sizes are checked, which matters on nodes with very many log-heavy containers. An alpha feature gate (PodLogsQuerySplitStreams) additionally lets the Pod log API return only the Stdout or Stderr stream; it is not enabled by default.

Source: Kubernetes Docs — Logging Architecture and Kubelet Configuration (v1beta1).

Logging architectures

Kubernetes does not ship cluster-level log storage. The documented patterns differ in where the collector runs.

flowchart LR subgraph N["Node"] C1["App container\nstdout / stderr"] --> F["Node log files\n/var/log/pods"] F --> A["Node-level agent\n(DaemonSet)"] C2["App container\nwrites file in volume"] --> S["Sidecar container\ntails file to its stdout"] S --> F end A --> B[("Log backend\nLoki / Elasticsearch / cloud")] C3["App pushes logs\ndirectly"] --> B

  • Node-level agent. One collector Pod per node, deployed as a DaemonSet, reads the node’s log files and ships them. This is the default choice: no change to the application, one agent per node, and it survives Pod deletion because the files are read as they are written. The agent enriches each line with Pod, namespace and label metadata from the API server.

  • Sidecar. Use it when an application writes to files rather than streams. A streaming sidecar copies a log file to its own stdout so the node agent sees it (and kubectl logs -c <sidecar> works); a shipping sidecar sends the file straight to a backend. A sidecar costs CPU and memory in every Pod, so prefer stdout.

  • Direct push. The application sends logs to the backend itself. It couples code to the backend and loses logs during outages, so it is the least common.

A Fluent Bit DaemonSet

Fluent Bit is a lightweight collector commonly used as the node agent. The official Helm chart installs it as a DaemonSet with RBAC, the Kubernetes metadata filter, and a configurable output.

helm repo add fluent https://fluent.github.io/helm-charts
helm upgrade --install fluent-bit fluent/fluent-bit -n logging --create-namespace -f fluent-bit-values.yaml
kubectl get daemonset fluent-bit -n logging
# fluent-bit-values.yaml (abbreviated)
config:
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
  outputs: |
    [OUTPUT]
        Name loki
        Match kube.*
        Host loki.logging.svc
        Port 3100
        Labels job=fluent-bit

The tail input reads files under /var/log/containers, the multiline.parser joins stack traces, the kubernetes filter adds metadata, and Merge_Log On parses JSON log lines into fields. Replace the loki output with Elasticsearch, OpenSearch or a cloud service as needed. Managed platforms provide their own agents, such as the ones in Azure Kubernetes Service (AKS). Application-side structured JSON logging is covered in Spring Boot Logging.

Source: Kubernetes Docs — Logging Architecture and Fluent Bit Docs — Kubernetes.

Events

Events are API objects recording what controllers and the kubelet did to an object: scheduling decisions, image pulls, probe failures, evictions. They are the first diagnostic for a Pod stuck in Pending or ImagePullBackOff. They are short-lived (retained about an hour by default) and are not a substitute for logs or metrics.

kubectl describe pod catalog-api-7d9c6b5f4-x2k8p -n bookshelf        # Events at the bottom
kubectl events -n bookshelf --for deployment/catalog-api
kubectl get events -n bookshelf --sort-by=.metadata.creationTimestamp
kubectl get events -A --field-selector type=Warning
LAST SEEN   TYPE      REASON             OBJECT                       MESSAGE
12s         Warning   FailedScheduling   pod/reviews-api-5c8d-abcde   0/3 nodes are available: 3 Insufficient memory.

Source: Kubernetes Docs — kubectl events and Kubernetes Docs — Debug Running Pods.

Resource metrics: metrics-server and kubectl top

kubectl top and the Horizontal Pod Autoscaler need current CPU and memory usage. They read it from the Metrics API, metrics.k8s.io, which is served by an add-on, normally metrics-server. The kubelet collects the numbers (cAdvisor on the node) and exposes them at /metrics/resource; metrics-server scrapes every kubelet, keeps only the latest sample in memory, and registers itself with the API aggregation layer so the API server proxies /apis/metrics.k8s.io to it. The Metrics API is stable (metrics.k8s.io/v1) as of 1.37. kind and minikube do not install metrics-server by default.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl get apiservice v1.metrics.k8s.io
kubectl top nodes
kubectl top pods -n bookshelf --containers --sort-by=memory
kubectl get --raw "/apis/metrics.k8s.io/v1/namespaces/bookshelf/pods"
NAME           CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
kind-control-plane   180m   9%       720Mi           9%
kind-worker          95m    4%       510Mi           6%

On kind the kubelet’s serving certificate is self-signed, so metrics-server typically needs the --kubelet-insecure-tls argument in a local cluster; do not use that flag in production. Two limits are worth knowing: metrics-server is not a monitoring system (no history, no per-request metrics, no custom metrics), and kubectl top reports usage, not requests or limits. Custom and external metrics for autoscaling come from other adapters; see Autoscaling.

Source: Kubernetes Docs — Resource metrics pipeline and metrics-server.

Cluster and system metrics

Every Kubernetes component exposes Prometheus-format metrics on /metrics (the kubelet adds /metrics/cadvisor, /metrics/resource and /metrics/probes). Each metric carries a stability level: alpha (may change or disappear at any time), beta (labels are not removed), stable, then deprecated and hidden. Alert on stable metrics where you can.

kube-state-metrics is a separate service that watches the API server and turns object state into metrics, such as kube_deployment_status_replicas_available, kube_pod_container_status_restarts_total and kube_pod_status_phase. It answers "what does the cluster think?", where cAdvisor and metrics-server answer "how much is being consumed?". Both are needed. The Prometheus-side scraping and discovery details are covered in Containers and Kubernetes Monitoring and are not repeated here.

Source: Kubernetes Docs — Metrics For Kubernetes System Components and Kubernetes Docs — Metrics for Kubernetes Object States.

kube-prometheus-stack and Grafana

The kube-prometheus-stack Helm chart installs a complete monitoring stack in one release: the Prometheus Operator and its CRDs, a Prometheus instance, Alertmanager, Grafana, kube-state-metrics, node-exporter, and default scrape configurations, dashboards and alert rules for the control plane and nodes. Instead of editing Prometheus configuration, you declare what to scrape with custom resources (see Custom Resources and Operators).

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace -f monitoring-values.yaml
kubectl get pods -n monitoring
kubectl port-forward svc/monitoring-grafana 3000:80 -n monitoring

The Grafana admin password is created by the chart in a Secret in the release namespace; read it from there instead of putting it in a values file (see Secrets). To scrape the Bookshelf API, add a ServiceMonitor that selects its Service:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: catalog-api
  namespace: bookshelf
  labels:
    release: monitoring          # must match the chart's serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app: catalog-api
  endpoints:
    - port: http
      path: /actuator/prometheus
      interval: 30s

A PodMonitor does the same for Pods without a Service, and a PrometheusRule holds alerting and recording rules. By default the operator only picks up monitors labelled with the release name, the most common reason a new ServiceMonitor "does nothing". The application must expose a metrics endpoint; for Spring Boot that is Actuator with Micrometer, in Metrics and Observability. Dashboards, panels and alerting in Grafana are covered in Grafana Dashboards and Visualization.

Source: kube-prometheus-stack chart and Prometheus Operator Docs.

OpenTelemetry

OpenTelemetry (OTel) is a vendor-neutral standard for emitting traces, metrics and logs, transported with the OTLP protocol. On Kubernetes two pieces matter: the Collector, which receives, processes and exports telemetry, and the OpenTelemetry Operator, which manages Collectors and can inject instrumentation into Pods.

Collector deployment modes

  • DaemonSet (agent). One Collector per node receives OTLP from local Pods, adds node and Pod attributes, and can read node log files and kubelet stats. Low latency, and Pods send to the node-local address.

  • Deployment (gateway). A small, scalable set of Collectors that agents forward to; the place for tail-based sampling, batching, retries and credentials for the backend.

  • Sidecar. One Collector per Pod for strict isolation; expensive, so rarely used.

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: agent
  namespace: observability
spec:
  mode: daemonset
  config:
    receivers:
      otlp:
        protocols:
          grpc: {}
          http: {}
    processors:
      k8sattributes: {}
      batch: {}
    exporters:
      otlp:
        endpoint: gateway-collector.observability.svc:4317
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [k8sattributes, batch]
          exporters: [otlp]

The mode field switches between daemonset, deployment, statefulset and sidecar. The k8sattributes processor needs RBAC to read Pods; the operator manages that for the standard modes. The insecure: true line is for a local lab only.

Auto-instrumentation

The Operator’s Instrumentation resource describes how to instrument applications, and an annotation on a Pod template asks for it. A mutating admission webhook (see Admission Control and Policy) adds an init container and environment variables so the language agent loads at start-up. Java, Python, Node.js, .NET and Go are supported.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: bookshelf
  namespace: bookshelf
spec:
  exporter:
    endpoint: http://agent-collector.observability.svc:4318
  propagators: [tracecontext, baggage]
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
# Pod template metadata of the catalog-api Deployment
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"

The endpoint uses port 4318 because the Java agent exports OTLP over http/protobuf by default; 4317 is the gRPC port that, for example, the Node.js agent uses. The Operator also needs cert-manager (or its own certificate handling) for the webhook. Application-level tracing with Micrometer is an alternative when you control the code.

Source: OpenTelemetry Docs — Kubernetes Operator and OpenTelemetry Docs — Injecting Auto-instrumentation.

System logs and traces

The Kubernetes components log through klog. Verbosity is set with --v (level 2 is the recommended default, higher levels for debugging). Structured logging is available for most components, and a JSON format can be selected through component configuration, which makes component logs easier to parse. Where the logs live depends on how the component is run: with systemd, journalctl -u kubelet; with a kubeadm control plane, static Pods, so kubectl logs -n kube-system kube-apiserver-<node> (or the files under /var/log/pods on the node). Managed clusters expose control-plane logs through the provider.

The kubelet and the API server can also export traces in OTLP format: kubelet tracing is stable since 1.34, while API server tracing is still beta (since 1.27). The kubelet propagates trace context to container runtimes such as containerd and CRI-O, so their spans join the same trace. They are configured with a tracing configuration file that names the OTLP endpoint and sampling rate, pointing at an OpenTelemetry Collector. They are aimed at cluster operators diagnosing control-plane latency, not at application tracing.

journalctl -u kubelet --since "10 minutes ago"
kubectl logs -n kube-system -l component=kube-apiserver --tail=100
kubectl get --raw /metrics | head

Source: Kubernetes Docs — System Logs and Kubernetes Docs — Traces For Kubernetes System Components.

References