Debugging and Troubleshooting
|
This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production; see the section bibliography for the reference material consulted while preparing these pages. |
Most Kubernetes problems fall into a small number of patterns, and the fastest way to find the pattern is to work
through the request path in a fixed order instead of guessing. This page gives that order, the tools for each step
(describe, events, logs, kubectl debug, get --raw, audit logs, crictl), a table of the common failure
states with a reproduction and a fix for each, and a decision tree. It is written against Kubernetes 1.37 and
kubectl 1.37; see the section index for the version baseline. All examples
use the Bookshelf application from Getting Started with Kubernetes
in the bookshelf namespace. Command basics are in kubectl
Essentials, and logs and metrics are covered in
Observability: Logging, Metrics and Tracing.
A systematic flow
A request reaches a container through several layers, and each layer can fail independently. Debug from the workload outward and stop at the first layer that is unhealthy:
-
Pod. Is it scheduled, are its containers running, and is it Ready? See Pods and Health Probes and Lifecycle.
-
Service. Does the selector match the Pod’s labels, so that the Service has ready endpoints? See Services.
-
DNS. Does the Service name resolve from another Pod? See DNS and Service Discovery.
-
Gateway or Ingress. Is the route accepted, is its backend resolved, and does the controller have an address? See Ingress and Gateway API.
-
Network policy. Is traffic denied between the two ends? See Network Policies.
-
Node and control plane. Is the node Ready, is the runtime healthy, is the API server responding?
The 2015 and 2017 books teach essentially this order (check the Pod, then the Service and its selector), but they
predate kubectl debug, EndpointSlices and ephemeral containers; the tooling below is the current equivalent.
Start with status, describe and events
Three commands answer most questions. get shows the state, describe shows the state plus the recent events
for the object, and logs shows what the process wrote.
kubectl get pods -n bookshelf -o wide
kubectl describe pod catalog-api-7d9c6b5f4-x2k8p -n bookshelf
kubectl get events -n bookshelf --sort-by=.lastTimestamp
kubectl get events -n bookshelf --field-selector type=Warning
kubectl logs catalog-api-7d9c6b5f4-x2k8p -n bookshelf -c catalog-api
kubectl logs catalog-api-7d9c6b5f4-x2k8p -n bookshelf -c catalog-api --previousIn describe, read three sections first: State / Last State / Reason / Exit Code for each container,
Conditions (PodScheduled, Initialized, ContainersReady, Ready), and the Events table at the bottom.
--previous shows the logs of the last terminated container, which is the one that matters for a crash loop.
Events are namespaced, have a short retention (about an hour by default), and are emitted by the scheduler, the
kubelet and the controllers, so check them soon after the failure.
To read a single field without parsing the full output, -o jsonpath works well. The template is shown in a
listing block so its braces are not interpreted by the documentation toolchain:
kubectl get pod catalog-api-7d9c6b5f4-x2k8p -n bookshelf \
-o jsonpath='{.status.containerStatuses[0].lastState.terminated}'If the container wrote a termination message, describe shows it under Last State; it comes from the file named
by terminationMessagePath (default /dev/termination-log).
kubectl debug
Production images are often minimal or distroless, with no shell, so kubectl exec has nothing to run. kubectl
debug fixes that in three ways.
Ephemeral containers
An ephemeral container is added to a running Pod without restarting it. The feature has been stable since
Kubernetes 1.25. The container shares the Pod’s network and volumes, and with --target it also joins the process
namespace of the named container, so you can inspect its processes (the container runtime must support this).
Ephemeral containers cannot be removed or changed once added, and have no probes or resources.
kubectl debug -it catalog-api-7d9c6b5f4-x2k8p -n bookshelf \
--image=busybox:1.37 --target=catalog-apiInside, ps, wget -qO- localhost:8080/healthz and cat /proc/1/environ (via /proc/<pid>/root) let you look at
the target from its own network namespace.
Debugging a copy of the Pod
When a container crashes on start, an ephemeral container in it is of little use. Make a copy of the Pod with a changed command or image, leaving the original untouched:
# Copy, keep the image, but replace the command so it does not crash
kubectl debug catalog-api-7d9c6b5f4-x2k8p -n bookshelf -it \
--copy-to=catalog-api-debug --container=catalog-api -- sh
# Copy, swap every container's image for a debug image, and share the process namespace
kubectl debug catalog-api-7d9c6b5f4-x2k8p -n bookshelf -it \
--copy-to=catalog-api-debug --set-image='*=ubuntu:24.04' --share-processes
kubectl delete pod catalog-api-debug -n bookshelfThe copy is a separate bare Pod, so it is not managed by the Deployment and must be deleted by hand. It is not selected by the Service unless it still carries the matching labels.
Debugging a node
kubectl debug node/<name> creates a Pod on that node that runs in the host IPC, network and PID namespaces with
the node’s root filesystem mounted at /host:
kubectl debug node/kind-worker -it --image=ubuntu:24.04
# inside: ls /host/var/log ; chroot /host (chroot may fail for an unprivileged Pod)
kubectl delete pod node-debugger-kind-worker-abcdeThe Pod is not privileged by default. Use a profile to change that.
Profiles
--profile selects a predefined set of security settings for the debug container. general is the default; the
older legacy profile (which kept the 1.22 behaviour) is no longer offered. Setting the profile explicitly still
makes the intent clear:
| Profile | Meaning |
|---|---|
|
A reasonable set of generic properties for most debugging (the default). |
|
Compatible with the corresponding Pod Security Standard, for namespaces that enforce it. |
|
Adds network-administration capabilities (packet capture, routing). |
|
Root-level privileges, like a privileged container; use sparingly. |
kubectl debug -it catalog-api-7d9c6b5f4-x2k8p -n bookshelf \
--image=busybox:1.37 --target=catalog-api --profile=general
kubectl debug --profile=sysadmin node/kind-worker -it --image=ubuntu:24.04In namespaces that enforce the restricted Pod Security Standard
(Pod Security), use --profile=restricted or the debug container will be
rejected. A --custom file can also patch parts of the debug container spec.
The API server and the node
When the workload looks fine, check the machinery around it.
kubectl get --raw and cluster health
kubectl get --raw sends a raw request to any API path, which is handy for health endpoints and for seeing what
the server itself reports:
kubectl get --raw='/readyz?verbose'
kubectl get --raw='/livez?verbose'
kubectl get --raw=/metrics | head
kubectl get --raw=/apis/metrics.k8s.io/v1/nodes
kubectl get nodes
kubectl describe node kind-worker
kubectl cluster-info dump --namespaces bookshelf --output-directory=./cluster-dumpdescribe node shows node conditions (Ready, MemoryPressure, DiskPressure, PIDPressure), allocatable
capacity and the non-terminated Pods with their requests, which explains many scheduling failures.
Audit logs
The API server can write an audit log recording who did what to which object, configured with an audit policy
file (--audit-policy-file) and a log backend (--audit-log-path) or webhook backend. It answers "who deleted this
Deployment" or "why was this request denied", and is the place to look for RBAC failures after the fact
(Authentication, Authorization and RBAC). On
managed services the audit log is exposed through the provider’s logging product instead of a file.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
resources:
- group: ""
resources: ["secrets", "configmaps"]
- level: RequestResponse
namespaces: ["bookshelf"]
verbs: ["delete"]
- level: None
users: ["system:kube-proxy"]Source: Kubernetes Docs — Auditing.
crictl on the node
If the kubelet or the container runtime misbehaves, the API may not tell the whole story. crictl talks to the CRI
runtime (containerd or CRI-O) directly on the node, for example from a kubectl debug node session or over SSH. It
works on containers that the kubelet knows about, regardless of the API server:
crictl pods --name catalog-api
crictl ps -a
crictl logs <container-id>
crictl inspect <container-id>
crictl images
crictl stats
journalctl -u kubelet --since "10 min ago"Point crictl at the runtime socket with --runtime-endpoint or /etc/crictl.yaml. Containers started with
docker are not visible to it. On a kind node, run these commands inside the node container, for example with
docker exec -it kind-worker crictl ps -a.
Common failure states
The table lists the states you will meet most often, how to reproduce each in the Bookshelf namespace, what the symptom looks like and how to fix it.
| State | Reproduce | Diagnose |
|---|---|---|
Fix |
|
Set |
|
Lower the request, add nodes or free capacity; see Resource Management. |
|
Add |
|
Label the node, add a toleration, or relax the rule; see Scheduling and Disruptions. |
|
Reference a PVC that names a missing StorageClass. |
|
Create the StorageClass or PV; see Persistent Volumes and Storage Classes. |
|
Use |
Events show |
Fix the name or tag; for private registries create an image pull secret and set |
|
Start the container with |
|
Fix the application or its config; tune probes if a liveness probe kills a slow start. |
|
Set |
|
Raise the memory limit or cap the heap to fit; see Resource Management. |
|
Reference a missing ConfigMap key or Secret with |
Event |
Create the object or key; see ConfigMaps and Secrets. |
Readiness failing |
Point the readiness probe at a wrong path or port. |
Pod |
Correct the probe, or fix the dependency it checks; raise |
Service without endpoints |
Set the Service selector to |
|
Make |
DNS failure |
Delete or scale CoreDNS to zero in a test cluster, or use a misspelled service name. |
|
Check the CoreDNS Pods and logs, the name and namespace, and that a NetworkPolicy does not block UDP/TCP 53. |
NetworkPolicy block |
Apply a default-deny ingress policy to |
Connection hangs until a timeout (not refused); works from a Pod excluded by the policy. |
Add an allow rule matching the client’s labels and port; see Network Policies. |
A few reproduction manifests and the matching checks follow. A Pod that cannot be scheduled because of an unsatisfiable request:
apiVersion: v1
kind: Pod
metadata:
name: too-big
namespace: bookshelf
spec:
containers:
- name: app
image: ghcr.io/example/bookshelf-catalog-api:1.0.0
resources:
requests:
cpu: "64"kubectl apply -f too-big.yaml
kubectl get pod too-big -n bookshelf
kubectl describe pod too-big -n bookshelf | sed -n '/Events:/,$p'NAME READY STATUS RESTARTS AGE
too-big 0/1 Pending 0 8s
Warning FailedScheduling ... 0/3 nodes are available: 3 Insufficient cpu.A Service whose selector matches no Pods, and how to confirm it:
kubectl get svc catalog-api -n bookshelf -o wide
kubectl get endpointslices -n bookshelf -l kubernetes.io/service-name=catalog-api
kubectl get pods -n bookshelf --show-labels
kubectl get pods -n bookshelf -l app=catalog-apiA DNS and connectivity check from a throwaway debug Pod, then a policy check:
kubectl run netcheck -n bookshelf --rm -it --restart=Never --image=busybox:1.37 -- \
sh -c 'nslookup catalog-api.bookshelf.svc.cluster.local; wget -qO- -T 3 http://catalog-api:8080/healthz'
kubectl get networkpolicy -n bookshelf
kubectl describe networkpolicy default-deny-ingress -n bookshelf
kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=30A timeout (as opposed to "connection refused") from an otherwise healthy Pod points to a NetworkPolicy or a firewall, and an immediate refusal points to nothing listening on that port. Note that `kind’s default CNI (kindnet) does enforce NetworkPolicies in current releases, but check your CNI (Cluster Networking and CNI) before relying on it.
For a Gateway or Ingress, read the status the controller writes back:
kubectl describe gateway bookshelf-gw -n bookshelf
kubectl describe httproute web -n bookshelf
kubectl describe ingress web -n bookshelfAn HTTPRoute reports Accepted and ResolvedRefs conditions per parent; ResolvedRefs=False usually means the
backend Service or port is wrong, and a missing address on the Gateway means its controller is not running.