Health Probes and Container Lifecycle

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production; see the section bibliography for the reference material consulted while preparing these pages.

A Pod that is merely running is not the same thing as a Pod that is working, or one that is ready to be sent traffic. This page covers the mechanism Kubernetes uses to tell the three apart — probes — and the mechanism it uses to shut a container down cleanly instead of abruptly — lifecycle hooks and graceful termination. It is written against Kubernetes 1.37 (latest patch 1.37.1); see the section index for the full version baseline. It assumes the Pod vocabulary introduced in Pods and builds directly on it: everything here happens to containers inside an already-scheduled Pod.

Liveness, readiness and startup probes

The kubelet can run three kinds of probe against a container, and each answers a different question:

Probe Question it answers On repeated failure

Probe	Question it answers	On repeated failure
`livenessProbe`	"Is this container’s process still healthy, or has it wedged?"	The kubelet kills the container, and it is restarted according to the Pod’s `restartPolicy`.
`readinessProbe`	"Is this container ready to receive traffic right now?"	The Pod’s IP is removed from every Service’s EndpointSlice; the container keeps running, it just stops receiving new requests until it reports ready again.
`startupProbe`	"Has this container finished starting up yet?"	While a startup probe is defined and has not yet succeeded, the kubelet does not run the liveness or readiness probes at all; if the startup probe itself keeps failing past its own `failureThreshold`, the container is killed and restarted, exactly like a failed liveness probe.

livenessProbe

"Is this container’s process still healthy, or has it wedged?"

The kubelet kills the container, and it is restarted according to the Pod’s restartPolicy.

readinessProbe

"Is this container ready to receive traffic right now?"

The Pod’s IP is removed from every Service’s EndpointSlice; the container keeps running, it just stops receiving new requests until it reports ready again.

startupProbe

"Has this container finished starting up yet?"

While a startup probe is defined and has not yet succeeded, the kubelet does not run the liveness or readiness probes at all; if the startup probe itself keeps failing past its own failureThreshold, the container is killed and restarted, exactly like a failed liveness probe.

A container can define all three at once, and normally should for anything with a non-trivial startup: the startup probe covers the slow, one-time initialization window; the liveness and readiness probes then run with short, cheap intervals for the rest of the container’s life.

The four check mechanisms

Every probe (whichever of the three kinds it is) defines exactly one handler, one of:

Mechanism How success is determined

Mechanism	How success is determined
`httpGet`	The kubelet performs an HTTP `GET` against `host` (defaults to the Pod IP), `port` and `path`; any status code in the `200`-`399` range counts as success. `httpHeaders` can add custom headers (for example an `Authorization` header a proxy strips before it reaches the app), and `scheme` selects `HTTP` or `HTTPS`.
`tcpSocket`	The kubelet attempts a TCP connection to `port`; the connection being accepted counts as success, with no application-level check at all. Useful for protocols with no convenient HTTP endpoint (a raw TCP server, or a database before it has one enabled).
`exec`	The kubelet runs `command` inside the container’s namespaces (the same code path as `kubectl exec`); an exit code of `0` counts as success, anything else — including the command not being found — counts as failure.
`grpc`	The kubelet calls the standard gRPC Health Checking Protocol `Check` RPC against `port` (and an optional `service` name for multi-service servers); a `SERVING` status counts as success. This mechanism reached general availability in Kubernetes 1.27 and needs no sidecar such as `grpc-health-probe` — the kubelet speaks the protocol itself.

httpGet

The kubelet performs an HTTP GET against host (defaults to the Pod IP), port and path; any status code in the 200-399 range counts as success. httpHeaders can add custom headers (for example an Authorization header a proxy strips before it reaches the app), and scheme selects HTTP or HTTPS.

tcpSocket

The kubelet attempts a TCP connection to port; the connection being accepted counts as success, with no application-level check at all. Useful for protocols with no convenient HTTP endpoint (a raw TCP server, or a database before it has one enabled).

exec

The kubelet runs command inside the container’s namespaces (the same code path as kubectl exec); an exit code of 0 counts as success, anything else — including the command not being found — counts as failure.

grpc

The kubelet calls the standard gRPC Health Checking Protocol Check RPC against port (and an optional service name for multi-service servers); a SERVING status counts as success. This mechanism reached general availability in Kubernetes 1.27 and needs no sidecar such as grpc-health-probe — the kubelet speaks the protocol itself.

All four mechanisms, and all three probe kinds, are generally available; none of them requires a feature gate on a current cluster.

Timing fields

Every probe shares the same five timing fields, regardless of its kind or mechanism:

Field Default Meaning

Field	Default	Meaning
`initialDelaySeconds`	`0`	Seconds to wait after the container starts before running the probe for the first time.
`periodSeconds`	`10`	How often to repeat the probe.
`timeoutSeconds`	`1`	How long to wait for the probe itself to respond before counting that attempt as a failure.
`successThreshold`	`1` (must be `1` for liveness and startup probes)	Consecutive successes needed, after a failure, before the container is considered healthy/ready again.
`failureThreshold`	`3`	Consecutive failures needed before the probe is considered to have failed and the action in the table above is taken.

initialDelaySeconds

0

Seconds to wait after the container starts before running the probe for the first time.

periodSeconds

10

How often to repeat the probe.

timeoutSeconds

1

How long to wait for the probe itself to respond before counting that attempt as a failure.

successThreshold

1 (must be 1 for liveness and startup probes)

Consecutive successes needed, after a failure, before the container is considered healthy/ready again.

failureThreshold

3

Consecutive failures needed before the probe is considered to have failed and the action in the table above is taken.

A probe can also set its own terminationGracePeriodSeconds, which overrides the Pod-level value from [termination] specifically for the container kill that a failed liveness or startup probe triggers — letting a slow-to-clean-up container get a longer (or, for a probe that should fail fast, shorter) grace period than an ordinary Pod deletion gets.

containers:
  - name: catalog-api
    image: ghcr.io/example/bookshelf-catalog-api:1.0.0
    ports:
      - containerPort: 8080
    startupProbe:
      httpGet:
        path: /actuator/health/liveness
        port: 8080
      failureThreshold: 30 # up to 30 * 5s = 150s to finish starting
      periodSeconds: 5
    livenessProbe:
      httpGet:
        path: /actuator/health/liveness
        port: 8080
      periodSeconds: 10
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /actuator/health/readiness
        port: 8080
      periodSeconds: 5
      failureThreshold: 2

Source: Liveness, Readiness and Startup Probes

Pitfalls

A liveness probe that checks a dependency. A liveness probe should only ask "is this process still able to make progress", never "is my database reachable". If catalog-api’s liveness probe fails whenever PostgreSQL is briefly unavailable, every `catalog-api Pod gets restarted at once — which does nothing to fix the database, removes every healthy instance’s warm caches and connections simultaneously, and can turn one dependency outage into a self-inflicted cascading failure across the whole Deployment. That check belongs in the readiness probe instead: a Pod that cannot reach its database should stop receiving traffic, not be killed.
A slow starter with no startup probe. If a container can legitimately take, say, 90 seconds to warm up, and the only probe defined is a liveness probe with periodSeconds: 10 and failureThreshold: 3 (fails at 30 seconds), the kubelet kills a perfectly healthy, still-starting container in a loop, forever. Raising initialDelaySeconds on the liveness probe to 90 "fixes" that case but then leaves a genuinely wedged process undetected for 90 seconds on every restart. A startupProbe is the correct fix: it owns the slow, variable startup window, and the liveness probe can then keep its normal, short interval for the rest of the container’s life.
Too aggressive timing under load. A timeoutSeconds set for the happy path can start failing the moment the application is under real load (garbage collection pauses, a busy thread pool) rather than actually unhealthy, which is the same cascading-restart problem as the first bullet with a different trigger.

History: Book 1’s pre-v1.0 probes

Book 1 (2015) describes health checks as they existed just before Kubernetes 1.0, and two things have changed since:

The book documents only a liveness check (a "process check", plus TCP, HTTP and exec handlers driving container restarts); a separate readiness probe, controlling only Service membership rather than restarts, did not exist yet at the time it describes and was added shortly afterwards. The startupProbe kind is newer still (stable since Kubernetes 1.20).
The book describes reaching application-specific health beyond a fixed set of mechanisms through a custom "prober" process that inspects the application and relabels Pods to add or remove them from a Service’s selector. That idea is exactly what the exec and grpc mechanisms above make unnecessary: a probe can now run arbitrary application logic (a shell command, or a gRPC health check) directly as part of the readiness probe itself, with the kubelet doing the relabelling-equivalent work (removing the Pod from EndpointSlices) natively, rather than a bespoke side process editing labels.

Lifecycle hooks and graceful termination

postStart and preStop hooks

Besides probes, a container can define two lifecycle hooks, each with the same exec, httpGet or sleep handler choices described in [lifecycle-hooks-table]:

Hook When it runs

Hook	When it runs
`postStart`	Immediately after the container is created, concurrently with the container’s own entrypoint — there is no guarantee the application has finished starting, or even started at all, when `postStart` runs. The container does not report as `Running` until the hook returns, so a slow `postStart` hook delays Pod startup the same way a slow startup probe would.
`preStop`	Immediately before the container is sent `SIGTERM`, whether the Pod is being deleted, evicted, or the container is being restarted after a failed liveness/startup probe. `preStop` blocks the shutdown sequence: `SIGTERM` is not sent until the hook returns (or fails), which is exactly what makes it useful for a controlled drain (see [termination]).

postStart

Immediately after the container is created, concurrently with the container’s own entrypoint — there is no guarantee the application has finished starting, or even started at all, when postStart runs. The container does not report as Running until the hook returns, so a slow postStart hook delays Pod startup the same way a slow startup probe would.

preStop

Immediately before the container is sent SIGTERM, whether the Pod is being deleted, evicted, or the container is being restarted after a failed liveness/startup probe. preStop blocks the shutdown sequence: SIGTERM is not sent until the hook returns (or fails), which is exactly what makes it useful for a controlled drain (see [termination]).

Both hooks accept a sleep action — a plain fixed pause, with no command or HTTP call — alongside exec and httpGet:

lifecycle:
  preStop:
    sleep:
      seconds: 5

A sleep preStop hook is the simplest possible way to buy time between a Pod being marked for deletion and the application receiving SIGTERM, without writing or shipping any script — see [termination] for why that gap matters. Hook delivery is at least once: a hook can, in rare cases (for example a kubelet restart while delivering it), run more than once, so a preStop/postStart handler should be safe to run twice. If a hook fails, the container is killed as a FailedPostStartHook / FailedPreStopHook Pod event.

Source: Container Lifecycle Hooks

Graceful termination

Deleting a Pod is not an instant kill. [termination-sequence] shows one step of a rolling update: the replacement Pod must pass its readiness probe and join the EndpointSlices before an old Pod is deleted, and the old Pod then goes through graceful termination. The two timelines that run in parallel at the start of that termination are the part most often gotten wrong:

sequenceDiagram autonumber participant RS as Deployment / ReplicaSet controllers participant API as kube-apiserver participant NewK as kubelet (new Pod's node) participant EPC as EndpointSlice controller participant KP as kube-proxy (every node) participant Kubelet as kubelet (old Pod's node) participant App as Old container process RS->>API: create new Pod from the new ReplicaSet (maxSurge) API-->>NewK: new Pod scheduled NewK->>NewK: start container, startupProbe then readinessProbe succeed NewK->>API: new Pod condition Ready=True API-->>EPC: Pod update (Ready) EPC->>API: add new Pod to every EndpointSlice RS->>API: DELETE one old Pod (scale down the old ReplicaSet) API->>API: set deletionTimestamp (shown as Terminating) par Endpoint removal API-->>EPC: Pod update (Terminating) EPC->>API: remove Pod from every EndpointSlice API-->>KP: EndpointSlice changed KP->>KP: reprogram node rules, stop sending new traffic to this Pod and preStop hook API-->>Kubelet: Pod update (Terminating) Kubelet->>App: run preStop hook (exec / httpGet / sleep) App-->>Kubelet: hook returns (or terminationGracePeriodSeconds elapses) end Kubelet->>App: SIGTERM App->>App: stop accepting new work, finish in-flight requests, exit alt exits before grace period elapses App-->>Kubelet: process exits else grace period elapses first Kubelet->>App: SIGKILL end Kubelet->>API: Pod fully terminated, remove from etcd

Endpoint removal and the preStop hook run in parallel, not one after the other. Endpoint removal has to propagate through the EndpointSlice controller, to every node’s kube-proxy, before traffic actually stops arriving — and that propagation is not instant. This is the endpoint-removal race: if SIGTERM were sent the moment the Pod is marked Terminating, the application could shut down before some nodes have finished removing it from their load-balancing rules, and a request could be routed to a Pod that is no longer there to answer it. A preStop hook that simply sleeps for a few seconds (the sleep action above) is the standard mitigation: it delays SIGTERM, and therefore delays the application actually stopping, long enough for endpoint removal to have finished everywhere, without the application needing to know anything about EndpointSlices at all.
terminationGracePeriodSeconds (Pod-level, default 30 seconds) is the total budget: it covers the preStop hook and the time the application takes to shut down after SIGTERM, combined. A preStop hook that sleeps for 10 seconds and an application that then takes 25 seconds to drain will together exceed a 30-second grace period and get SIGKILL`ed mid-shutdown — raise `terminationGracePeriodSeconds on the Pod spec (not just the probe-level override from Liveness, readiness and startup probes) if the two combined need more room.
SIGKILL after the grace period elapses is unconditional and gives the process no further chance to clean up; a container that needs more time than that should get a longer grace period, not rely on catching SIGKILL (which cannot be caught at all).

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: catalog-api
      image: ghcr.io/example/bookshelf-catalog-api:1.0.0
      lifecycle:
        preStop:
          sleep:
            seconds: 10

Source: Pod Lifecycle (Termination of Pods)

Readiness gates

A readinessProbe is not always the whole story: sometimes a Pod’s readiness genuinely depends on a condition an external controller determines, not something the container itself can probe (for example, a load-balancer controller confirming the Pod has actually been registered as a healthy backend). spec.readinessGates names one or more extra condition types that must also be True before the Pod is considered ready, alongside the built-in Ready condition and any readinessProbe:

spec:
  readinessGates:
    - conditionType: "www.example.com/load-balancer-registered"

An external controller sets that condition on the Pod’s status.conditions once it has confirmed whatever it is responsible for; until it does, the Pod counts as not ready even if every container’s own readiness probe is passing. This is the same mechanism Gateway API implementations and some ingress controllers use to make sure a Pod only receives traffic once it is actually wired into the data plane, not just once the application says it is ready.

Source: Pod Lifecycle (Pod readiness gate)

Application-side probe endpoints

Probes only work if the application exposes something meaningful to check against. Two examples, one per runtime already used elsewhere on this site:

Spring Boot Actuator

Spring Boot’s Actuator module ties directly into the liveness/readiness distinction above through its health groups, once management.endpoint.health.probes.enabled is turned on:

management:
  endpoint:
    health:
      probes:
        enabled: true
      group:
        readiness:
          include: readinessState,db   # opt the DataSource health indicator into readiness
  health:
    livenessstate:
      enabled: true
    readinessstate:
      enabled: true

With that in place, /actuator/health/liveness reports the application’s own liveness state (a deadlock or a broken internal loop), separately from /actuator/health/readiness. By default each group contains only its own state indicator (livenessState or readinessState); external HealthIndicator checks — a database connection pool, a message broker, a downstream client — are not included. Add them to readiness explicitly with management.endpoint.health.group.readiness.include, as the db entry above does, and never add them to the liveness group: a database outage would then restart every replica. The Pod spec then simply points each probe at its matching endpoint, as in the startupProbe/livenessProbe/readinessProbe example earlier on this page. See Metrics and Observability for how the broader /actuator family (metrics, tracing, info) is configured and secured alongside these two endpoints.

Source: Spring Boot — Kubernetes Probes

FastAPI

A minimal FastAPI service can expose the same two endpoints directly (a startupProbe can reuse /healthz/live), and use a lifespan handler for its startup and shutdown work:

from contextlib import asynccontextmanager
from fastapi import FastAPI, Response

state = {"ready": False}


@asynccontextmanager
async def lifespan(app: FastAPI):
    # startup: open connection pools, warm caches, ...
    state["ready"] = True
    yield
    # shutdown: runs after Uvicorn has finished the in-flight requests
    state["ready"] = False
    # close connection pools, flush buffers, ...


app = FastAPI(lifespan=lifespan)


@app.get("/healthz/live")
def liveness():
    return {"status": "ok"}


@app.get("/healthz/ready")
def readiness(response: Response):
    if not state["ready"]:
        response.status_code = 503
        return {"status": "starting"}
    return {"status": "ok"}

The application does not install its own SIGTERM handler. Uvicorn already handles SIGTERM gracefully: it stops accepting new connections, waits for in-flight requests to finish, runs the lifespan shutdown code and exits. Replacing that handler with signal.signal(signal.SIGTERM, …) would stop Uvicorn from ever shutting down, and the container would be SIGKILL`ed when the grace period from [termination] runs out. Stopping new traffic is Kubernetes' job: the terminating Pod is removed from the EndpointSlices, and the `preStop sleep gives that removal time to finish before SIGTERM arrives.

Source: FastAPI — Lifespan Events and Liveness, Readiness and Startup Probes