Resource Management

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production; see the section bibliography for the reference material consulted while preparing these pages.

Every container in a cluster competes for the same finite CPU, memory, disk and process slots. Kubernetes handles that competition with two declarations per container — a request (what the scheduler reserves) and a limit (what the kubelet and the container runtime enforce) — and a small set of namespace-level policies that cap what a team may ask for. This page is written against Kubernetes 1.37; see the section index for the version baseline. It builds on Pods and feeds directly into Scheduling and Disruptions and Autoscaling, both of which consume requests as their input.

Source: Kubernetes Docs — Resource Management for Pods and Containers.

Requests and limits

Where docker run takes --memory and --cpus flags, Kubernetes declares the equivalent limits, plus scheduling requests, in the Pod spec.

The two fields live under resources in each container and are expressed per resource name:

cpu is measured in CPU units: 1 is one core (or hyperthread); 500m is half a core (m = millicpu).
memory is measured in bytes with binary suffixes: 256Mi, 1Gi. Beware that 256M (decimal) differs from 256Mi.
ephemeral-storage covers the container’s writable layer, logs and emptyDir volumes (see Volumes and Ephemeral Storage).
Extended resources such as GPUs (nvidia.com/gpu) can only be requested as whole numbers, and requests must equal limits (see Dynamic Resource Allocation and Devices).

The request is used by the scheduler: a Pod only fits on a node whose unreserved allocatable capacity covers the sum of its containers' requests. Requests do not measure actual usage; a node can be "full" on paper and idle in practice. The limit is enforced at run time, and the two resource types behave very differently when the limit is reached:

Resource Enforced how When the limit is exceeded

Resource	Enforced how	When the limit is exceeded
CPU (compressible)	The kernel CFS quota (`cpu.max` in cgroup v2) caps CPU time per scheduling period.	The container is throttled: it keeps running, only slower. Latency spikes are the symptom, not restarts.
Memory (incompressible)	The cgroup memory limit (`memory.max`).	The kernel OOM killer terminates a process; the container shows `OOMKilled` (exit code 137) and is restarted per the Pod’s `restartPolicy`.
Ephemeral storage	The kubelet measures usage periodically.	The Pod is evicted.

CPU (compressible)

The kernel CFS quota (cpu.max in cgroup v2) caps CPU time per scheduling period.

The container is throttled: it keeps running, only slower. Latency spikes are the symptom, not restarts.

Memory (incompressible)

The cgroup memory limit (memory.max).

The kernel OOM killer terminates a process; the container shows OOMKilled (exit code 137) and is restarted per the Pod’s restartPolicy.

Ephemeral storage

The kubelet measures usage periodically.

The Pod is evicted.

A container with a CPU request but no CPU limit can use idle CPU on the node, and is guaranteed its requested share when the node is busy (requests become cgroup weights). Many teams therefore set CPU requests but omit CPU limits to avoid throttling, while always setting memory requests equal to limits.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog-api
  namespace: bookshelf
spec:
  replicas: 2
  selector:
    matchLabels:
      app: catalog-api
  template:
    metadata:
      labels:
        app: catalog-api
    spec:
      containers:
        - name: catalog-api
          image: ghcr.io/example/bookshelf-catalog-api:1.0.0
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
              ephemeral-storage: 100Mi
            limits:
              memory: 256Mi            # memory request == limit, no CPU limit
              ephemeral-storage: 500Mi

Source: Kubernetes Docs — Requests and limits.

kubectl top pod -n bookshelf                       # needs metrics-server
kubectl describe node kind-worker | sed -n '/Allocated resources/,/Events/p'
kubectl get pod catalog-api-xyz -n bookshelf \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

Allocated resources:
  Resource           Requests     Limits
  cpu                750m (9%)    0 (0%)
  memory             768Mi (4%)   768Mi (4%)

Source: Kubernetes Docs — Resource metrics pipeline.

Quality of Service classes

From the requests and limits of all containers, Kubernetes derives a QoS class stored in status.qosClass. It decides eviction order under node pressure and the kernel OOM score.

Class Rule Under pressure

Class	Rule	Under pressure
`Guaranteed`	Every container has CPU and memory requests equal to limits (Pod-level resources can satisfy this too).	Evicted last; lowest OOM score.
`Burstable`	At least one container has a request or limit, but the Guaranteed rule is not met.	Evicted in order of how far usage exceeds requests.
`BestEffort`	No requests or limits anywhere.	Evicted first.

Guaranteed

Every container has CPU and memory requests equal to limits (Pod-level resources can satisfy this too).

Evicted last; lowest OOM score.

Burstable

At least one container has a request or limit, but the Guaranteed rule is not met.

Evicted in order of how far usage exceeds requests.

BestEffort

No requests or limits anywhere.

Evicted first.

Source: Kubernetes Docs — Pod Quality of Service Classes.

Pod-level resources

Setting spec.resources on the Pod (feature PodLevelResources, beta since 1.34) declares one budget shared by all containers, which suits a main container plus short-lived sidecars whose individual peaks do not coincide. Container values may still be set within that total. Verify the current maturity on the feature-gates page before relying on it; the fields are not available on clusters where the gate is off.

apiVersion: v1
kind: Pod
metadata:
  name: reviews-api
  namespace: bookshelf
spec:
  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: "1"
      memory: 512Mi
  containers:
    - name: reviews-api
      image: ghcr.io/example/bookshelf-reviews-api:1.0.0
    - name: log-shipper
      image: ghcr.io/example/bookshelf-log-shipper:1.0.0

Source: Kubernetes Docs — Pod-level resource specifications.

LimitRange: defaults and bounds per container

A LimitRange is a namespace policy, applied at admission, that supplies default requests/limits to containers that omit them and rejects values outside a min/max range. It is the simplest way to stop BestEffort Pods appearing in a shared namespace.

apiVersion: v1
kind: LimitRange
metadata:
  name: container-defaults
  namespace: bookshelf
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      default:
        memory: 256Mi
      max:
        memory: 2Gi
      min:
        cpu: 10m
        memory: 16Mi

Source: Kubernetes Docs — Limit Ranges.

LimitRange acts only on objects created after it exists; it never changes running Pods. Limits can also target Pod and PersistentVolumeClaim (storage size) objects.

ResourceQuota: aggregate caps per namespace

A ResourceQuota limits the total a namespace may consume, and the number of objects it may create. Once a quota constrains requests.cpu or limits.memory, every Pod must declare those fields (or get them from a LimitRange), otherwise creation is rejected.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: bookshelf-quota
  namespace: bookshelf
spec:
  hard:
    persistentvolumeclaims: "10"
    count/deployments.apps: "20"
    count/configmaps: "50"
    services.loadbalancers: "0"
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: bookshelf-standard-compute
  namespace: bookshelf
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.memory: 12Gi
  scopeSelector:
    matchExpressions:
      - operator: In
        scopeName: PriorityClass
        values: ["standard"]

Source: Kubernetes Docs — Resource Quotas.

Quota scopes (Terminating, NotTerminating, BestEffort, NotBestEffort, PriorityClass, CrossNamespacePodAffinity) restrict which objects a quota counts. A scoped quota may only track Pods and compute resources, and only counts Pods that match the scope, so object counts such as persistentvolumeclaims or services.loadbalancers go into a separate, unscoped quota, as above. Check state with kubectl describe quota -n bookshelf. Quotas and LimitRanges together are the soft-tenancy building blocks discussed in Admission Control and Policy.

PID limits, Pod overhead and node allocatable

PID limiting. A fork bomb in one container can exhaust the node’s process table. The kubelet’s podPidsLimit caps processes per Pod, and pid reservations under systemReserved/kubeReserved (for example systemReserved: {pid: "1000"}) protect system daemons. Source: Kubernetes Docs — Process ID Limits And Reservations.
Pod overhead. Sandboxed runtimes (Kata, gVisor — see Dynamic Resource Allocation and Devices) consume memory beyond the containers. A RuntimeClass with overhead.podFixed adds that amount to the Pod’s requests for scheduling and quota. Source: Kubernetes Docs — Pod Overhead.
Node allocatable. A node’s allocatable is capacity minus kubeReserved, systemReserved and the eviction threshold. The scheduler only ever uses allocatable, which is why a 16 GiB node never offers 16 GiB. Inspect it with kubectl get node <name> -o jsonpath='{.status.allocatable}'. Source: Kubernetes Docs — Reserve Compute Resources for System Daemons.

Swap, cgroup v2 and cgroup v1

Kubernetes has historically required swap to be off. Node swap support (NodeSwap) reached stable in 1.34 in the LimitedSwap mode, where Burstable Pods may use swap in proportion to their memory request while Guaranteed and BestEffort Pods may not; it remains opt-in via the kubelet’s memorySwap setting and requires cgroup v2.

cgroup v2 is the recommended and default hierarchy on modern distributions (and on kind node images) and is what powers memory QoS, swap and pressure-stall information. cgroup v1 support is deprecated (since 1.35) and in maintenance mode: the kubelet refuses to start on a cgroup v1 node by default unless its failCgroupV1 setting is explicitly false, and new features are not implemented for v1, so plan node images and container runtimes (containerd 1.4+, CRI-O 1.20+) around v2. Check a node with stat -fc %T /sys/fs/cgroup/ (cgroup2fs means v2).

Source: Kubernetes Docs — About cgroup v2.

Requests to scheduling, limits to enforcement

flowchart LR subgraph Declare["Container spec"] REQ["requests\ncpu, memory, ephemeral-storage"] LIM["limits\ncpu, memory, ephemeral-storage"] end REQ --> SCHED["kube-scheduler\nfits sum of requests\ninto node allocatable"] REQ --> QUOTA["ResourceQuota\ncounts requests"] REQ --> CGW["cgroup weights\ncpu.weight, memory.low"] LIM --> CGL["cgroup limits\ncpu.max, memory.max"] CGL --> THR["CPU: throttled"] CGL --> OOM["Memory: OOM kill\nexit code 137"] REQ --> QOS["QoS class\nGuaranteed, Burstable, BestEffort"] LIM --> QOS QOS --> EVICT["kubelet eviction order\nunder node pressure"]

Right-sizing

Requests are guesses until measured. Compare kubectl top (and the Prometheus series described in the Prometheus containers and Kubernetes monitoring page) with declared requests, then adjust. The Vertical Pod Autoscaler can run in recommendation-only mode (updateMode: "Off") and publish suggested requests in its status without touching Pods; in-place Pod resize (stable in 1.35) lets requests and limits change without recreating the Pod. Horizontal scaling based on CPU utilization is computed against requests, so inflated requests silently make an HPA less sensitive (see Autoscaling).

Requests also govern bin-packing: the scheduler places Pods like items in bins, and each over-generous request strands capacity that other workloads could have used. Rensin’s 2015 book pictured this as cluster-wide resource scheduling across many hosts; today the same idea is the practical argument for accurate requests, for Guaranteed QoS only where latency demands it, and for a cluster autoscaler or Karpenter to add or remove nodes to match the sum of requests.

Source: Kubernetes Docs — Vertical Pod autoscaling.