Scheduling and Disruptions
|
This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production; see the section bibliography for the reference material consulted while preparing these pages. |
Scheduling is the decision of which node runs each Pod; disruption is what happens when a node or Pod has to go away
again. Rensin’s 2015 book frames Kubernetes itself as a scheduler for the cloud, and that remains the core of the system:
the scheduler is a control loop that watches for Pods without a spec.nodeName and binds them. This page is written
against Kubernetes 1.37; see the section index for the version baseline. Its input
is the requests described in Resource Management.
How kube-scheduler works
For each unscheduled Pod the scheduler runs a scheduling cycle followed by a binding cycle:
-
Filter — discard nodes that cannot run the Pod: insufficient allocatable resources, unmatched
nodeSelectoror required affinity, untolerated taints, unbound volumes that cannot be satisfied there, and so on. -
Score — rank the remaining feasible nodes (spread, image locality, preferred affinity, balanced resource use).
-
Bind — pick the highest score and write the binding to the API server; the kubelet on that node then starts the Pod.
If no node is feasible the Pod stays Pending and kubectl describe pod shows a FailedScheduling event explaining
which filters rejected how many nodes. Every step is a plugin attached to an extension point (PreFilter, Filter,
Score, Reserve, Permit, Bind, …) of the scheduling framework, configured through a KubeSchedulerConfiguration
with profiles; that is how alternative policies and out-of-tree plugins are added.
kubectl get pods -n bookshelf -o wide # which node each Pod landed on
kubectl describe pod notifier-7c9d -n bookshelf | sed -n '/Events/,$p'
kubectl get events -n bookshelf --field-selector reason=FailedSchedulingWarning FailedScheduling 0/3 nodes are available: 1 node(s) had untolerated taint
{node-role.kubernetes.io/control-plane: }, 2 Insufficient memory.Steering placement
nodeSelector
The simplest constraint: the Pod runs only on nodes carrying all listed labels. kind nodes carry
kubernetes.io/hostname, and you can add your own with kubectl label node kind-worker disktype=ssd.
spec:
nodeSelector:
disktype: ssdSource: Kubernetes Docs — nodeSelector.
Node affinity
Node affinity expresses the same idea with operators (In, NotIn, Exists, DoesNotExist, Gt, Lt) and two
strengths: requiredDuringSchedulingIgnoredDuringExecution (a hard filter) and
preferredDuringSchedulingIgnoredDuringExecution (a weighted score). "IgnoredDuringExecution" means running Pods are not
moved if labels later change.
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/arch
operator: In
values: ["amd64", "arm64"]
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 50
preference:
matchExpressions:
- key: disktype
operator: In
values: ["ssd"]Source: Kubernetes Docs — Node affinity.
Pod affinity and anti-affinity
Pod (anti-)affinity is evaluated against the labels of other Pods within a topology domain named by topologyKey.
Affinity co-locates (for example web next to catalog-api to save a network hop); anti-affinity separates (two
catalog-api replicas should not share a node). It is expensive on large clusters, so prefer topology spread constraints
for plain replica spreading.
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app: catalog-apiTaints and tolerations
Where affinity attracts Pods to nodes, a taint repels them: a node says "do not schedule here unless tolerated". A taint has a key, an optional value and an effect:
| Effect | Meaning |
|---|---|
|
New Pods without a matching toleration are not scheduled on the node. |
|
The scheduler tries to avoid the node but may use it. |
|
As |
kubectl taint nodes kind-worker2 dedicated=batch:NoSchedule
kubectl taint nodes kind-worker2 dedicated=batch:NoSchedule- # removespec:
tolerations:
- key: dedicated
operator: Equal
value: batch
effect: NoScheduleA toleration only permits scheduling on a tainted node; pair it with a nodeSelector or affinity to actually force
placement there, for instance to keep the notifier worker on batch nodes. Control-plane nodes carry the
node-role.kubernetes.io/control-plane:NoSchedule taint, which is why workload Pods avoid them on kind.
Topology spread constraints
topologySpreadConstraints spread replicas evenly across failure domains. maxSkew is the permitted difference in Pod
count between the most and least loaded domain, topologyKey names the domain label, and whenUnsatisfiable chooses
between a hard DoNotSchedule and a soft ScheduleAnyway. To practise zone spreading on kind, label worker nodes with
topology.kubernetes.io/zone, for example zone-a and zone-b, via the kind config’s node labels.
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: catalog-api
matchLabelKeys:
- pod-template-hashmatchLabelKeys: [pod-template-hash] makes each Deployment revision be spread independently during rolling updates
(see Deployment Strategies).
PriorityClasses and preemption
A PriorityClass assigns an integer priority to Pods. When a high-priority Pod cannot fit, the scheduler may preempt
(evict) lower-priority Pods on a node to make room. preemptionPolicy: Never yields a class that queues ahead of others
but never evicts. Two built-in classes, system-cluster-critical and system-node-critical, are reserved for system
components.
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: bookshelf-critical
value: 100000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "catalog-api and reviews-api serving paths"Priority also decides which Pods the kubelet evicts first under node pressure, and quotas can be scoped by priority class (see Resource Management).
Scheduling gates
A Pod created with spec.schedulingGates is held in SchedulingGated state and is ignored by the scheduler until an
external controller removes every gate. Queueing systems use it to admit Pods only when quota or capacity is ready.
spec:
schedulingGates:
- name: example.com/capacity-reservedDisruptions
Pods disappear for two kinds of reasons. Involuntary disruptions are unavoidable: hardware failure, a kernel panic, the
VM being deleted, node-pressure eviction. Voluntary disruptions are initiated by an operator or controller: draining a
node for an upgrade, a cluster autoscaler removing a node, deleting a Deployment. Replicas and anti-affinity protect you
from the first kind; PodDisruptionBudget protects you from the second.
PodDisruptionBudgets
A PDB names a set of Pods with a selector and states either minAvailable or maxUnavailable. The Eviction API refuses
an eviction that would breach the budget, so kubectl drain waits instead of taking down the last healthy replica.
PDBs do not constrain involuntary disruptions, direct deletes, or Deployment rollouts (those use the
Deployment’s own surge and unavailable settings).
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: catalog-api
namespace: bookshelf
spec:
minAvailable: 1
selector:
matchLabels:
app: catalog-api
unhealthyPodEvictionPolicy: AlwaysAllowAvoid minAvailable equal to the replica count (or maxUnavailable: 0): the drain then blocks forever. For a
single-replica StatefulSet such as PostgreSQL in StatefulSets, accept some
downtime or run more replicas. unhealthyPodEvictionPolicy: AlwaysAllow lets broken Pods be evicted even when the budget
is exhausted.
Eviction
There are two very different mechanisms. API-initiated eviction is a request to the Eviction subresource, honouring PDBs
and terminationGracePeriodSeconds (see
Health Probes and Lifecycle). Node-pressure eviction is the
kubelet proactively terminating Pods when memory, disk or PID signals cross thresholds; it does not respect PDBs and
orders victims by QoS class, priority and usage over requests.
cordon, drain and uncordon
kubectl cordon kind-worker2 # mark unschedulable; existing Pods keep running
kubectl drain kind-worker2 --ignore-daemonsets --delete-emptydir-data --timeout=120s
# ... maintain or upgrade the node ...
kubectl uncordon kind-worker2 # schedulable againnode/kind-worker2 cordoned
evicting pod bookshelf/catalog-api-6f7d-abcde
pod/catalog-api-6f7d-abcde evicted
node/kind-worker2 draineddrain cordons the node and then evicts every Pod through the Eviction API, skipping DaemonSet Pods
(see DaemonSets) when asked. Pods without a controller are refused unless
--force is given. Draining is the routine step in
Cluster Setup and Administration upgrades.
Gang and PodGroup scheduling
Batch and training workloads often need all Pods of a group to start together or not at all. Kubernetes is adding
native support through a Workload/PodGroup API and a gang-scheduling plugin with a minCount. In 1.37 the
separate GangScheduling and workload-aware preemption gates were removed and merged into the GenericWorkload feature
gate, which is beta and off by default and needs the scheduling.k8s.io/v1beta1 API group enabled, so treat it as
something to evaluate, not to depend on. Until it is enabled, projects
such as Kueue, Volcano and the Jobs features in
Jobs and CronJobs fill the gap.
Source: Kubernetes Docs — Gang Scheduling.