Cluster Setup and Administration

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production; see the section bibliography for the reference material consulted while preparing these pages.

Most teams consume Kubernetes as a managed service (see Managed Kubernetes and Distributions), but someone still has to understand how a cluster is assembled, kept patched and recovered when something breaks. This page covers the self-managed route with kubeadm, the official bootstrapping tool, and the routine administration tasks that follow. It is written against Kubernetes 1.37 (1.37.1); see the section index for the full version baseline. For a disposable local cluster use kind or minikube from Getting Started with Kubernetes; the components described here are introduced in Introduction and Architecture, and the controls that protect an administrator’s access are in Authentication, Authorization and RBAC.

Production options

kubeadm is deliberately narrow: it bootstraps the control plane and joins nodes, and nothing else. It does not provision machines, load balancers, a CNI plugin or storage. Other routes exist:

Approach	What it gives you
kubeadm	Official, minimal building block. You supply machines, a load balancer, a container runtime and a CNI plugin.
Cluster API, kOps, Kubespray	Higher-level tools that provision infrastructure and typically run kubeadm or equivalent logic underneath.
Managed service (AKS, EKS, GKE)	The provider runs and upgrades the control plane; you manage node pools and workloads.

Approach

What it gives you

kubeadm

Official, minimal building block. You supply machines, a load balancer, a container runtime and a CNI plugin.

Cluster API, kOps, Kubespray

Higher-level tools that provision infrastructure and typically run kubeadm or equivalent logic underneath.

Managed service (AKS, EKS, GKE)

The provider runs and upgrades the control plane; you manage node pools and workloads.

Whatever the tool, a production cluster needs an odd number of control-plane nodes (three is the usual minimum), a stable endpoint for the API server, swap disabled on nodes, unique hostnames and MAC addresses, and open ports between nodes.

Source: Kubernetes Docs — Production environment.

Container runtimes

The kubelet talks to the runtime through the Container Runtime Interface (CRI). The two mainstream CRI runtimes are containerd (the default in most distributions and managed services) and CRI-O. Docker Engine itself is not a CRI runtime. Early Kubernetes (the era of the books) called Docker directly through a built-in shim called dockershim; it was deprecated in 1.20 and removed in 1.24. Images built with docker build still run unchanged, because they follow the OCI image format, but a node no longer has a Docker daemon to inspect: use crictl (see Debugging and Troubleshooting) instead of docker ps. If you must use Docker Engine as the runtime, the external cri-dockerd adapter provides the CRI endpoint.

For kubeadm clusters the most common configuration choice is the cgroup driver: on systemd-based distributions both the kubelet and the runtime should use systemd, and kubeadm defaults the kubelet to that. See Orchestration: Swarm and Kubernetes for the Docker-side context.

Source: Kubernetes Docs — Container Runtimes.

Installing the packages from pkgs.k8s.io

Kubernetes packages are published on community-owned repositories at pkgs.k8s.io, with one repository per minor version. The legacy Google-hosted repositories (apt.kubernetes.io, yum.kubernetes.io) were deprecated in September 2023 and are gone, so any guide that uses them is out of date. The repository URL contains the minor version, which is also how an upgrade to the next minor version is selected.

# Debian / Ubuntu: add the v1.37 repository and install the three packages
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.37/deb/Release.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.37/deb/ /' \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl   # upgrade deliberately, never by accident
sudo systemctl enable --now kubelet

On RPM-based distributions the repository is https://pkgs.k8s.io/core:/stable:/v1.37/rpm/ and the packages are held back with exclude=kubelet kubeadm kubectl in the .repo file (install with --disableexcludes=kubernetes).

Source: Kubernetes Docs — Installing kubeadm.

Bootstrapping with kubeadm

Single control plane: init and join

# On the control-plane node
sudo kubeadm init --pod-network-cidr=10.244.0.0/16

# Make kubectl work for your user
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# Install a CNI plugin (Cilium, Calico, Flannel...) -- nodes stay NotReady without one
kubectl apply -f <cni-manifest-url>

# On each worker: run the join command printed by init
sudo kubeadm join <api-endpoint>:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

kubeadm init runs preflight checks, generates the cluster CA and certificates under /etc/kubernetes/pki, writes kubeconfig files, starts the control plane as static Pods (manifests in /etc/kubernetes/manifests that the kubelet runs directly), and installs CoreDNS and kube-proxy as add-ons. The bootstrap token in the join command expires after 24 hours; create a new join command on the control plane with kubeadm token create --print-join-command. The CA hash lets the joining node verify it is talking to the real control plane. Treat both as credentials: never commit them. The CNI choice is covered in Cluster Networking and CNI.

Source: Kubernetes Docs — Creating a cluster with kubeadm.

The ClusterConfiguration file

Flags quickly become unmanageable, so use a configuration file, which is also what makes a cluster reproducible and reviewable in Git. The current kubeadm configuration API is kubeadm.k8s.io/v1beta4.

# kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.37.1
clusterName: bookshelf-prod
controlPlaneEndpoint: "k8s-api.example.com:6443"
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
apiServer:
  certSANs:
    - k8s-api.example.com
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

sudo kubeadm init --config kubeadm-config.yaml --upload-certs
kubeadm config print init-defaults        # see every field and its default

Source: Kubernetes Docs — kubeadm Configuration (v1beta4).

High availability: stacked or external etcd

A single control-plane node is a single point of failure. kubeadm supports two HA topologies, both of which need a TCP load balancer in front of the API servers (the controlPlaneEndpoint above points at it).

Topology	Advantages	Disadvantages
Stacked etcd: an etcd member runs on every control-plane node	Fewer machines; simpler to set up and manage; replicas are easier to reason about.	Coupled failure: losing a node loses both an API server and an etcd member, so failure handling needs care.
External etcd: a separate etcd cluster, three or more hosts	Control plane and datastore fail independently; suits large clusters.	Roughly double the hosts, and more to operate (certificates, backups, upgrades).

Topology

Advantages

Disadvantages

Stacked etcd: an etcd member runs on every control-plane node

Fewer machines; simpler to set up and manage; replicas are easier to reason about.

Coupled failure: losing a node loses both an API server and an etcd member, so failure handling needs care.

External etcd: a separate etcd cluster, three or more hosts

Control plane and datastore fail independently; suits large clusters.

Roughly double the hosts, and more to operate (certificates, backups, upgrades).

# First control-plane node (stacked etcd)
sudo kubeadm init --config kubeadm-config.yaml --upload-certs

# Every additional control-plane node (values printed by the command above)
sudo kubeadm join k8s-api.example.com:6443 --control-plane \
  --token <token> --discovery-token-ca-cert-hash sha256:<hash> \
  --certificate-key <certificate-key>

For the external topology, replace the local etcd in ClusterConfiguration with a pointer to the existing cluster:

etcd:
  external:
    endpoints:
      - https://<etcd-0-ip>:2379
      - https://<etcd-1-ip>:2379
      - https://<etcd-2-ip>:2379
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
    keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key

--upload-certs stores the control-plane certificates encrypted in a Secret for two hours; the <certificate-key> decrypts them and must be handled like a password. Use an odd number of members so a majority (quorum) survives: three members tolerate one failure, five tolerate two.

Source: Kubernetes Docs — Creating Highly Available Clusters with kubeadm.

The release cycle and support windows

Kubernetes ships about three minor releases per year (the 1.37 release is the August 2026 one; 1.38 is due in December 2026). Each minor release is supported for roughly 14 months: about 12 months of regular patch releases followed by about 2 months of maintenance mode (critical fixes and CVEs only). Patch releases appear about monthly. Because of this, a cluster that is never upgraded falls out of support within a little over a year.

Minor release	Maintenance mode from	End of life
1.37	28 Aug 2027	28 Oct 2027
1.36	28 Apr 2027	28 Jun 2027
1.35	28 Dec 2026	28 Feb 2027
1.34	27 Aug 2026	27 Oct 2026

Minor release

Maintenance mode from

End of life

1.37

28 Aug 2027

28 Oct 2027

1.36

28 Apr 2027

28 Jun 2027

1.35

28 Dec 2026

28 Feb 2027

1.34

27 Aug 2026

27 Oct 2026

The table is a snapshot taken when this page was written; the authoritative schedule is the patch-releases page in the references.

The version-skew policy

Components may run at different versions during an upgrade, but only within defined limits:

Component	Allowed skew relative to kube-apiserver
kube-apiserver (HA members)	The newest and oldest instances differ by at most one minor version.
kubelet	Never newer; up to three minor versions older (1.37 API server supports kubelets 1.37 to 1.34).
kube-proxy	Never newer; up to three minor versions older.
kube-controller-manager, kube-scheduler, cloud-controller-manager	Never newer; up to one minor version older.
kubectl	One minor version older or newer (1.37 API server: kubectl 1.36 to 1.38).

Component

Allowed skew relative to kube-apiserver

kube-apiserver (HA members)

The newest and oldest instances differ by at most one minor version.

kubelet

Never newer; up to three minor versions older (1.37 API server supports kubelets 1.37 to 1.34).

kube-proxy

Never newer; up to three minor versions older.

kube-controller-manager, kube-scheduler, cloud-controller-manager

Never newer; up to one minor version older.

kubectl

One minor version older or newer (1.37 API server: kubectl 1.36 to 1.38).

The consequence is the upgrade order: control plane first (API server before the other control-plane components), then kubelets and kube-proxy on the nodes. Minor versions must be upgraded one at a time: 1.35 to 1.36 to 1.37, and never directly from 1.35 to 1.37. Book-era Kubernetes (0.x and 1.0) had no formal skew policy and was commonly upgraded by rebuilding the cluster.

Source: Kubernetes Docs — Version Skew Policy.

Upgrading a kubeadm cluster

An upgrade has five phases: check the release notes and deprecations, back up etcd, upgrade the first control-plane node, upgrade the remaining control-plane nodes, then upgrade the workers one at a time.

Before you upgrade

Read the release notes and the deprecation guide for the target version.
Find resources that use removed API versions. pluto checks manifests kept in Git (pluto detect-files -d manifests/) and Helm releases (pluto detect-helm); kubent scans the live cluster (kubent). The kubectl convert plugin rewrites a manifest to a newer apiVersion.
Take an etcd snapshot (see below) and make sure workloads have PodDisruptionBudgets and more than one replica, so draining does not cause an outage (see Scheduling and Disruptions).

The sequence

# 0. Switch the package repository to the next minor version (edit the URL, e.g. v1.36 -> v1.37), then:
sudo apt-get update
sudo apt-cache madison kubeadm | grep 1.37          # choose the latest 1.37.x patch

# 1. On the FIRST control-plane node: upgrade kubeadm, inspect, apply
sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm='1.37.1-*' && sudo apt-mark hold kubeadm
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.37.1

# 2. Drain (from a machine with kubectl), upgrade kubelet and kubectl, restart, uncordon
kubectl drain cp-1 --ignore-daemonsets
sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet='1.37.1-*' kubectl='1.37.1-*' && sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
kubectl uncordon cp-1

# 3. OTHER control-plane nodes and every worker, ONE AT A TIME: upgrade kubeadm, then
sudo kubeadm upgrade node
# ...followed by the same drain / kubelet upgrade / restart / uncordon steps

# 4. Verify
kubectl get nodes

kubeadm upgrade plan lists the versions you can reach and whether the configuration needs manual changes; kubeadm upgrade apply upgrades the control-plane static Pods, CoreDNS and kube-proxy on the first node, while kubeadm upgrade node does the equivalent for every other node. Draining is mandatory for a kubelet minor upgrade because the container specification hash changes and all containers on the node restart. The same flow is what a managed service automates for you, which is one reason nodes are better treated as replaceable (a pattern covered in Deployment Strategies for applications and in Autoscaling for nodes).

flowchart TD A["Read release notes;\nscan for removed APIs (pluto / kubent)"] --> B["etcd snapshot save"] B --> C["Switch pkgs.k8s.io repo\nto next minor version"] C --> D["CP node 1: upgrade kubeadm\nkubeadm upgrade plan / apply"] D --> E["Drain CP node 1"] E --> F["Upgrade kubelet + kubectl,\nrestart kubelet"] F --> G["Uncordon CP node 1"] G --> H{"More control-plane\nnodes?"} H -->|yes| I["kubeadm upgrade node\n(then drain, upgrade, uncordon)"] I --> H H -->|no| J{"More workers?"} J -->|yes| K["Worker: kubeadm upgrade node,\ndrain, upgrade kubelet, uncordon"] K --> J J -->|no| L["kubectl get nodes:\nall on the new version"]

Source: Kubernetes Docs — Upgrading kubeadm clusters and Changing the Kubernetes package repository.

Backing up and restoring etcd

etcd holds every object in the cluster, so a good etcd snapshot is a good cluster backup (persistent volume contents are separate and need their own backups; see Persistent Volumes and StorageClasses). Use etcdctl for snapshots and etcdutl for operations on data files. Kubernetes documents etcd 3.4.29+ and 3.5.11+ as the minimum for production.

# Save a snapshot from a kubeadm control-plane node
sudo ETCDCTL_API=3 etcdctl snapshot save /var/backups/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify it
etcdutl snapshot status /var/backups/etcd-snapshot.db --write-out=table

# Restore into a NEW data directory, then point the etcd static Pod at it
etcdutl snapshot restore /var/backups/etcd-snapshot.db --data-dir /var/lib/etcd-restored

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 3c8a91f2 |   482913 |       1207 |     6.1 MB |
+----------+----------+------------+------------+

After restoring, edit the hostPath of the etcd-data volume in /etc/kubernetes/manifests/etcd.yaml to the new directory; the kubelet restarts etcd automatically. In a multi-member cluster every member must be restored from the same snapshot. Keep snapshots off the node, encrypted, and test a restore regularly. Never expose etcd to anything but the API servers: access to it is equivalent to root on the cluster. Encrypting Secrets at rest is covered in Secrets.

Source: Kubernetes Docs — Operating etcd clusters for Kubernetes.

Certificate rotation

kubeadm issues leaf certificates valid for one year and CA certificates valid for ten years (both configurable with certificateValidityPeriod and caCertificateValidityPeriod). A cluster that is never upgraded will therefore stop working after a year when the API server certificates expire. Upgrading with kubeadm upgrade renews them automatically, which is another reason to upgrade regularly.

sudo kubeadm certs check-expiration        # table of every certificate and its residual time
sudo kubeadm certs renew all               # renew manually (then restart the control-plane static Pods)
sudo kubeadm certs renew apiserver         # or renew one certificate

The kubelet’s client certificate is rotated automatically through the certificates API (rotateCertificates: true in the kubelet configuration); signing requests can be approved with kubectl certificate approve <csr-name>. The CA itself is not renewed by these commands and needs a planned rotation, so monitor expiry with the tooling described in Observability.

Source: Kubernetes Docs — Certificate Management with kubeadm.

Windows nodes

A cluster can include Windows worker nodes (Windows Server 2022 and later) to run Windows containers, while the control plane always runs on Linux. Nodes join with kubeadm join like Linux nodes, and workloads are steered to them with the kubernetes.io/os: windows node selector and, typically, a taint such as os=windows:NoSchedule. The limitations are significant: Linux-only features (privileged containers, many security-context fields, some volume types) do not apply, images are large, and the node OS version must match the container base image. Plan Windows nodes as a separate node pool and read the support matrix before committing.

Source: Kubernetes Docs — Windows containers in Kubernetes.

Where the book is outdated

The 2015 book’s chapter on setup describes a local cluster built from Vagrant virtual machines and distribution "getting started" scripts, with Google Container Engine as the only managed service, plus the "Ubernetes" cluster federation idea. Today a developer machine uses kind or minikube, a self-managed cluster uses kubeadm (or a tool built on it), and every major cloud offers a managed service. Federation was replaced by multi-cluster tooling; see Managed Kubernetes and Distributions. Nodes were called "minions" and control was by kube-up.sh; neither exists any more.