Managed Kubernetes and Distributions

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production; see the section bibliography for the reference material consulted while preparing these pages.

Every example so far ran on a local kind cluster (Getting Started). Production needs a real cluster, and the first decision is who operates it. This page compares the options and surveys the distributions and multi-cluster tools. It is written against Kubernetes 1.37; see the section index for the version baseline. Building a cluster yourself is covered in Cluster Setup and Administration; this page is about the alternatives to doing so. Cloud products change quickly, so treat product names and feature sets here as a snapshot and confirm them in the vendor documentation listed under References.

Bare metal, IaaS or managed

The 2015 book (Rensin, chapter 4) and the 2017 sampler (chapter 1) both frame the same three-way choice, and it still holds:

  • Bare metal. You own servers and networking. Maximum control and often the lowest unit cost at scale, but you run everything: machine provisioning, load balancers, storage, the control plane and its etcd backups.

  • IaaS (self-managed on cloud VMs). You rent virtual machines and install Kubernetes with kubeadm or a distribution. You still run the control plane and upgrades, but cloud load balancers and disks integrate through the cloud controller manager and CSI drivers.

  • Managed Kubernetes. The provider runs the control plane (API server, etcd, scheduler, controller manager), patches and upgrades it, and sells you nodes or, increasingly, a fully provider-operated data plane.

In the books' era the managed offer was essentially a single hosted service and most readers built clusters by hand (the 2017 sampler mentions kops for self-managed clusters on a cloud). Today every major cloud and many smaller providers sell a conformant managed service, and self-managing is usually a deliberate choice for regulatory, latency, edge or cost reasons rather than the default.

Aspect Bare metal IaaS self-managed Managed service

Control plane operated by

You

You

Provider

Node operating system

You

You

You, or the provider (Automatic modes)

Upgrades

You (kubeadm upgrade)

You

Provider-assisted, on a schedule you choose

Load balancers and disks

You (MetalLB, Ceph, …​)

Cloud integrations

Cloud integrations, preconfigured

Typical fit

Edge, data-centre, sovereignty

Custom needs on a cloud

Most product teams

What "managed" covers

A managed service normally includes: a highly available control plane with an uptime SLA (often a paid tier), etcd backup, control-plane certificate rotation, version upgrades and security patches, integration with the cloud’s identity, networking and storage, and a conformance guarantee (see Conformance). It normally does not include: your workloads, your RBAC design (Authentication, Authorization and RBAC), application-level backups, network policy design, image supply-chain security, or the bill. Nodes are the grey area, and providers have moved along a spectrum:

  • Standard mode. You choose VM sizes and node pools; the provider operates only the control plane. You patch or auto-upgrade node images.

  • Automatic modes. The provider also manages nodes: AKS Automatic, Amazon EKS Auto Mode and GKE Autopilot provision, scale, patch and bin-pack nodes for you from your Pod requests, apply opinionated defaults (managed ingress or Gateway, autoscaling, policy and monitoring add-ons), and bill mostly by what Pods request rather than by VM. The trade-off is fewer knobs: restrictions on privileged Pods, host access and some node customisation. Pod resources.requests matter more than ever here (Resource Management).

Because the control plane is hidden, some of this site’s administration topics (etcd snapshots, kubeadm certificates) do not apply; your operational surface shifts to workloads, add-on versions and cluster upgrades.

AKS, EKS and GKE

Azure Kubernetes Service (AKS) is covered in depth on this site: AKS Clusters (cluster creation, node pools, networking, identity and upgrades) and AKS Microservices (Gateway API, workload identity, KEDA and the Istio add-on). AKS Automatic is the opinionated, node-managed SKU.

Amazon EKS and Google GKE are referenced here in prose only; dedicated guides are planned for this site (issues #163 for AWS and #164 for GCP). In outline, EKS runs the control plane in AWS-managed accounts and supports managed node groups, Fargate and EKS Auto Mode; GKE offers Standard and Autopilot clusters, regional control planes and release channels. Creating an Automatic-style cluster on each provider looks like this (resource group, project and region names are placeholders):

# Azure: AKS Automatic
az aks create --resource-group bookshelf-rg --name bookshelf --sku automatic

# AWS: EKS with Auto Mode
eksctl create cluster --name bookshelf --region eu-west-1 --enable-auto-mode

# Google Cloud: GKE Autopilot
gcloud container clusters create-auto bookshelf --region europe-west1
# After any of them, fetch credentials into the local kubeconfig and verify
az aks get-credentials --resource-group bookshelf-rg --name bookshelf
kubectl get nodes
kubectl version

Source: Microsoft Learn — AKS Automatic, AWS Docs — EKS Auto Mode and Google Cloud Docs — GKE Autopilot.

Once credentials exist, the Bookshelf manifests from the rest of this section apply unchanged; what differs is the glue: the StorageClass names (Persistent Volumes and Storage Classes), the LoadBalancer annotations (Services), the Gateway class (Gateway API) and how Pods obtain cloud identity.

Distributions

A distribution packages upstream Kubernetes with an installer, defaults, lifecycle tooling and often a support contract. Conformance means the API behaves the same, so your manifests are portable, but the install and operation experience differs.

Distribution Character

OpenShift

Red Hat’s enterprise platform: Kubernetes plus an integrated registry, builds, console, Operators-based lifecycle, stricter default security and its own Route and SecurityContextConstraints history. See Quarkus on Kubernetes and OpenShift for deploying to it.

RKE2

SUSE/Rancher’s security-focused distribution, often called RKE Government: hardened defaults, FIPS support, containerd, embedded etcd, single-binary install. Aimed at data-centre and regulated environments.

k3s

A lightweight, single-binary distribution (SQLite or embedded etcd by default, bundled Traefik and service load balancer) for edge, IoT, CI and small clusters.

MicroK8s

Canonical’s snap-packaged distribution with microk8s enable add-ons; convenient on Ubuntu workstations, small clusters and edge.

Talos Linux

An immutable, minimal operating system that exists only to run Kubernetes: no SSH or shell, configured entirely through an API (talosctl). Declarative and GitOps-friendly.

 
# k3s: one-line server install, then use the generated kubeconfig
curl -sfL https://get.k3s.io | sh -
sudo k3s kubectl get nodes

# MicroK8s
sudo snap install microk8s --classic
microk8s enable dns storage
microk8s kubectl get nodes

# Talos: a throwaway local cluster in containers
talosctl cluster create
# /etc/rancher/rke2/config.yaml -- minimal RKE2 server configuration
write-kubeconfig-mode: "0640"
token: "<token>"
tls-san:
  - bookshelf-api.example.com
cni: cilium

Source: K3s Docs, RKE2 Docs, MicroK8s Docs and Talos Linux Docs.

Pick by operating model: choose OpenShift when you want a supported platform with batteries included, RKE2 when hardening and compliance lead, k3s or MicroK8s for small or edge footprints, and Talos when you want an immutable, API-driven node OS. Each tracks upstream releases with a short lag; check the distribution’s own release notes for the Kubernetes minor it ships before planning an upgrade. For local development, kind and minikube (Getting Started) remain the simplest choices, and the packaging-and-vendor sites maintain the authoritative install instructions.

Multi-cluster management

Running several clusters (per region, per environment, per team or per tenant for isolation) is common, and the question becomes how to keep them consistent.

  • KubeFed and "Ubernetes" (history). The 2015 book (chapter 4) anticipates cluster federation under the working name "Ubernetes": one control plane spreading workloads over clusters. That became Kubernetes Cluster Federation and then the KubeFed project, which was archived. Do not start new designs on it.

  • Cluster API. A Kubernetes sub-project that manages the lifecycle of clusters themselves through Kubernetes objects: a management cluster holds Cluster, machine and control-plane resources, and provider packages create the infrastructure on AWS, Azure, vSphere, bare metal and others. Upgrades become edits to a declared version.

  • GitOps fleets. Tools such as Argo CD ApplicationSets and Flux apply the same Git-stored configuration to many clusters, parameterised per cluster (GitOps and CI/CD), often combined with Cluster API for provisioning.

  • Service mesh and multi-cluster networking. Cross-cluster traffic and discovery are provided by meshes and CNIs (Cluster Networking and CNI), not by Kubernetes itself.

# Turn a local kind cluster into a Cluster API management cluster
clusterctl init --infrastructure docker
clusterctl generate cluster bookshelf-dev --flavor development \
  --kubernetes-version v1.37.0 --control-plane-machine-count 1 \
  --worker-machine-count 2 > bookshelf-dev.yaml
kubectl apply -f bookshelf-dev.yaml
clusterctl describe cluster bookshelf-dev
NAME                                   REPLICAS  AVAILABLE  READY
Cluster/bookshelf-dev                  3/3       3          3

Source: Cluster API Book — Quick Start.

Conformance

The CNCF Certified Kubernetes programme runs the conformance tests (Sonobuoy) against each vendor’s release. A certified distribution or service passes the upstream API-behaviour suite for that Kubernetes version, which is what makes the portability promise credible and lets you move the Bookshelf manifests between providers. Certification is per version and per product, and the CNCF publishes the list; the Kubernetes documentation’s turnkey-solutions page lists certified hosted and on-premises providers from the same landscape data.

Cost awareness

Cost is now part of the design, not an afterthought:

  • Managed control planes usually have a free tier and a paid SLA tier, plus per-hour or per-vCPU fees for automatic modes.

  • The largest line is almost always compute. Right-size requests, enable autoscaling (Autoscaling) and use cheaper interruptible nodes for tolerant workloads.

  • Hidden costs: cross-zone and egress traffic, load balancers (one per LoadBalancer Service; prefer a shared Gateway), persistent disks left behind by deleted claims, logging and metrics volume.

  • Shared clusters with namespaces and quotas improve bin-packing, the efficiency argument made in the 2017 sampler’s introduction, but weigh it against isolation needs.

  • Use cost tooling (provider cost analysis or the OpenCost project) and tag or label resources by team.

Where the books are outdated

  • Managed offers. The 2015 book describes Google’s hosted container service and Google Compute Engine as the obvious homes and treats other clouds as roadmap or community territory; the 2017 sampler lists the same kind of short set of hosted options. Both "only one cloud" and "no AWS or Azure" statements are obsolete: AKS, EKS, GKE and many other certified services exist, and the Azure container service (ACS) the 2017 era mentioned was retired in favour of AKS.

  • Cluster federation. "Ubernetes" and KubeFed are history, as above; Cluster API and GitOps replaced them.

  • Local setup. Book 1’s Vagrant-based local clusters and the distribution-specific getting-started scripts are superseded by kind, minikube and the distributions above.

  • Alternatives. The books name other container services (ECS, Service Fabric) as alternatives; Kubernetes is now the common denominator and those are niche choices for new systems.

A decision sketch

flowchart TD A["Need a cluster"] --> B{"Who runs the control plane?"} B -->|"Provider"| C{"Manage nodes yourself?"} B -->|"Me"| D{"Where?"} C -->|"Yes"| E["Managed Standard\nAKS / EKS / GKE"] C -->|"No"| F["Automatic modes\nAKS Automatic / EKS Auto Mode / GKE Autopilot"] D -->|"Cloud VMs"| G["kubeadm or a distribution\n(RKE2, OpenShift)"] D -->|"Edge or small"| H["k3s / MicroK8s / Talos"] D -->|"Own hardware"| I["kubeadm, RKE2, Talos\n+ Cluster API for fleets"]

References