Stateful Applications in Practice

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production; see the section bibliography for the reference material consulted while preparing these pages.

The earlier pages introduced the building blocks for state one at a time: Services, persistent volumes and storage classes, StatefulSets, ConfigMaps and probes. This page puts them together the way the O’Reilly Kubernetes: Up and Running chapters on integrating storage and deploying real-world applications did, but rebuilt for Kubernetes 1.37 (kubectl 1.37, kind 0.33): apps/v1 and discovery.k8s.io/v1 instead of the beta APIs, kubectl create instead of kubectl apply for ConfigMaps from files, EndpointSlices instead of hand-made Endpoints, native sidecar containers instead of a second long-running container, and primary and replica instead of the older vocabulary. It then explains why, for anything that matters, the last step of the journey is an operator or a managed service rather than a hand-written StatefulSet. All examples use the Bookshelf application and its bookshelf namespace.

State and data gravity

A stateless Pod can be replaced anywhere; a stateful one cannot, because its data has gravity. Moving compute to the data is cheap, moving the data to the compute is slow and risky. Kubernetes therefore offers a menu of options, in increasing order of what the cluster owns:

  • Import the service — the data lives outside the cluster (a managed database) and the cluster only needs a stable name for it.

  • Singleton — one Pod owns one volume, and the platform restarts it when it fails.

  • Replicated StatefulSet — several Pods with stable identity, each with its own volume, and the application handles replication itself.

  • Operator — a controller that encodes the operational knowledge (failover, backup, upgrades) a person would otherwise perform by hand.

Importing an external database

Workloads in the cluster should not hard-code the address of a database that lives outside it. Give the database a Service name inside the namespace and every Pod, in every environment, dials the same name while the target differs per namespace. This also gives production and test parity: postgres.bookshelf.svc.cluster.local can be a throwaway container in a test namespace and a cloud database in production. Two mechanisms exist.

ExternalName for a DNS name

If the managed database has a DNS name, an ExternalName Service makes CoreDNS answer with a CNAME for it:

apiVersion: v1
kind: Service
metadata:
  name: catalog-db
  namespace: bookshelf
spec:
  type: ExternalName
  externalName: bookshelf.db.example.com

Source: Kubernetes Docs — Service, ExternalName

There is no proxying, no virtual IP and no port mapping, only DNS, so the application must use the real port. TLS certificates are validated against the name the client dialed, which can surprise TLS clients that connect to catalog-db but receive a certificate issued to bookshelf.db.example.com.

Selectorless Service plus EndpointSlice for an IP address

When the target is only an IP address, create a Service with no selector, so no controller fills in its endpoints, and supply the addresses yourself with an EndpointSlice. The EndpointSlice is tied to the Service by the kubernetes.io/service-name label. This replaces the manually written Endpoints objects the book used; the Endpoints API has been deprecated since Kubernetes 1.33 in favour of EndpointSlices.

apiVersion: v1
kind: Service
metadata:
  name: catalog-db
  namespace: bookshelf
spec:
  ports:
    - name: postgres
      port: 5432
      protocol: TCP
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: catalog-db-1
  namespace: bookshelf
  labels:
    kubernetes.io/service-name: catalog-db
addressType: IPv4
ports:
  - name: postgres
    protocol: TCP
    port: 5432
endpoints:
  - addresses:
      - 203.0.113.10

Source: Kubernetes Docs — Services without selectors

Kubernetes performs no health checks on imported targets, so a dead address stays in the slice until someone edits it. Put failover behind the address (a managed service’s own endpoint, a virtual IP) rather than expecting Kubernetes to route around failure. See DNS and Service Discovery for how the names resolve.

A reliable singleton

The simplest stateful workload is one replica, one PersistentVolumeClaim and a Service. The book’s example was a MySQL singleton; the equivalent for PostgreSQL uses a Deployment with the Recreate strategy, because a ReadWriteOnce volume can be attached to one node at a time and a rolling update would briefly run two Pods against it.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: reports-db-data
  namespace: bookshelf
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reports-db
  namespace: bookshelf
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: reports-db
  template:
    metadata:
      labels:
        app.kubernetes.io/name: reports-db
    spec:
      containers:
        - name: postgres
          image: postgres:18
          envFrom:
            - secretRef:
                name: reports-db-credentials
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "bookshelf"]
          livenessProbe:
            exec:
              command: ["pg_isready", "-U", "bookshelf"]
            initialDelaySeconds: 30
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: reports-db-data

Source: Kubernetes Docs — Run a Single-Instance Stateful Application

The Secret referenced above is created from a file or generated value, never typed into a manifest that goes into Git:

kubectl -n bookshelf create secret generic reports-db-credentials \
  --from-literal=POSTGRES_USER=bookshelf \
  --from-literal=POSTGRES_PASSWORD="$(openssl rand -base64 24)"

Source: Kubernetes Docs — Managing Secrets using kubectl

A singleton is reliable only in the restart sense: if the node dies the Pod is rescheduled and the volume reattaches, after a delay. It offers no replication and no zero-downtime upgrade. See Secrets for safer ways to supply credentials.

A replicated Redis StatefulSet

The second book chapter built a Redis deployment with one primary, several replicas and Redis Sentinel for failover. The pieces map directly onto the earlier pages: a headless Service for per-Pod DNS names, a ConfigMap with several files, and an init step that decides, per ordinal, whether a Pod is the primary or a replica. The example below uses redis:8.

A multi-file ConfigMap

One ConfigMap carries the three files. Create it from real files with kubectl create configmap (the book’s kubectl apply cm --from-file is an erratum, apply has no such flag):

kubectl -n bookshelf create configmap redis-config \
  --from-file=master.conf --from-file=replica.conf \
  --from-file=sentinel.conf --from-file=init.sh

Source: Kubernetes Docs — Configure a Pod to Use a ConfigMap

The files hold the following settings. master.conf is the stock redis.conf plus persistence, replica.conf adds a replicaof redis-0.redis 6379 line, and sentinel.conf names the monitored primary:

# sentinel.conf
sentinel resolve-hostnames yes
sentinel announce-hostnames yes
sentinel monitor bookshelf redis-0.redis 6379 2
sentinel down-after-milliseconds bookshelf 5000
sentinel failover-timeout bookshelf 30000
sentinel parallel-syncs bookshelf 1

Sentinel accepts a host name such as redis-0.redis only with resolve-hostnames yes, and announce-hostnames yes keeps it using that name rather than a Pod IP that changes on restart. The name must also resolve before the Pod is Ready, which is why the headless Service below sets publishNotReadyAddresses: true.

Source: Redis Docs — Sentinel: IP Addresses and DNS names

The init script runs once per Pod start, before the server, and selects the file from the Pod’s ordinal:

#!/bin/sh
# init.sh: choose the role from the ordinal in the Pod name
ordinal="${HOSTNAME##*-}"
if [ "$ordinal" = "0" ]; then
  cp /mnt/config/master.conf /etc/redis/redis.conf
else
  cp /mnt/config/replica.conf /etc/redis/redis.conf
fi
cp /mnt/config/sentinel.conf /etc/redis/sentinel.conf

The scripts and setting names are illustrative; the point is the wiring, not a hardened Redis configuration. The project uses the original term master only inside the sentinel monitor directive where Redis itself still expects it.

The headless Service and the StatefulSet

apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: bookshelf
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/name: redis
  ports:
    - name: redis
      port: 6379
    - name: sentinel
      port: 26379
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  namespace: bookshelf
spec:
  serviceName: redis
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: redis
  template:
    metadata:
      labels:
        app.kubernetes.io/name: redis
    spec:
      initContainers:
        - name: init-config
          image: redis:8
          command: ["sh", "/mnt/config/init.sh"]
          volumeMounts:
            - name: config
              mountPath: /mnt/config
            - name: conf
              mountPath: /etc/redis
        - name: sentinel
          image: redis:8
          restartPolicy: Always        # native sidecar, GA in 1.33
          command: ["redis-sentinel", "/etc/redis/sentinel.conf"]
          volumeMounts:
            - name: conf
              mountPath: /etc/redis
      containers:
        - name: redis
          image: redis:8
          command: ["redis-server", "/etc/redis/redis.conf"]
          ports:
            - containerPort: 6379
          readinessProbe:
            exec:
              command: ["redis-cli", "ping"]
          volumeMounts:
            - name: data
              mountPath: /data
            - name: conf
              mountPath: /etc/redis
      volumes:
        - name: config
          configMap:
            name: redis-config
            defaultMode: 0755
        - name: conf
          emptyDir: {}
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi

Source: Kubernetes Docs — StatefulSets

Where the book ran Sentinel as a second regular container, a current manifest declares it as an init container with restartPolicy: Always: a native sidecar that starts before the main container, keeps running beside it and is shut down after it. See Pods for the lifecycle rules. Verify the result the way the book did, with kubectl exec -c and kubectl get pods -o wide:

kubectl -n bookshelf get pods -l app.kubernetes.io/name=redis -o wide
kubectl -n bookshelf exec redis-0 -c redis -- redis-cli info replication
kubectl -n bookshelf exec redis-1 -c sentinel -- redis-cli -p 26379 sentinel get-master-addr-by-name bookshelf
kubectl -n bookshelf delete pod redis-0

Source: Kubernetes Docs — kubectl exec

Deleting redis-0 is the failover test: Sentinel promotes a replica, and when the Pod returns with the same name the init script starts it as a replica-configured member that Sentinel then reconfigures. The design has limits that explain the next section: clients must be Sentinel-aware, configuration is baked into scripts, and there is no backup, upgrade or resharding story. For the Redis features themselves, and Valkey, see Redis on Kubernetes.

Why production databases use an operator or a managed service

A StatefulSet guarantees identity and storage, nothing more. It does not know which Pod is the primary, it cannot promote a replica consistently, take a base backup, archive the write-ahead log, restore to a point in time or perform a minor-version upgrade in the right order. An operator (see Custom Resources and Operators) encodes that knowledge in a controller and exposes it as a custom resource. For PostgreSQL the CNCF project CloudNativePG is the common choice:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: catalog-pg
  namespace: bookshelf
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:18
  storage:
    size: 10Gi
    storageClassName: standard
  bootstrap:
    initdb:
      database: catalog
      owner: bookshelf
  plugins:
    - name: barman-cloud.cloudnative-pg.io
      isWALArchiver: true
      parameters:
        barmanObjectName: bookshelf-store

Source: CloudNativePG Documentation

The operator creates the Pods and PVCs directly (it does not use a StatefulSet), elects the primary, publishes catalog-pg-rw, -ro and -r Services and generates the application credentials in a Secret. A managed service (Amazon RDS, Azure Database for PostgreSQL, Cloud SQL) goes one step further and removes the cluster from the picture; import it with the ExternalName pattern above. Choose by asking who is paged at night: if the answer is "a platform team with the skills", run an operator; otherwise buy the managed service.

Backups and snapshots

Replication is not a backup: a dropped table replicates in milliseconds. A backup strategy has two halves, a base copy and a continuous log of changes that permits point-in-time recovery. With CloudNativePG the WAL archive and base backups go to object storage through the Barman Cloud plugin (the ObjectStore named in the Cluster above), and a ScheduledBackup triggers the base copies. Its schedule is a six-field cron expression that includes seconds:

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: catalog-pg-daily
  namespace: bookshelf
spec:
  schedule: "0 0 2 * * *"      # every day at 02:00:00
  method: plugin
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io
  cluster:
    name: catalog-pg

Source: CloudNativePG Documentation — Backup

Native Barman object-store backups in the core operator are deprecated in favour of plugins, so new clusters should start with the plugin or with volume snapshots. A VolumeSnapshot (CSI, snapshot.storage.k8s.io/v1) captures a PVC at the storage layer:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: reports-db-before-upgrade
  namespace: bookshelf
spec:
  volumeSnapshotClassName: csi-snapclass
  source:
    persistentVolumeClaimName: reports-db-data

Source: Kubernetes Docs — Volume Snapshots

A snapshot of a running database is crash-consistent unless the database is quiesced, and it lives in the same storage system as its source, so keep a copy elsewhere too. Rehearse restores: an untested backup is only a hope. See Persistent Volumes and Storage Classes for snapshot classes and Jobs and CronJobs for the generic scheduled alternative.

Where the books are outdated

  • kubectl apply cm --from-file, an erratum in the book, is kubectl create configmap --from-file.

  • Hand-written Endpoints objects are replaced by EndpointSlices, and the Endpoints API is deprecated.

  • apps/v1beta1 StatefulSets and volume.alpha.kubernetes.io annotations are gone; use apps/v1, storageClassName and volumeClaimTemplates.

  • Redis "master" and "slave" are primary and replica, and the second container of the book’s pattern is a native sidecar.

  • DNS names such as mongo-0.mongo are fully qualified as <pod>.<service>.<namespace>.svc.cluster.local.

See Bookshelf End to End for the complete application and What’s Changed and Migration for the wider list.

References