Releases, Upgrades and Autoscaling on Kubernetes

In the tiered setup nothing is edited on a server. A release is a new container image holding WordPress core, the plugins, the themes and the Lumen Press code (see Composer-Based WordPress), a helm upgrade that points the Deployment at it, and, between the two, a database schema upgrade. The sequence is the same for a theme change, a plugin update and a WordPress core update.

The old Pods keep serving while the Job runs, so the schema upgrade has to be compatible with the code that is still running (see Rollback limits). The first installation of a fresh cluster is the Job lumenpress-install of the chart, not this pipeline.

WordPress stores the schema version in the db_version option and compares it with the version in the code on every admin request. When they differ, the Dashboard shows "Database Update Required" and the update runs as the first administrator who clicks it. With several Pods, a rolling update and a CI pipeline, that is the wrong moment and the wrong actor: the first request after a rollout would take locks on wp_options and wp_posts while real traffic arrives. The same work is wp core update-db, which compares the versions and runs wp_upgrade() once; on a multisite network wp core update-db --network visits every site.

The chart runs it as a Helm hook. A hook is a template with the annotation helm.sh/hook; Helm creates it at the named point of the release lifecycle, waits for it to finish and aborts the release if it fails. pre-upgrade runs after the templates are rendered and before any resource of the new revision is applied.

Source: Helm Docs: Chart Hooks

Details of the hook that are easy to get wrong:

WooCommerce and other plugins that ship their own table upgrades (WooCommerce’s wc update, many membership plugins) run through their own WP-CLI commands or admin notices, not through wp core update-db. Add them to the same Job as extra lines after the core update, for example wp wc update, once the plugin is installed in the image.

An administrator who clicks "Update" in the Dashboard on one Pod changes only that Pod’s filesystem, which disappears on the next restart and is not visible on the other Pods. In production the Dashboard is therefore read-only (DISALLOW_FILE_MODS, see Updates and Auto-Updates) and every plugin, theme and core change is a commit that builds a new image:

A plugin that must be activated or configured by the update (wp plugin activate, wp option update) belongs in the same Job or in an idempotent wp eval-file script, never in a manual step. A plugin that writes to its own directory at run time (page-builder caches, generated CSS) needs an emptyDir or an object-store option; see Media: S3 Offload, EFS and CloudFront for uploads and Object Cache, Sessions and Cron in a Cluster for the other state.

helm rollback lumenpress <revision> re-applies the manifests of an earlier revision. It restores the image tag, the replica count and the configuration; it does not restore the database, the object store or the Valkey cache.

Source: Helm Docs: helm rollback

Three practices keep a release recoverable:

A pre-upgrade hook does not run on helm rollback: Helm has separate pre-rollback hooks, and the migration is not reversed. The rollback does cancel the in-flight rolling update because Helm re-applies the old Deployment, and the old ReplicaSet scales up first.

WordPress has a built-in maintenance mode: a .maintenance file in the WordPress directory makes every request answer "Briefly unavailable for scheduled maintenance". WP-CLI toggles it with wp maintenance-mode activate, deactivate and status, and core creates the file itself during a manual update. In Kubernetes the file lives in the filesystem of one Pod: it does not exist on the other replicas, it is lost on restart, and on a read-only root filesystem it cannot be written at all. A rolling update needs no maintenance mode in the first place, because old and new Pods serve together; reserve a maintenance window for a release whose schema change the old code cannot read.

The pattern that works across Pods is a flag outside the Pod filesystem. A ConfigMap mounted into the Nginx container appears as a file whose presence Nginx tests on each request, so creating or deleting the ConfigMap switches every Pod without a restart.

Source: nginx: ngx_http_rewrite_module, the if directive

/healthz answers before the check, so the load balancer keeps the Pods in service and shows the maintenance page instead of failing health checks and returning its own 503. The ConfigMap is mounted with optional: true so that Pods start without it:

Source: Kubernetes Docs: Volumes, configMap

Source: Kubernetes Docs: Configure a Pod to Use a ConfigMap

The kubelet refreshes mounted ConfigMaps periodically, so the switch takes up to about a minute to reach every Pod, and a mount with subPath never refreshes, which is why the whole directory is mounted. The Nginx if is safe here because its only content is return; combining if with other directives in a location is the unsafe use the Nginx documentation warns about.

A WordPress Pod is a fixed pool of PHP-FPM workers (pm.max_children). When all workers are busy, further requests wait in the listen queue of the socket and then time out at the proxy, long before the Pod’s CPU is saturated if the requests mostly wait for the database or an external API. CPU is therefore a reasonable first signal for cache-friendly traffic and a poor one for anything I/O bound. The signal that describes saturation directly is the number of busy workers.

Capacity numbers (requests per second per Pod, the right pm.max_children, the HPA target) come from measurements, not from defaults. Test the staging cluster, which has the same chart and a smaller instance class, never somebody else’s site and never production at a peak. A cache-hit test and a cache-miss test measure two different systems, so run both: the first with a normal URL set, the second with a query string that bypasses the page cache.

Source: Grafana k6 Docs: Ramping arrival rate

A constant arrival rate models a public site better than a fixed number of virtual users, because real visitors do not slow down when the site does. During the run, watch kubectl get hpa lumenpress-web --watch, the Pod count, the phpfpm_listen_queue series (any sustained value above zero means requests are waiting for a worker), the 95th percentile latency and the RDS DatabaseConnections and CPU. The result to look for is the replica count at which the latency threshold is still met; set maxReplicas above it and activeProcessesPerPod so that scale-up starts well before that point. Checking that the object cache is used during the test (the Redis keyspace_hits) separates a slow database from a cold cache.

Four sources describe a WordPress release in production:

Source: Terraform Registry: aws_db_instance (AWS provider)

(The registry page documents the AWS provider that OpenTofu uses unchanged.) The snippet only shows the logging and backup arguments; networking, encryption and the rest of the instance are on Managed Database: RDS and Aurora and the OpenTofu page. A long_query_time of 1 second logs every query above that; WordPress with a page cache logs few, and without one it logs the wp_options autoload query and WP_Query meta queries first. Retention in CloudWatch Logs is set per log group (/aws/rds/instance/lumenpress-prod/slowquery) and the group is created by RDS, so set its retention explicitly or it keeps everything. In the console the Performance Insights experience was replaced by CloudWatch Database Insights, which also covers execution plans in its Advanced mode; see the RDS page for the current wording.

The single-server backup of Backups and Disaster Recovery (one dump plus a tarball) becomes a set of independent stores, each with its own recovery point:

Source: AWS Backup Developer Guide: Creating backup copies across AWS Regions

Source: Terraform Registry: aws_backup_plan (AWS provider)

Vaults can also be encrypted with a customer-managed KMS key per region; add kms_key_arn once the keys exist (see Secrets Manager and KMS). A backup that was never restored is not a backup, so the DR drill below is part of the operation, not an extra:

Source: Amazon RDS User Guide: Restoring a DB instance to a specified time

Two things to decide in advance. The RPO of the database is minutes (point-in-time recovery is limited to about the last five minutes), that of the media is the replication lag of S3, and that of the code is zero because it is in Git and ECR. The RTO of a regional disaster is dominated by the steps that are not automatic: applying the OpenTofu stack in the DR region, restoring the database from the copied snapshot, and changing DNS. Keep the DR region as a pilot light (vaults, ECR and secret replicas, and the OpenTofu code exercised in staging) and rehearse the whole sequence at least twice a year. Route 53 health checks and failover records, and the TTL for the cutover, are on Route 53 and Certificates.