Cloud: Autoscaling, Observability, Backups and Security

The number that ties scaling to the database is pods x pm.max_children. The legacy database class of PrestaShop does not use persistent connections (there is no PDO::ATTR_PERSISTENT in classes/db/), so a request opens a connection when it first needs the database and closes it when it ends; the number of open connections is the number of requests in flight, never more than pods x pm.max_children, plus the cron Job and the pocketcoffee-migrate Job. With 12 workers per pod and a ceiling of 12 pods that is 144 connections from the web tier, which has to fit under max_connections of the instance with room for the Jobs and for the monitoring user. The arithmetic and RDS Proxy are in Cloud: Managed Database (RDS and Aurora); the HPA maxReplicas below is chosen from it, not from the budget of the cluster.

The HorizontalPodAutoscaler computes CPU utilisation as a percentage of the CPU request of the containers of the pod, and reports unknown if there is no request, so the php-fpm container needs one that reflects real use. The sidecars count in the average too (the exporter and the log shipper below have small requests, which lowers the average a little). The listing scales on two metrics: the CPU of the pod and the number of busy workers per pod (phpfpm_active_processes, next section). With two metrics the controller computes a desired number of replicas for each and takes the larger.

Source: Kubernetes documentation: Horizontal Pod Autoscaling

The numbers follow from the pool of the next section: with pm.max_children = 12, a target of 7 busy workers keeps about 40 percent of the workers free for a burst while new pods start. Three is the minimum so that the loss of one availability zone leaves two pods, and a PodDisruptionBudget (maxUnavailable: 1) has room to drain nodes. Scale up fast, because a campaign e-mail brings the traffic within a minute; scale down slowly (one pod every two minutes after ten minutes of lower load), because a new pod starts with a cold OPcache and an empty var/cache, which is the slowest the shop will ever be. The cache warm-up that shortens it is in Cloud: Releases and Upgrades.

The Pods metric needs an adapter that exposes it through the custom.metrics.k8s.io API: the Kubernetes SIGs prometheus-adapter with a rule over phpfpm_active_processes, or KEDA, which replaces the HPA by a ScaledObject. The rule, the PodMonitor and the trade-off are the same as for WordPress and are in Releases, Upgrades and Autoscaling on Kubernetes. Until the adapter exists, delete the Pods entry and keep the CPU metric; an HPA that lists a metric it cannot read stops scaling with FailedGetPodsMetric.

The HPA adds pods; it does not add nodes. Pods that do not fit stay Pending until the node layer reacts: Karpenter, or EKS Auto Mode which runs it for you, creates a node for them, and removes empty nodes later. Both are described in EKS clusters. Two consequences for this shop. First, set the CPU and memory requests of both containers honestly, because the node scheduler packs pods by requests. Second, an Auto Mode node is replaced at most every 21 days and Karpenter consolidates nodes, so pods are evicted regularly: keep the PodDisruptionBudget, a preStop pause so that nginx stops receiving traffic before the container ends, and terminationGracePeriodSeconds longer than the slowest checkout request. A pod that is killed in the middle of a payment callback is the one place where graceful termination has a cost you can measure in orders.

Once everything is on the container output, collection is the same as for any pod. The Amazon CloudWatch Observability add-on of EKS installs the CloudWatch agent and Fluent Bit and sends the container output of every pod to log groups under /aws/containerinsights/pocketcoffee-prod/ (the application logs in the group application). Whether the add-on is supported on an EKS Auto Mode cluster in your configuration, and the IAM role it needs (the managed policy CloudWatchAgentServerPolicy, through EKS Pod Identity or IAM roles for service accounts), are in the AWS documentation; verify both before choosing it over a log collector you run yourself.

Source: Amazon CloudWatch User Guide: Quick Start with the Amazon CloudWatch Observability EKS add-on

A Logs Insights query over the group finds the requests that failed. The field names of the Kubernetes metadata depend on the collector version, so look at one event before saving the query:

Source: Amazon CloudWatch Logs User Guide: CloudWatch Logs Insights query syntax

The Monolog line is plain text ([2026-10-29T17:52:51.316648+00:00] app.ERROR: message [] []), not JSON, which is why the query matches on text. Alerting on it is a metric filter on the log group with an alarm; the pattern is in Monitoring and observability on AWS. If you prefer a collector that you run, a Fluent Bit sidecar with the cloudwatch_logs output does the same without the add-on.

A snapshot before every release is the rollback of the database part of a release; it takes the place of the dump that a single server would make. The restore drill is the part that proves the rest, and it belongs in the calendar (quarterly is a reasonable cadence), not in the incident.

Source: Amazon RDS User Guide: Restoring a DB instance to a specified time

The restored instance has the data and the engine settings, and not the security groups, the parameter group or the users' network paths unless you pass them; give it the same subnet group, security group and parameter group as production, and attach the same monitoring. Point a staging release at it (database.host of the chart) and check the shop: the home page, a product, a checkout in test mode, a back office login, and the numbers of ps_orders and ps_customer against what you expect. A drill that ends at "the instance is available" has not shown that the shop works.

The file system is restored from a recovery point of the vault. The metadata of the restore is read from the recovery point and changed to ask for a new file system; the file system id of the production volume stays untouched:

Source: AWS Backup Developer Guide: Restore an Amazon EFS file system

The scripts were checked with shellcheck; the jq expression was run on a sample metadata object, and the exact keys of the restore metadata for your recovery point must be read from the output of get-recovery-point-restore-metadata as above, not typed. When the job finishes, create a mount target and an access point (uid and gid 33) on the new file system, point a drill PersistentVolume at it, and check that img/p/ and upload/ have the files you expect.

The database stores the ids of products, images and attachments, and the file system stores the files under paths made from them (img/p/1/2/3/123.jpg), so a database from Monday with the images of Wednesday is not a disaster, but a database of Wednesday with the images of Monday shows broken product pictures for everything uploaded in between. The rules that follow:

The RPO and the RTO of a regional loss are different from the single-instance backup. The database is as old as the last copy in the vault of eu-central-1 (daily with this plan, so up to a day; shorten the schedule or add a cross-region read replica if that is too long), the files are as old as their copy, the secret and the image are replicated, and the code is in Git. The RTO is dominated by the steps that are not automatic: applying the OpenTofu stack in the DR region, restoring the database and the file system, and changing the DNS record of www.pocketcoffee.example (a health-checked failover record in Route 53 does the last step when the primary is down). Write the order down, rehearse it once a year in a scratch account or region, and keep the runbook outside the platform that it recovers.

The values reach the pod as environment variables built from a Kubernetes Secret that the External Secrets Operator keeps in step with Secrets Manager. A change in Secrets Manager updates the Secret within refreshInterval, but a running container never sees a changed environment variable, and there is a second reason that a restart is needed in PrestaShop: config/bootstrap.php evaluates app/config/parameters.php, writes the resulting array with var_export to var/cache/<environment>/appParameters.php, and reuses that file for as long as it is newer than parameters.php; the Symfony container compiled in var/cache also contains the parameters. Values read with getenv() in parameters.php are therefore frozen into the cache at the first request of the pod. Two rules follow: the rotated value takes effect when a pod starts with an empty var/cache, which is why the cache directory is a per-pod emptyDir (never a cache baked into the image with placeholder values, or the placeholders win because the cache file is newer than parameters.php), and every rotation ends with a rolling restart of the Deployment.

Source: External Secrets Operator: AWS Secrets Manager

The names of the environment variables, of the Secret pocketcoffee-secrets and of the secret keys are those of the chart on the Helm page, which renders this object. The database credentials are in their own secrets (prestashop-db-app and prestashop-db-migrate), and not in pocketcoffee/prod/prestashop next to the cookie keys, and the Secret key DB_PASSWORD_APP reaches the pods as DB_PASSWD: the rotation functions of Secrets Manager expect a secret with the structure of a database credential (username, password, engine, host, dbname and port, and for the alternating-users strategy a masterarn that points to the secret of an administrative user), and a secret that also holds cookie keys should not be handed to a function that rewrites it. The cookie keys and secret have no rotation function; they are changed on purpose and by hand.

Source: AWS Secrets Manager User Guide: Rotate Amazon RDS for MySQL database credentials

The alternating users strategy keeps two database users (ps_app and ps_app_clone) and rotates the password of the one that is not in use, then switches the secret to it; the old password stays valid for one more rotation period. That window is what lets the rolling restart of the pods happen after the secret changed, without a moment in which the pods have a password that the database has already dropped. With the single-user strategy the password changes in the database and in the secret at the same instant, and every pod that has not restarted fails until it does. The function itself is created from the AWS templates for RDS MySQL (it runs in the VPC and needs a network path to Secrets Manager and to the database); the page does not reproduce it. The secret of the user that creates the schema and runs the upgrades (ps_migrate) is read by the Jobs only, so a rotation of it does not touch the web pods. The chart of the Helm page takes DB_USER from the value database.appUser and only the password from the secret, which fits the single-user strategy; with alternating users the user name changes with every rotation, so add - secretKey: DB_USER_APP with property: username to the data list of the ExternalSecret above and read DB_USER from that key in the helper pc.dbEnv. The listing of rotation.tf shows only the rotation of the secret that the OpenTofu configuration of the Helm page already declares as aws_secretsmanager_secret.db_app; in that configuration add the aws_secretsmanager_secret_rotation resource and not a second secret.

Source: External Secrets Operator: FAQ

The annotation trick is the one of the Operator’s documentation for forcing a reconciliation. A controller such as Reloader (stakater/Reloader) can restart the Deployment automatically when the Secret changes; it replaces step 3 and nothing else, and it should be tested on staging, because an unexpected restart in the middle of a campaign is also a rotation.

A pod without the keys invents its own. In config/bootstrap.php, if cookie_key, cookie_iv or new_cookie_key is missing from the parameters, the constant is defined anyway "to prevent failure in the composer post-install script", from Tools::passwdGen or PhpEncryption::createNewRandomKey(), at every start of PHP. In a deployment where the environment variable is empty or misspelled, nothing fails: the shop answers, but every request has a different key, nobody can stay logged in, and the cart seems to empty itself. All pods must have the same three values, and an empty one must fail the deployment, not start the shop. The entrypoint of the container should therefore refuse to start when one of the three values is empty.

A cookie that cannot be decrypted is a logout, not an error. PhpEncryptionEngine::decrypt returns false for a wrong key, Cookie::update() then finds no checksum and calls logout(), which deletes the session record and empties the cookie. Visitors see no error page; customers are logged out, a guest loses the link to the cart (the cart row stays in the database, a logged-in customer finds it again), and employees have to sign in again. The behaviour was checked in the source, and the key behaviour with defuse/php-encryption (a different key raises WrongKeyOrModifiedCiphertextException, which PrestaShop turns into false).

There is no old-key fallback. Nothing in 9.2.0 accepts two cookie keys at once, so a rotation is a cut. During a rolling restart, a visitor can hit an old pod and a new pod in turn, and is logged out each time the pod changes; the shorter the rollout, the shorter that period. Rotate in a quiet hour, announce it for the back office (employees lose their unsaved forms), and use a rollout with a high maxUnavailable, or accept a short maintenance page (see Cloud: Releases and Upgrades).

cookie_key is the dangerous one. Passwords are hashed with bcrypt first; Customer::getByEmail and Employee::getByEmail re-hash a password to bcrypt at the first successful login of an account whose hash is still the old md5 form (the isFirstHash check in classes/Customer.php and classes/Employee.php). A bcrypt hash does not depend on the static salt. An account that has not logged in since it was created by an old version still has the md5 hash, which is verified with _COOKIE_KEY_: after a change of cookie_key it cannot log in until the password is reset. The recovery links in e-mails already sent stop working as well. For a shop that began on PrestaShop 1.6 or 1.7 there can be many such accounts. So: never change cookie_key as routine; change it only if it was disclosed, accept the resets, and expect support tickets.

The keys are not rotated by a schedule. There is no reason to rotate new_cookie_key and cookie_iv every month; the lifetime of the cookie is a setting of the shop (PS_COOKIE_LIFETIME_FO and PS_COOKIE_LIFETIME_BO, 480 hours by default in the install data) and the cookie is worthless without the key. Rotate on a suspicion of exposure (a leaked secret, a departed administrator with access to Secrets Manager) and as part of the incident response of Security.

cookie_key and cookie_iv are random strings (the installer of PrestaShop makes them with Tools::passwdGen, 64 and 32 characters of letters and digits). new_cookie_key has to be a valid Defuse key, which the installer makes with PhpEncryption::createNewRandomKey(), that is Key::createNewRandomKey()->saveToAsciiSafeString() of the defuse/php-encryption library that ships in the vendor/ directory of the shop. The script generates a fresh new_cookie_key and cookie_iv from inside a running pod (so that the library and PHP of the production image are used), writes them to a new version of the secret and starts the rollout of the previous section. It leaves cookie_key alone. The values are never printed.

Source: AWS CLI Command Reference: put-secret-value

The php -r line was run with defuse/php-encryption 2.x and produced a 136-character key; the whole script was checked with shellcheck, but never run against a cluster. The key is in the vendor/ of the image, so no extra tool is needed in the container. Two cautions: the key is passed through shell variables and never printed, so keep shell tracing (set -x) off; and put a version of the secret with the previous values at hand (aws secretsmanager get-secret-value --version-stage AWSPREVIOUS), because a rotation that is rolled back needs them.