Cloud: Autoscaling, Observability, Backups and Security

A Magento pod is a fixed pool of PHP-FPM workers, as for any PHP shop: when every worker is busy, requests wait in the listen queue of the FastCGI socket and then time out at nginx, often before the CPU of the pod is saturated, because the workers wait for the database, for OpenSearch or for a payment call. Two things are specific to Magento and change how the HPA is tuned.

First, Varnish sits in front. With the full page cache at the edge of the cluster (see Cloud: Varnish and Full Page Cache on Kubernetes), the traffic that reaches pocketcoffee-web is the misses, the passes (cart, checkout, customer account) and the API calls. A campaign e-mail raises the visitors tenfold and the load on PHP only as much as the cache is cold for the landing pages; a cache purge after a release does the opposite, and sends the whole catalogue to PHP at once. The HPA therefore has to react quickly to a sudden rise and the minimum has to cover a cold cache. Second, the number of pods is bounded by the database: each running request holds a database connection, so maxReplicas x pm.max_children plus the cron Job, the consumers and the migration Job must fit under max_connections. The arithmetic and RDS Proxy are in Cloud: Managed Database (RDS and Aurora); choose maxReplicas from it, not from the size of the cluster.

PHP-FPM publishes the state of its pool on a status page when pm.status_path is set. The page is served by FPM itself over FastCGI, not by nginx, so it is reachable only by a client that speaks FastCGI to 127.0.0.1:9000 inside the pod, which is what the exporter does; nginx has no location that passes /status on, so it is not exposed to the internet. The Magento nginx configuration (the nginx.conf.sample shipped with Magento) passes only the entry points index.php, get.php, static.php, health_check.php and a few others to FPM, which keeps the status page out of reach; keep that list closed when you adapt it.

Source: PHP Manual: FastCGI Process Manager (FPM) configuration

The file is read after the defaults of the official image because of its name (zz-) and was checked with php-fpm -t in php:8.4-fpm (the pool keys are the same on PHP 8.5; they are PHP’s, not Magento’s). clear_env = no matters for the secrets: with the default clear_env = yes the PHP code cannot read the environment variables of the container, and env.php of this series reads everything with getenv(). Size pm.max_children from the memory of a worker under your own traffic (a Magento worker with the compiled code and a warm OPcache is far larger than a blog worker; measure a catalog page, a checkout request and an Admin save separately) and the memory limit of the container, and set pm.max_requests so that a leaky extension is bounded.

The exporter is a sidecar of the pocketcoffee-web pod. The community project hipages/php-fpm_exporter (not an official PHP or Magento project) turns the status page into Prometheus metrics on port 9253. There are two documented ways to collect FPM metrics, and the choice is about who owns the Prometheus stack:

The metrics that matter are phpfpm_active_processes (the value the HPA scales on), phpfpm_idle_processes, phpfpm_listen_queue (requests waiting for a free worker; anything above zero for more than a few seconds is saturation), phpfpm_max_children_reached (alert on its increase, not its value) and phpfpm_slow_requests (only when request_slowlog_timeout is set). Check the names on the /metrics endpoint of the version you deploy before building a dashboard or an HPA on them. The excerpt of the Deployment shows the pod with the sidecar and the probes:

Source: Kubernetes documentation: Sidecar containers

The readiness probe is /health_check.php of Magento and the liveness probe is a static /healthz served by nginx, on purpose: pub/health_check.php checks the backends (database, cache) and a database outage must take the pods out of the Service, not restart them in a loop; the behaviour of the script is described on Cloud: EKS Infrastructure and Helm Deployment, and the generic rules in Health probes and lifecycle. The PodMonitor is a custom resource of the Prometheus Operator, so kubeconform skips it with -ignore-missing-schemas; the rest was validated against the Kubernetes 1.36 schemas. The real chart also mounts env.php, the nginx and PHP ConfigMaps, the media volume when media.mode is efs, and the rest of the security context. The alerts that earn a page are an increase of phpfpm_max_children_reached over five minutes, a phpfpm_listen_queue above zero for two minutes, a 5xx rate at the load balancer, and a Varnish hit ratio that drops after a release without recovering.

Metrics of the pool say that the shop is slow; a trace says why. Three options exist, and Adobe’s statements about each are different.

Source: Adobe Commerce developer documentation: Observability installation

Source: Adobe Experience League: Investigate performance with New Relic

Source: OpenTelemetry documentation: PHP zero-code instrumentation

The vendor-neutral choice for a shop that runs the open-source edition on its own cluster is the combination that needs no Adobe entitlement: the exporter above for the pool, the CloudWatch Observability add-on or your own collector for the logs, and either an APM vendor’s PHP agent or the OpenTelemetry extension for traces. Whichever you choose, install it in the image at build time (a PHP extension is not something to add to a running pod), keep it out of the setup:* commands of the migration Job, and watch the memory: an agent adds to the resident memory of every worker, which changes pm.max_children.

Magento writes its logs to files, not to stdout, and the files are in the var/log directory of the pod, which is an emptyDir in this layout, so they are lost with the pod. In app/etc/di.xml of the 2.4 line the main logger Magento\Framework\Logger\Monolog is configured with three handlers named system, debug and syslog. Magento\Framework\Logger\Handler\System writes var/log/system.log at level INFO, hands records that carry an exception to the exception handler (which writes var/log/exception.log), and Magento\Framework\Logger\Handler\Debug writes var/log/debug.log. The base class Magento\Framework\Logger\Handler\Base builds the path as the base directory of the installation followed by a relative file name, and it strips . and .. from the name, so a stream such as php://stderr cannot be set through a handler’s fileName argument. There is no environment variable that changes this. There are two honest options:

Source: magento/magento2: Framework/Logger/Handler/Base.php

Source: Adobe Commerce developer documentation: The di.xml file

The two items replace the system and debug handlers by key (arrays of arguments are merged by item name), so the exception records, which the stock System handler diverts to exception.log, now arrive in the same stream; syslog is left alone. Two caveats. The handler uses the Monolog 3 API (Monolog\Level), which is what composer.json of the 2.4 line requires (monolog/monolog ^3.6); confirm with composer show monolog/monolog in your build. And the debug handler of Magento is conditional on the deployment mode, which this replacement ignores: in production mode a handler at level INFO is the right ceiling, and the shop should never run developer mode in the cluster. Magento also writes var/report/<id> files for uncaught exceptions; they stay on the pod and the error page shows the report id, so keep the log line that contains it.

The database is where a Magento shop spends much of its time, and RDS keeps a log of the statements that took too long. It is off by default: turn it on in the parameter group and export it to CloudWatch Logs, so that it can be searched from the same place as the application logs. The listing is the part of the OpenTofu stack that concerns logging; the instance itself is in Cloud: Managed Database (RDS and Aurora).

Source: Terraform Registry: aws_db_parameter_group (AWS provider)

The registry page documents the AWS provider that OpenTofu uses unchanged. The parameter group above holds only the logging parameters; the group of the database page (log_bin_trust_function_creators and the other Magento settings) must contain them as well, so merge the two rather than creating two groups with the same name. The log group is created by RDS the first time a log is exported; declaring it first, with the exact name, gives it a retention. On Aurora MySQL the same parameters live in the cluster parameter group of pocketcoffee-prod-aurora. A long_query_time of 1 second is a starting point: lower it for a week during an investigation, then back, because the log costs I/O and storage.

Source: Amazon RDS User Guide: MySQL database log files

Statement-level analysis is the job of CloudWatch Database Insights. It replaced the Performance Insights console experience (end of life 2026-07-31), shows the load of the instance by wait event, SQL statement and user (so the runtime user mg_app, the mg_migrate Job and the Admin are separate sources only because they use separate users, one reason for having them), and in its Advanced mode also execution plans and on-demand analysis. The modes, the retention and their cost drivers are in Amazon RDS; choose the mode per instance and check the current wording on the AWS page before relying on the old Performance Insights flags in automation.

What to expect in the log depends on the shop; which statements appear is a question for your own slow log, not for this page. Two Magento tools exist for finding where a query comes from in the code, and both are for a developer environment, not for the cluster:

A single-server Magento is backed up by a database dump and an archive of the code and pub/media. On EKS the same information is in separate stores with separate recovery points, and only some of them are backed up at all. The question to answer is what is restored, what is rebuilt and whether what is restored is consistent.

Source: AWS Backup Developer Guide: Creating backup copies across AWS Regions

Three of these rows deserve a sentence each, because the AWS documentation is explicit and the consequence for Magento is not obvious. OpenSearch automated snapshots are for cluster recovery only; the service takes them hourly for domains on OpenSearch and retains up to 336 of them for 14 days, in a bucket it manages. If the cluster goes red, automated snapshots stop, and AWS warns that data can be lost if the state is not corrected within two weeks. Because the catalog and search index of Magento is derived from the database, the usual recovery of a lost domain is a new domain and a reindex, which can take hours for a large catalog; that duration, not the data, is what a snapshot would save. ElastiCache for Valkey can take snapshots, and the AWS documentation says they are available for Valkey, but a cache and a session store are not data to restore: a snapshot of the session group from yesterday signs in sessions that have since ended. For RabbitMQ, Amazon MQ documents the broker, its Multi-AZ cluster deployment and maintenance; no backup of queue messages is documented, so plan on the database being the source of truth and verify the broker’s durability settings (quorum queues in a cluster) on the AWS page for your engine version.

The OpenTofu below is a daily AWS Backup plan that protects the RDS instance, the media bucket and, when it is the media mode, the EFS file system with one plan, and copies the recovery points to a vault in eu-central-1. The instance keeps its own automated backups and point-in-time recovery as well (backup_retention_period on the aws_db_instance, 14 days is a common choice): point-in-time recovery has a granularity of minutes, the AWS Backup plan gives the long retention and the second region. AWS Backup requires S3 Versioning to be enabled on a bucket it protects and recommends a lifecycle expiration for the versions, which the second listing sets.

Source: Terraform Registry: aws_backup_plan (AWS provider)

Source: Terraform Registry: aws_s3_bucket_versioning (AWS provider)

Versioning protects the media against what Magento does to it as well as against a mistake: the S3 driver of remote storage overwrites and deletes objects when images are replaced or products are deleted (see Cloud: Media and Remote Storage), and an old version is what brings back a picture that an editor replaced by accident. The expiry of non-current versions after 35 days matches the AWS Backup retention above; without it every replaced image is kept for ever and, per the AWS Backup documentation, every version and delete marker is included in the recovery point.

A web ACL in front of the load balancer (or of CloudFront) removes a class of traffic before PHP sees it: known bad inputs, SQL injection patterns, bots with a bad reputation and the brute force against the login forms. It does not replace patching Magento and the extensions (the patch cadence and the incidents are on Security); it buys time between an advisory and the rollout of the fix. The pieces that are specific to Magento are these paths:

Source: Adobe Experience League: Secure your Commerce site and infrastructure

The paths of the storefront controllers are those of the core modules (route customer, controller account, actions loginPost, createpost, forgotpasswordpost); confirm them against the routes of your own installation and your theme, because a PWA or a checkout extension can add its own. The first four rows are what the page can say from the platform; the callback path depends on how PocketCoffee_Pay registers its route (see Checkout and Payments).

Source: Terraform Registry: aws_wafv2_web_acl (AWS provider)

The ACL is the regional web ACL pocketcoffee-prod of this series. It is associated with the load balancer that the Gateway API implementation creates (the annotation or the Gateway configuration depends on the controller; see Load balancers and gateways on AWS) or, with a CloudFront distribution in front of the shop, by a web ACL of scope CLOUDFRONT created in us-east-1 (see CloudFront). Behind CloudFront the client address that the rate rules see is the one of the CloudFront edge unless the rule aggregates on the forwarded address (aggregate_key_type = "FORWARDED_IP" with a forwarded_ip_config), and Magento itself must trust the proxy that sets X-Forwarded-For and X-Forwarded-Proto (the cache and proxy settings are on Cloud: Varnish and Full Page Cache on Kubernetes).

Operate it in this order. Deploy with enforce_managed_rules = false, so that the five groups only count; read the sampled requests for a week of normal traffic, including an Admin session, a checkout, a catalog import and several payment callbacks; add an override or a scope-down for each rule that counted a legitimate request; then set it to true. Two rules of the Common group are already overridden to count in the listing, because the Admin posts HTML (CMS pages, product descriptions) and the REST and GraphQL APIs post large JSON bodies, and that is where a body-inspection rule fires on a legitimate save; the body inspection stays on at the URL, query string and cookie level, and the Admin is behind the address list. The rule names and labels of each group are in the AWS managed rule groups documentation; read the current list rather than trusting this page for names that AWS adds and renames. The rate limits are a starting point to tune against real traffic: a shop behind a corporate proxy or a mobile carrier can put hundreds of customers on one address.

AWS also offers managed rule groups for bots and for credential abuse: Bot Control (AWSManagedRulesBotControlRuleSet), account takeover prevention (AWSManagedRulesATPRuleSet) and account creation fraud prevention (AWSManagedRulesACFPRuleSet). They need a configuration that names the login or sign-up path and the fields of the form, and they work with the WAF client-side token (JavaScript challenge), which a Magento storefront must load; the Magento login form posts login[username] and login[password], so the field identifiers differ from the generic examples. They are worth evaluating for a shop that is under credential stuffing or scraping; the configuration was not tested for this page, and the pricing is on the AWS pages, not here.

The Admin is the highest-value target of the shop, and most of the hardening that does not depend on code is about it. The platform-independent hardening (a non-default URI, 2FA settings, session lifetime, lockout, roles, Content Security Policy) is on Security; this section is about how it is realised in the cluster.

Not every secret rotates in the same way, and in Magento the difference is large. The table lists what the shop reads from the Kubernetes Secret pocketcoffee-magento and what happens when it changes.

crypt/key in env.php is what Magento encrypts stored secrets with: the payment gateway credentials and other obscured configuration values in core_config_data, saved card data, and the provider credentials of 2FA. It is not one key but a list, and understanding that list is what makes a rotation safe. From lib/internal/Magento/Framework/Encryption/Encryptor.php: the constructor reads the configured string, splits it on whitespace into keys (preg_split('/\s+/s', ...)), and treats the last key as the active one for new encryption; each encrypted value carries the version of the key it was encrypted with, and decrypt() picks that key, returning an empty string if that index no longer exists. The method exportKeys() joins the list with a newline, and a key must be a 32-character string without whitespace.

What Magento offers, as read in the 2.4-develop source and in Adobe’s documentation:

Source: Adobe Commerce developer documentation: Data re-encryption

Source: Adobe Experience League: Encryption key

Adobe’s guidance after a security incident says that rotating the key makes the new key the primary one for reading and writing sensitive data, that data encrypted earlier is read with the key recorded for it, and that rotation does not necessarily re-encrypt what was already stored; the same pages state that a key change invalidates all customer and admin sessions (integration users excluded), so everyone must sign in again. The documented on-premises procedure is: enable maintenance mode, disable cron, run the key change, flush the cache, enable cron, disable maintenance mode.