Cloud: Varnish and Full Page Cache on Kubernetes

Four caches can hold a Magento page on the way to a shopper. Only one of them is under Magento’s control.

The built-in page cache (caching_application 1) stores pages in the cache backend, that is in Valkey, and answers from PHP. Adobe’s documentation calls Varnish the production choice and the built-in cache the development one, and the naming sheet of this series follows it: the full page cache is Varnish, not Valkey. A page served by Varnish does not run PHP at all; a page served by the built-in cache boots Magento’s front controller, reads the page from Valkey and returns it, which keeps most of the CPU cost.

On one server three facts are so obvious that nobody writes them down. Each of them is a decision on Kubernetes:

A cache is the textbook stateless workload, and a Deployment with a normal Service would be the usual choice. The naming sheet of this series deliberately uses a StatefulSet instead, and only for one reason: per-pod addressing. Magento purges by sending a request to each address in http_cache_hosts. A Deployment gives its pods random names, and the only stable name is the Service, which balances every request to one pod. With a StatefulSet and a headless Service each replica has a DNS name that never changes while the replica exists:

The StatefulSet owns no storage. The cache is memory only (-s malloc), a restarted pod starts empty, and an empty Varnish can never serve a stale page, so nothing has to be kept. The StatefulSet is not used for ordering either: podManagementPolicy: Parallel starts and stops the replicas together. The general mechanics (ordinals, the headless Service, per-pod DNS, update strategies) are in StatefulSets.

Two Services select the same pods and serve different callers:

The values of the Varnish part of the chart:

varnish.replicas is the only number that matters outside the Varnish manifests, because the list of purge targets is derived from it (next section). varnish.size is the size of the malloc storage; the process uses more than that for workspaces, thread stacks and the hash, so the memory limit above is set to one and a half times the storage as a starting point, to be corrected from container_memory_working_set_bytes after a day of traffic. The names varnish.image.*, varnish.size and varnish.resources are not part of the shared naming sheet; page 36 owns the final spelling of the values. The image is the official varnish image copied to the ECR repository pocketcoffee/varnish (a copy keeps the cluster independent of Docker Hub rate limits and lets the pipeline scan it; pull-through caching is an alternative); the tag is the exact version, never latest. The official image reads the VCL from /etc/varnish/default.vcl, runs as the user varnish (uid 1000 in the 8.0.2 image) and, started with arguments that do not begin with a dash, runs the command given, which is what the StatefulSet does so that every listener and parameter is explicit.

The ConfigMap with the VCL, the two Services and the PodDisruptionBudget:

Source: Kubernetes: Headless Services

The StatefulSet:

Source: Kubernetes: StatefulSets, Docker Hub: the official varnish image

Choices in that manifest that are easy to get wrong:

Varnish does not read Magento’s configuration. Magento produces a VCL file from a template, replacing six placeholders with the settings of the store, and the file is what the replicas run. The command is varnish:vcl:generate:

Besides the options, the command reads two values from the store configuration: web/secure/offloader_header (the header that carries the protocol, described below) and design/theme/ua_regexp (design exceptions: different themes for different user agents, which become extra hash_data() calls). Adobe’s Admin does the same thing with the buttons under Stores > Configuration > Advanced > System > Full Page Cache > Varnish Configuration. The six replaced placeholders are /* {{ host }} */, /* {{ port }} */, /* {{ ips }} */, /* {{ design_exceptions_code }} */, /* {{ ssl_offloaded_header }} */ (underscores replaced by dashes, because web servers drop headers with underscores) and /* {{ grace_period }} */.

Which template for Varnish 8. Adobe’s system requirements list Varnish 8 for 2.4.9 (and for 2.4.8-p5), yet the command in the source offers 6 and 7. The two templates are identical except for two comment lines ("the used Varnish version is 6" and "7", and the minimal version), so the choice does not change the behaviour. This page uses --export-version=7 because its comment states the right minimum. The generated file, with the six values of Pocket Coffee, compiles with varnishd -C of Varnish 8.0.2, and so does the 6 variant; whether Adobe publishes a template for 8 after the date of this page is something to check in the release notes before an upgrade.

The access list is the part of the file that needs thought in a cluster. The generator writes each entry as a quoted string. Adobe’s page for the Admin says the field accepts "hostnames, IP addresses, or CIDR ranges", and the compiled result confirms CIDR notation: "10.10.64.0/18"; compiles in Varnish 8.0.2 and matches the whole range (a request from inside the range got 200, a request from outside got 405). Pod addresses change at every restart, so a list of single IPs is useless; a hostname in the ACL is resolved once when the VCL is compiled, so it is useless too. Use the CIDR of the private subnets that hold the pods. Do not use the VPC CIDR: the load balancer’s addresses belong to it, and a customer’s request arrives at Varnish with the load balancer’s address as the client address, so with the whole VPC in the ACL anyone could send a PURGE through the public URL. Keep the load balancer in subnets of their own, outside the range in the ACL, and see the variant below for a stronger separation.

An excerpt of what the command produced for the store (the placeholders replaced by the values above):

Source: Mage-OS: the Varnish 7 VCL template of Magento_PageCache, Adobe Commerce docs: Configure Magento to use Varnish

Two details of the acl purge block matter for the next section. A purge is a ban, not an immediate removal: Varnish records the expression and tests every cached object against it on its next lookup (or in the background), which is why a purge returns at once even for thousands of pages. And the check is on client.ip, the address of the TCP connection, not on a header: it cannot be forged by a visitor, but it is also exactly the address of whoever’s connection reaches the listener.

Magento sends a purge when something that appears on a cached page changes. The chain, in the source of the Magento_CacheInvalidate module:

Three consequences follow from the code and matter in a cluster. The purge is synchronous: PHP opens a socket per host and waits for the answer, with a timeout of 10 seconds, inside the request that saved the entity (or the consumer or cron run that did). A black-holed Varnish address therefore slows saves down instead of failing them. The purge is per host with no retry: if one replica does not answer, a warning is logged ("Unresponsive cache server(s) hit") and the others are purged; only if no host answered is a critical entry written and false returned. A replica that was unreachable at that moment keeps the old page until its TTL ends. And the purge comes from the pod that saved, so the ACL must contain the addresses of all kinds of Magento pods, which is why the pod subnet CIDR is the right entry and not a list of addresses.

A healthy Varnish pod, a healthy backend and a healthy target of the load balancer are three different facts, and they get three different checks.

The backend probe. The template polls /health_check.php, which is correct for a document root of pub/ (the layout of the image). Adobe’s page for a document root of the Magento root says the path is /pub/health_check.php, and its health-check text states the rest: the script must answer within 2 seconds, it returns HTTP 500 when a resource is down, and "six failures in ten attempts mark the backend unhealthy" (the window and threshold above: the backend is healthy while at least 5 of the last 10 probes passed). The script itself, read in the source: it boots the object manager, opens a connection to every database configured, and calls test() on every cache frontend that has a backend configured (so the Valkey of the Magento cache). It does not check the session store, OpenSearch or RabbitMQ. A page can therefore fail while the probe is green, and the probe turns red when the Magento cache in Valkey is unreachable even if the database is fine. The probe goes through the Service pocketcoffee-web, so each probe lands on one of the ready pods; a single sick pod does not turn the backend unhealthy but one probe out of five fails when one of five pods is broken, which is why the pods' own readiness probe on /health_check.php (Health Probes and Lifecycle) must take that pod out of the Service first.

In the two-replica test the probe showed 8/10 healthy shortly after the start; after the backend was stopped the counter fell by about one per 5 seconds (6/10, then 5/10) and the state was still healthy, because it changes only when fewer than 5 of the last 10 probes pass, which is the sixth failure. It is the same arithmetic as Adobe’s "six failures in ten". varnishadm backend.list prints the state:

Source: Varnish: varnishadm

Why the load balancer does not use the backend probe. If the target group’s health check used /health_check.php, a database outage would make every Varnish target unhealthy at the same moment, the load balancer would answer 503 to everyone, and the pages that Varnish still holds (and is designed to serve with its grace mode) would be unreachable. /varnish-health is answered by vcl_recv with synth(200) and never reaches the backend, so the load balancer only learns whether the cache process works. Configuring the health check of a Gateway API target group is a feature of the AWS Load Balancer Controller’s own resources, whose field names change between versions; they are not reproduced here (see Load Balancers and Gateways and Gateway API). With a target type of ip the controller registers the pod addresses, so the health check also decides which replicas receive traffic during a rollout.

Three numbers decide how long a visitor can see an old page, and all three can be read in the generated file:

The logic is the vcl_hit shown earlier. Inside the TTL, the object is delivered. After the TTL and within the 300 seconds, Varnish delivers the stale copy and refreshes it in the background (the header Grace of the response says normal (healthy server)); after the 300 seconds the request goes to the backend (restart). If the probe says the backend is unhealthy, Varnish delivers the stale copy for as long as it holds one, which can be days (unlimited (unhealthy server)). That is the reason for the three-day figure on the backend side, and Adobe’s text gives the same maximum of grace plus TTL.

What this means for a store in a cluster:

The generated vcl_recv and vcl_backend_response decide, in this order:

The admin area does not appear in that table because the cluster does not send it to Varnish at all: the HTTPRoute below has a first rule for the path prefix /admin_pocketcoffee that goes to pocketcoffee-admin, and Varnish never sees an admin request. The VCL has no admin rule of its own, and the admin pages are non-cacheable only because Magento marks them so (Magento sends no-cache headers for them), so the route is the safe layer and the headers are the second. The name of the admin path is the frontName that the shop changed at build time (MAGENTO_BACKEND_FRONTNAME); a route that matches the old name /admin would send admin traffic to Varnish.

Source: Gateway API: HTTPRoute

The rule order does not matter for correctness here because the two path prefixes do not overlap, but the longer prefix is the one to keep first; Gateway API gives precedence to the longest matching path regardless of the order. The Service pocketcoffee-admin exists only when admin.enabled is true; when it is not, the same rule points to pocketcoffee-web, which still bypasses Varnish.

Not cached, and the reason in one line each: the cart, checkout and account pages (personal); every request with a body (not idempotent); GraphQL mutations; authenticated GraphQL requests; the admin; /health_check.php; any page with one block declared cacheable="false" in the layout (the whole page becomes uncacheable, and the test shows X-Magento-Cache-Debug: UNCACHEABLE; this is the most common reason that a hit rate stays near zero after launch, and the fix is in the module’s layout XML, not in Varnish).

The page that Varnish stores is the same for every visitor of one variant, so anything personal is added by the browser afterwards.

Varnish speaks plain HTTP, and the ALB terminates TLS with the ACM certificate (see Route 53 and Certificates). Magento decides "is this request secure?" in Magento\Framework\HTTP\PhpEnvironment\Request::isSecure: the request is secure if the server variable named by the configuration value web/secure/offloader_header, with dashes changed to underscores and upper-cased, equals the string https (with or without the HTTP_ prefix). The default value is X-Forwarded-Proto, which is what the ALB adds. If the header is missing or different, Magento believes the request is plain HTTP and, with secure URLs enabled for the storefront, redirects to the HTTPS URL, which comes back through the ALB as the same plain request: an endless redirect, usually seen as "too many redirects" on the home page.

Three places must agree:

Source: Adobe Commerce docs: Set configuration values

caching_application 2 is Varnish; the use_in_frontend and use_in_adminhtml settings make the store build HTTPS URLs and are set to 0 by default in Magento_Store/etc/config.xml. The base URLs themselves are not shown: they come from the environment (MAGENTO_BASE_URL) and belong to Cloud: EKS Infrastructure and Helm Deployment. These commands were not run in this validation (no store was available); they are the documented ones. They need a Magento that can boot (a build or development environment with a database), and the resulting config.php is committed.

The header reaches PHP through nginx without any rule: nginx passes every request header to PHP-FPM as HTTP_ plus the upper-cased name with dashes changed to underscores, so X-Forwarded-Proto becomes HTTP_X_FORWARDED_PROTO, which is the second of the two server variables that isSecure reads. What would break it is an nginx rule that sets fastcgi_param HTTPS from the local connection and a Magento that reads HTTPS first; the image’s nginx must not hard-code it to off.

Where the client address goes. The ALB, Varnish and nginx each add to X-Forwarded-For, so the address that PHP sees in the TCP connection is nginx’s neighbour and Magento’s own remote-address logic is what the shop sees. Admin IP allow-lists and fraud checks need the real address, and Magento’s handling of forwarded headers was not verified for this page; the WooCommerce page in this series shows the mechanism on the other platform (Cloud: Caching, CDN and Checkout), and the Magento handling is a point to check before relying on it.

CloudFront is optional, and adding it raises one design question: does it cache the HTML? No. Magento purges Varnish and nothing else. A product page cached in CloudFront for ten minutes would show the old price for ten minutes after the purge, and there is no purge request that can reach it. Varnish is the page cache; CloudFront is there for TLS and the network at the edge, for DDoS absorption, and for the static files that never change under one URL.

Source: Terraform Registry: aws_cloudfront_distribution (OpenTofu uses the same provider), AWS: managed cache policies

The decisions in the listing:

A web ACL for CloudFront has the scope CLOUDFRONT and lives in us-east-1, which is why the regional web ACL pocketcoffee-prod of the ALB is not attached here; the WAF choices belong to Cloud: Autoscaling, Observability, Backups and Security. The certificate for CloudFront must also be in us-east-1 (the variable of the listing); the ALB’s certificate is in eu-west-1. More on distributions in CloudFront.

The same checks, in the order that finds problems fastest. The outputs in the comments are from the two-replica test with the generated VCL.

Source: Varnish: varnishstat

X-Magento-Cache-Debug is set by the template in vcl_deliver on every response (HIT, MISS or UNCACHEABLE); in a cluster with several replicas a request can be a MISS on each replica once, so the second request that returns HIT may need a third try. The test also showed the exact reason to purge every replica: after a page was cached in both replicas and a purge was sent to the first one only, the first replica answered MISS and the second still answered HIT; only after the second received its purge did it answer MISS.