Caching, Indexing and Performance

Magento’s application cache is divided into cache types, each declared by a module in etc/cache.xml, each switched on or off on its own and each holding a different kind of data. The test store has these 16 (15 core types plus one from a Mage-OS package; the output of bin/magento cache:status lists them; the descriptions are those of the cache.xml files):

Source: Adobe Commerce docs: Manage the cache

Adobe Commerce adds types for its own modules (for example target_rule, admin_ui_sdk and webhooks_response, which appear in Adobe’s own sample output); Open Source does not have them. The enabled/disabled state of each type is stored in app/etc/env.php under cache_types ('config' => 1, ...), and that is why a deployment pipeline has to carry that file or run cache:enable.

Source: Adobe Commerce docs: Manage the cache

The Admin equivalents are System > Tools > Cache Management, with the Flush Magento Cache button (the same as cache:clean) and Flush Cache Storage (the same as cache:flush). A type that is marked Invalidated in the grid has been marked stale by code (Magento\Framework\App\Cache\TypeListInterface::invalidate(), which the indexer below calls) and wants a refresh at the next opportunity.

A module can add its own cache type, so that its entries can be cleaned or switched off separately. It takes three things: the declaration in etc/cache.xml, a class extending Magento\Framework\Cache\Frontend\Decorator\TagScope that adds a tag to everything it stores, and a reader that uses the class.

Source: Adobe Commerce developer docs: Create a cache type

Source: Adobe Commerce developer docs: Create a cache type

Source: Adobe Commerce developer docs: Create a cache type

What each part does:

What happened when it was installed and used:

Source: Adobe Commerce docs: Manage the cache

In Valkey the entry appeared as three keys in database 0: the data under <prefix>:POCKETCOFFEE_ROAST_SHELF_ROWS (a string), the set cache:id_tags:<prefix>POCKETCOFFEE_ROAST_SHELF_ROWS (the tags of the entry) and the set cache:tags:<prefix>POCKETCOFFEE_ROAST (the identifiers that carry the tag), which is how clean-by-tag works without scanning. The prefix is the id_prefix of the frontend in env.php.

Redis was Magento’s recommended cache and session store. The 2.4.9 release notes introduced comprehensive support for Valkey 9.x, "driven by Redis 7.2 end-of-support and licensing changes", and the system requirements list Valkey 9 and no Redis for 2.4.9 (2.4.8-p5 and 2.4.7-p10 list Valkey 8.1; Redis is not listed for 2.4.7-p10 either). Valkey is a Redis-protocol server forked from Redis, so the wire protocol, the commands and the data layout are the same, and the PHP client does not change.

Two details of the 2.4.9 code base matter when you read older guides or configurations:

The installer wrote these nodes to app/etc/env.php (shortened; the id_salt of the GraphQL cache and other secrets are not shown, and the server name is the container’s):

Source: Adobe Commerce docs: env.php reference

On the test store: valkey-cli info keyspace showed database 0 (the cache) with several hundred keys, no keys in database 1 (the built-in page cache had not been filled at that moment) and about two dozen sessions in database 2. The whole Valkey instance used 4.6 MB of memory, and maxmemory was 0 (unlimited) with the policy noeviction, which are the defaults of the image. A production server should set maxmemory and the policy deliberately, as the next list says.

Source: Adobe Commerce docs: Install and set up Valkey

Adobe’s page on software recommendations still says "Redis" in places; read it as Valkey for 2.4.9. Commands to look inside the server:

Source: Adobe Commerce docs: Install and set up Valkey

The full page cache (FPC) stores the finished HTML of cacheable pages (home, category, product, CMS, …​) and serves it without building the layout. Magento has two implementations, selected by system/full_page_cache/caching_application (Stores > Configuration > Advanced > System > Full Page Cache > Caching Application): 1 is the built-in cache, which runs inside PHP and stores in the page_cache frontend of env.php; 2 is Varnish, an HTTP accelerator that sits in front of the web server. Adobe says both things plainly: use Varnish in production, because the built-in cache is "much slower than Varnish", and the built-in module is for development. The test store uses the built-in cache.

Source: Adobe Commerce docs: Configure Varnish

How a cacheable page is identified and refreshed, in both implementations:

The built-in cache can be seen at work with the debug header it adds in developer mode:

Source: Adobe Commerce docs: Varnish and page caching

The query string is part of the key, which is why the measurements below add a throw-away pcprobe parameter to force a miss without touching the cache configuration.

Fifteen requests to distinct URLs of the home page (every one a miss), then twenty-five requests to one of them (every one a hit), with curl from the host to the Apache container, developer mode, empty catalog, shared laptop:

The hit is only half of the miss because PHP still boots and the built-in cache is a PHP script reading Valkey; with a populated catalog the miss grows into hundreds of milliseconds while the hit stays near the same value, and with Varnish the hit leaves PHP out entirely. These numbers show the shape, not the speed of a real server: the container is capped and developer mode adds work to every request.

Varnish 8 is the supported version on-premises in 2.4.9 (the system requirements list it with nginx 1.30). The VCL file is generated from the Admin (Stores > Configuration > Advanced > System > Full Page Cache > Varnish Configuration, button Export VCL for Varnish 6 or 7 files) or from the command line:

The options of varnish:vcl:generate are --access-list (IPs allowed to PURGE; repeatable; default localhost), --backend-host (default localhost), --backend-port (default 8080), --export-version (Varnish major version, default 6), --grace-period (seconds, default 300), --input-file (a custom template), --output-file (without it the VCL is written to the console) and --enable-bfcache (default 1: allow the browser Back-Forward cache). The command takes the template module-page-cache/etc/varnish7.vcl (or varnish6.vcl) and replaces the placeholders /* {{ host }} */, /* {{ port }} */, /* {{ ips }} */, /* {{ ssl_offloaded_header }} */, /* {{ design_exceptions_code }} */ and /* {{ grace_period }} */.

On the Mage-OS test store the command ran and wrote a file, but the placeholders were left unreplaced in the output. The reason is that Mage-OS ships the third-party module Elgentos_VarnishExtended, which provides its own varnish:vcl:generate and VCL generator; the output above is therefore not what Adobe’s command produces, and a stock Magento Open Source installation was not available to compare. Varnish itself is not installed on the test store, so no VCL was loaded or served. What the template (the module’s varnish7.vcl read from vendor) contains is stable and worth knowing:

Source: Adobe Commerce docs: Configure Magento to use Varnish

The deployment rules that matter: Varnish does not terminate TLS, so put nginx (or another TLS terminator) in front of it and pass the offloaded-protocol header named in the configuration (web/secure/offloader_header, X-Forwarded-Proto by default); the web server then listens on the backend port (8080) and Varnish on 80; if you delete var/cache by hand, restart Varnish; and add the Varnish hosts to env.php as http_cache_hosts (bin/magento setup:config:set --http-cache-hosts=varnish.example.test:80) so Magento knows where to send the PURGE requests. Adobe describes optional extras: storing static assets in Varnish instead of a CDN (a VCL edit that removes cookies for static files), and several Varnish servers.

With several web nodes, the built-in full page cache is replaced by Varnish in front of the pods (and purge fan-out to every Varnish), and the cache and session backends are managed Valkey; see Cloud: Varnish and Full Page Cache on Kubernetes and Cloud: OpenSearch, Valkey and RabbitMQ as Managed Services.

A Magento catalog is stored in EAV tables that are good for flexibility and bad for reading. Indexers copy the data into flat or denormalised tables (and into OpenSearch) that the storefront reads quickly. The test store has 11 indexers, all in the mode Update by Schedule:

Adobe Commerce adds indexers for its own features (Sales Rule, Target Rule, the SaaS data exporter feeds) and can move price indexing to a SaaS service when Live Search, Catalog Service or Product Recommendations are used; Open Source does not have them.

An indexer runs in one of two modes:

The mechanism behind Update by Schedule is mview (materialised view, emulated with triggers). When an indexer is switched to schedule, Magento creates a changelog table named after the view with the suffix _cl (for the custom indexer below pocketcoffee_roast_shelf_cl, with columns version_id auto-increment and entity_id) and AFTER INSERT, AFTER UPDATE and AFTER DELETE triggers on every table the view subscribes to. The triggers insert the changed entity ids into the changelog; the cron job indexer_update_all_views (group index) reads the rows after the last applied version_id and passes the ids to the indexer’s execute() method. The state is in the table mview_state (mode enabled, status idle or working, version_id).

Source: Adobe Commerce docs: Manage the indexers

The behaviour of each, from Adobe’s page and the test store:

The jobs of the index cron group, from the crontab.xml of Magento_Indexer: indexer_reindex_all_invalid (reindex every invalid indexer), indexer_update_all_views (apply the changelogs) and indexer_clean_all_changelogs (delete processed changelog rows). The group is run by bin/magento cron:run --group=index, or as part of a plain cron:run.

A custom indexer takes four artifacts: the table that holds the result (declarative schema, see Declarative Schema and Patches), the indexer declaration, the mview declaration that says which tables to watch, and one class that implements both Magento\Framework\Indexer\ActionInterface and Magento\Framework\Mview\ActionInterface.

Source: Adobe Commerce developer docs: Custom indexer

bin/magento setup:db-declaration:generate-whitelist --module-name=PocketCoffee_RoastIndex wrote etc/db_schema_whitelist.json with the table’s columns, index and primary key; declarative schema needs that file for later removals.

Source: Adobe Commerce developer docs: Custom indexer

Source: Adobe Commerce developer docs: Custom indexer

Source: Adobe Commerce developer docs: Indexing

The attributes: in indexer.xml, id is the indexer id (what you pass to indexer:reindex), view_id links it to the mview with the same id, class is the action executed, title and description are shown by the CLI and the Admin; in mview.xml, view id and class name the same action, group is indexer (the changelog is processed by cron group index) and each table node names a table to watch and the entity_column that holds the id of the entity to reindex. Additional table subscriptions (a price table, a category link table) make the index react to more changes; each costs a trigger, so subscribe only to the tables your index really depends on.

The five methods and when they are called: executeFull() for indexer:reindex and the Admin; execute($ids) from mview with the ids read from the changelog; executeList($ids) and executeRow($id) when the indexer is in Update on Save mode and code asks the indexer to reindex a batch or one entity (for example $indexer->reindexRow($id)). The class must be idempotent, because mview can pass the same id more than once, and must not assume a particular order of ids.

Result on the test store:

Source: Adobe Commerce docs: Manage the indexers

Notes from the run: a new indexer starts as Reindex required and in schedule mode; the first cron:run --group=index only wrote the cron_schedule rows and the second executed indexer_update_all_views; the changelog row stays after it was applied and is removed later by indexer_clean_all_changelogs. The probe product was deleted afterwards and the index was rebuilt. The execute() of the class invalidates the custom cache type with TypeListInterface::invalidate(), so the Admin shows Invalidated for it after every index change.

The PHP settings above go in php.ini (or in the PHP-FPM pool, with php_admin_value); the PHP settings page also asks for the same opcache.save_comments=1 and the memory limits, and says to restart the web server afterwards. The test store’s PHP already had the 10M/7200 realpath cache, opcache.save_comments=1 and a 256 MB OPcache with 60000 files; the 512 MB figure of the table is Adobe’s sizing for stores with many extensions. A production pool for nginx and PHP-FPM looks like the following sketch. The numbers are starting points to be checked against memory_limit and the measured memory of a worker, not Adobe values (Adobe gives no PHP-FPM pool numbers); it was not run, since the test store uses Apache.

Source: Adobe Commerce docs: Required PHP settings

Source: Adobe Commerce docs: Configure nginx

What the nginx.conf.sample of the 2.4.9 code base does, read from the file in the container: it forwards /index.php, /get.php, /static.php, health_check.php and the error pages to fastcgi_backend with fastcgi_buffers 16 16k and fastcgi_buffer_size 32k, a PHP memory limit of 756M and max_execution_time 18000 (five hours) set through PHP_VALUE, fastcgi_read_timeout 600s; serves /static/ and /media/ directly with long expiries (the /static/ block carries Cache-Control "public, max-age=31536000, immutable" for files of the versioned URL; the comment says to uncomment the header in production); sets no-store for downloads such as .zip, .csv and .xml; compresses text with gzip level 6; blocks .php, .phtml, .htaccess and .git outside the entry points; and carries a commented load_module line for the optional image filter module. The setup and update tools sit behind their own location blocks with higher limits. For Apache the equivalent is the .htaccess in the root, pub/.htaccess (mod_deflate) and pub/static/.htaccess (immutable expiry for versioned static files, no-store for archives), which need mod_rewrite, mod_expires, mod_headers and mod_deflate.

The Magento 2 books of 2016 and 2017 still turn up in searches. Their performance chapters describe the 2.0 and 2.1 platform and need a correction before use: they suggest running the PHP engine HHVM (not supported by Magento any more), use the MySQL fulltext search (the supported engines in 2.4 are OpenSearch and, in older lines, Elasticsearch), tune Apache prefork or mod_php as the main path (Adobe now documents nginx with PHP-FPM as the production pairing), configure Varnish 3 or 4 VCL by hand (use the VCL exported for the Varnish version you run), use Redis rather than Valkey, and do not know about immutable static URLs, HTTP/2, WebP or mview-based indexers on separate schedules. The advice that has not aged is the principle: measure first, cache at the outermost layer that is safe, put sessions and cache on a fast in-memory store, give the database memory, and keep indexes on schedule.

Product images are the heaviest part of a catalog page. Magento resizes them to the sizes declared in the theme’s etc/view.xml and stores the results under pub/media/catalog/product/cache; the resizing can run asynchronously through the media.storage.catalog.image.resize consumer. Serve pub/media and pub/static from a CDN or from the web server with long expiries (the samples already do), use WebP or AVIF where the platform and theme support it, and set explicit width and height on images in templates to avoid layout shifts. The general techniques are the subject of the responsive images and thumbnails page and of the loading performance page on this site; the Hyvä and Edge Delivery storefronts discussed in Frontend: Hyvä, PWA Studio and Edge Delivery change the front-end half of this checklist, but not the caches, indexers or database below them.