Cloud: Cron, Queue Consumers and Indexers

bin/magento cron:run is not a daemon and not a single job: it is one pass of a scheduler. For each cron group it takes a lock on the group, removes the old rows of the table cron_schedule, writes the rows for the jobs that will be due in the next minutes, and then runs the rows that are due and whose status is pending. The rows are the whole state: a job that is running is a row with the status running, a job that ran is a row with success or error, and a job whose window passed before it started is a row with missed. The behaviour below was read from the 2.4.9 source (Magento\Cron\Observer\ProcessCronQueueObserver and Magento\Cron\Console\Command\CronCommand; the files of the tag 2.4.9 and of the branch 2.4-develop that were read are identical) and agrees with the description on CLI, Cron and Message Queues:

Source: Magento 2 source: ProcessCronQueueObserver

Because the locks and the atomic status change exist, two schedulers would not run the same row twice, and that is sometimes quoted as a reason to run the cron in every web pod. It is not a good reason. Each pass of each scheduler still bootstraps Magento, reads and writes the same table, and competes for the group lock; with ten pods the table is written by ten processes every minute for no extra throughput; a pod that is scaled in or evicted kills whatever job it was running; and a web pod that runs a long indexer_update_all_views takes a PHP-FPM worker’s memory away from customers. The rule of the other shops of this section applies here as well: the web pods never schedule anything, and exactly one CronJob does. That is also what Adobe’s documentation says for multi-node installations, where the crontab must run on one node only (CLI, Cron and Message Queues).

The Kubernetes side of "exactly one" is concurrencyPolicy: Forbid on the CronJob: if the Job of the previous minute is still running when the next one is due, the controller skips the new one instead of starting a second pod. The Magento side is the pair of locks above, which are the second line of defence when a Job is started by hand while the CronJob is running, or when an old pod outlives a Job that the control plane has already given up on.

The diagram shows one minute of the scheduler. The decision on the left is Kubernetes', the two locks on the right are Magento’s; the web pods and the consumers are not part of it.

Source: Kubernetes Documentation: CronJob concurrency policy

The cron group index and the group consumers have use_separate_process set to 1 in their cron_groups.xml; the group default has 0. For a group with 1 the scheduler does not run the jobs itself: it starts php bin/magento cron:run --group=<id> --bootstrap=standaloneProcessStarted=1 through a shell command renderer that sends the process to the background (the virtual type shellBackgroundCron in the di.xml of Magento_Cron) and goes on to the next group. That is a good design on a server, where the parent returns after a second and the crontab entry is free to start the next pass. In a container it is a trap: when the main process of the container ends, the pod ends, and the background children are killed with it. A CronJob that runs the plain cron:run therefore succeeds in a few seconds, shows a green Job every minute, and the group index never gets a chance to run; the indexers fall behind and nothing reports an error.

The same source shows the way out. A process that is started with --bootstrap=standaloneProcessStarted=1 does not fork again: it runs the group in the foreground. So the container starts one cron:run --group=<id> --bootstrap=standaloneProcessStarted=1 per group and waits for all of them, which is what the template below does. The parameter is not listed in the Adobe documentation of cron:run; it is an internal parameter that the scheduler passes to itself, so it is one more reason to test the chart with the version you run and to read the cron_schedule table after an upgrade (the commands are in Verifying).

The groups differ per installation: every module can add jobs to the groups it chooses or declare a new one. The groups that matter on a Magento Open Source shop are default (most core jobs: e-mail queue, cleaning, sitemap, currency rates) and index (the three jobs indexer_reindex_all_invalid, indexer_update_all_views and indexer_clean_all_changelogs). Adobe Commerce and several extensions add groups. The list of groups of your image is in the XML of its modules:

Source: Adobe Commerce developer docs: Configure a custom cron job and cron group

The chart takes the list from cron.groups (default default and index) and the groups of the extensions of the shop are added to it. The group consumers is never in the list, because its only job, consumers_runner, is what the next part replaces. A group that is not listed does not run at all, which is the failure to look for when a feature of an extension "does nothing" in the cluster only: look at the rows of cron_schedule for the job code, and at the groups of that extension’s crontab.xml.

Two nodes of env.php belong to this page. The first switches off the consumer runner (the whole file is a template of the ConfigMap pocketcoffee-env-php, as in Cloud: EKS Infrastructure and Helm Deployment):

Source: Adobe Commerce docs: env.php reference

Both values have a trap that was found in the source. ConsumersRunner::run() reads cron_consumers_runner/cron_run and returns when it is falsy, so a getenv() string such as "false" (which is truthy in PHP) would leave the runner active: write the literal. And the consumer loop treats queue/consumers_wait_for_messages as "wait for messages" only when the value is the integer 1 (the comparison is strict), so true or "1" makes the consumer end as soon as the queue is empty.

The values and the template. The pod runs the same image as the web pods, with the same environment (pc.envFrom), the same env.php mounted read-only and the same writable paths: var/ and /tmp as emptyDir, pub/media only when the media mode is the shared file system (with the S3 remote storage the pods need no media volume). The root file system is read-only, and generated/ and pub/static are baked into the image. The helper templates at the top repeat those mounts once for the cron, the consumers and the reindex Job.

Source: Helm Docs: Values files

Source: Helm Docs: Named templates

Source: Kubernetes Documentation: CronJob

The decisions in the template:

On one server consumers_runner is the supervisor. Magento\MessageQueue\Model\Cron\ConsumersRunner::run() is called by the cron group consumers every minute. When cron_consumers_runner/cron_run is true it takes the consumers of the installation (or the list cron_consumers_runner/consumers), and for each one that is not already running it starts a background process of php bin/magento queue:consumers:start <name> --single-thread --max-messages=<n> (default 10000, cron_consumers_runner/max_messages), or one process per entry of multiple_processes. "Already running" is decided with a named lock: the consumer takes md5(<name>) with --single-thread, the runner asks whether that lock is taken. The design assumes one machine, and breaks in three ways in a cluster:

So cron_consumers_runner/cron_run is false (see above) and each consumer that the shop needs is a Deployment whose one container runs one queue:consumers:start in the foreground. The consumers that exist are listed by bin/magento queue:consumers:list (24 on the Mage-OS 3.6.0 test shop of the other pages); a shop runs only those whose modules it uses. Pocket Coffee runs three: async.operations.all (the individual operations of bulk and asynchronous API calls and mass actions), product_action_attribute.update (the mass attribute updates of the Admin) and media.storage.catalog.image.resize (catalogue image resizing, see Cloud: Media and Remote Storage). In the XML of the three modules the name of the consumer and the name of its queue are the same, which is also the name of the queue on the broker.

Source: Magento 2 source: ConsumersRunner

The command is the one of the single-server page; the options that matter in a container were read from StartConsumerCommand and the loop in CallbackInvoker:

Source: Adobe Commerce docs: Manage message queues

With the AMQP connection an idle consumer polls the broker (dequeue() with a one-second sleep by default), so every idle replica is one request per second to RabbitMQ. That is nothing for a few replicas and one more reason to let KEDA take idle consumers to zero.

The values describe each consumer once; the template renders one Deployment per entry and the name of the Deployment is derived from the name of the consumer by replacing dots and underscores with dashes.

Source: Helm Docs: Values files

Source: Kubernetes Documentation: Deployments

Notes on the template:

A consumer that reaches --max-messages ends with code 0 and, because a Deployment’s pods have restartPolicy: Always, the kubelet starts the container again. Kubernetes applies a back-off to restarts of a container that ends quickly (10 seconds, doubling, capped at five minutes, reset after the container has run for ten minutes), whatever the exit code, so a busy queue with a small --max-messages shows restarts with a delay and counts in the RESTARTS column. That is expected, not a failure; choose --max-messages so that a consumer lives for minutes under load (1000 messages of the resize consumer are minutes of work) and the restart count stays low. Check the behaviour of your Kubernetes version in the pod lifecycle documentation: the decay of the back-off has been under change in recent releases.

On a server the command bin/magento queue:consumers:restart tells the running consumers to restart after a deployment. It writes a new version to the table queue_poison_pill; each consumer compares the version it saw when it started with the current one after each message it receives, and if they differ it rejects that message (the broker gets it back) and exits with code 0. In a cluster it is not part of a release, because the rolling update replaces the pods. It is still useful in one case: to make every consumer drop its in-memory state (for example after a change of configuration that is read at start) without a new image, and a consumer with an empty queue notices the pill only when its next message arrives.

A consumer is stopped by SIGTERM, whenever a pod is evicted, a node is drained, KEDA scales in or a rollout replaces it. In the consumer classes that were read (Consumer, CallbackInvoker, StartConsumerCommand) no signal handler is registered, so PHP’s default applies and the process ends at once, in the middle of the handler if it is running. With RabbitMQ that is survivable by design: the message was delivered but not acknowledged, and a message that is not acknowledged when the connection closes goes back to the queue and is delivered again. What the shop has to guarantee is the other half of that contract, which is also what the single-server page already says: a handler must be idempotent, because "delivered twice" is a normal event in a cluster.

So a long terminationGracePeriodSeconds would not help here (there is nothing that finishes the current message), and the value is 30 seconds only to give the container time to close its connection. Two precautions follow: keep the work of one message short (the resize consumer handles one image per message), and spread the replicas over nodes with topologySpreadConstraints as for the web pods so that one node drain does not stop a whole consumer (Scheduling and Disruptions). Whether the 2.4.9 consumers register signal handlers anywhere that was not read is not documented by Adobe; test a kubectl delete pod of a consumer that is in the middle of a slow handler and look at whether the message is redelivered once.

Source: RabbitMQ Documentation: Consumer Acknowledgements and Publisher Confirms

With the AMQP connection all the replicas of a consumer read the same queue and the broker gives each message to one of them (competing consumers), which is what makes scaling work. Three things have to be true for a consumer to be scaled beyond one replica:

Consumers that use the database queue (connection="db", the table queue_message) are not covered by this page: the broker is not involved, and the queue depth is a COUNT on queue_message_status that a KEDA scaler for MySQL can follow, as for the Action Scheduler on Cloud: Action Scheduler Workers on Kubernetes. On a production shop the connection is amqp for everything that is queue-driven; the managed service exists for that.

KEDA is an operator that turns an external number (here the number of messages waiting in a queue) into a replica count. A ScaledObject names a Deployment and one or more triggers; KEDA creates and owns a HorizontalPodAutoscaler for it and adds what a plain HPA lacks, scaling from zero and back to zero. The general mechanics are on Autoscaling; KEDA is installed once per cluster (the Helm chart or the EKS add-on of the cluster page, EKS Clusters) and is not part of the Magento chart.

The RabbitMQ scaler can read the queue length over AMQP or over the management HTTP API. The HTTP API is the right one here, for a reason that is in the scaler’s documentation: only the HTTP API counts the unacknowledged messages, so a queue whose messages are all in the hands of busy consumers still reads as non-empty. On Amazon MQ the management API is served on the same listener ports as the web console, 443 and 15671 (the documentation says they can be used interchangeably), under the broker’s endpoint, with the broker’s users. The endpoint of Pocket Coffee’s broker is the placeholder b-example.mq.eu-west-1.on.aws; the AMQP port of the consumers is 5671 and the management port used by KEDA is 443. Both are reachable only inside the VPC, so the security group of the broker must allow 443 from the nodes that run the KEDA operator (not from the application pods): the scaler runs inside the operator, which is in the namespace keda and in the same VPC.

Source: Amazon MQ Developer Guide: Broker (RabbitMQ)

The chart on the Cloud: EKS Infrastructure and Helm Deployment page keeps the setup minimal: its ScaledObject reads the broker user of env.php from the consumer’s environment. This section is the stricter variant, with a monitoring-only user and a TriggerAuthentication; the keda.* values below extend the chart’s keda block and replace its trigger metadata.

The consumers connect as the broker user magento. KEDA gets a user of its own, keda, that can only look: Amazon MQ supports the tags administrator, management and monitoring on RabbitMQ users, and a user is created, and its tags and permissions set, through the management API or the web console with an administrator (the first user, created with the broker). The commands below use placeholders; the administrator password is read from the environment and never typed into the shell history, and the password of keda goes into the secret pocketcoffee/prod/magento under the key already used for the AMQP user, or under a key of its own (the template further down reads the key from keda.remoteKey).

Source: Amazon MQ Developer Guide: Amazon MQ for RabbitMQ broker users

The permission set is the smallest that was expected to work for reading a queue’s counters, and it was not tried against a broker: the second curl is the test, and a 401 or 403 means that the tag or the permissions need widening (try management next). The permission pattern syntax and the API are those of RabbitMQ itself.

Source: RabbitMQ Documentation: HTTP API reference

Three objects per release plus one per consumer: an ExternalSecret that builds the secret with the user and the password of keda from the secret store, one TriggerAuthentication that tells KEDA which keys of that secret are the username and the password of the scaler, and one ScaledObject per consumer Deployment. The {{ .password }} inside quotes in the first object is for External Secrets, not for Helm: the template writes it as a Helm string so that Helm leaves it alone.

Source: KEDA Docs: ScaledObject specification

Source: KEDA Docs: RabbitMQ Queue scaler

The fields, as documented for KEDA 2.21 and checked against the scaler’s source (pkg/scalers/rabbitmq_scaler.go):

Validation: helm lint and helm template | kubeconform -strict -ignore-missing-schemas pass; the ExternalSecret, the TriggerAuthentication and the three ScaledObject objects are custom resources that kubeconform skips (five skipped resources in the run of this page), so they are validated on a cluster with kubectl apply --dry-run=server once KEDA and External Secrets are installed.

An indexer in Update on Save rebuilds the affected rows inside the request that changed the data (the Admin save, the import, the API call), so the web pods, the one thing that must stay quick, pay for the indexing, and several of them can try the same tables at once. In Update by Schedule the change is only recorded: MySQL triggers on the source tables write the id of the changed row to a changelog table (_cl appended to the name of the view), and the cron job indexer_update_all_views of the group index, once a minute, reads the new ids and reindexes those entities in portions. This is the mview mechanism of Caching, Indexing and Performance, and it is what Adobe recommends for production. In a cluster it has an additional merit: the work moves to the one pod that is allowed to do background work, the CronJob.

The mode is state of the database, not of the code. The triggers and the changelog tables live in RDS, the mode in the tables mview_state and indexer_state; config.php of the image and the ConfigMap of env.php hold nothing about it. Two consequences:

Source: Adobe Commerce docs: Manage the indexers

indexer:set-mode schedule creates the triggers on the watched tables; without ids it does it for every indexer. Adobe’s advice is to run it in maintenance mode with cron stopped, to avoid database locks, and in a cluster that is simple: run it before the first traffic of a new environment (in the installation Job’s last step or by hand right after it) and, on a running shop, scale the CronJob out of the way by setting cron.enabled to false in the same release, or use cron/enabled as the switch described above. The customer grid indexer changed in 2.4.8 and supports both modes; read the output of indexer:status rather than assuming.

The status of an indexer has two parts, the state of the index (Ready, Reindex required, Processing) and the backlog of the schedule (idle (3 in backlog)), and the second one is the cheapest health check of the whole background machinery. A backlog that grows is one of three things: the CronJob does not run the group index, it runs it but every pass fails or is killed, or the indexer is too slow for the rate of changes. The commands to tell them apart are in Verifying.

The group index has three jobs: indexer_reindex_all_invalid and indexer_update_all_views every minute, indexer_clean_all_changelogs every five. Its cron_groups.xml has schedule_lifetime of 2 minutes and schedule_ahead_for of 4, so a missed row is quickly replaced, and use_separate_process of 1, which is the reason for the standaloneProcessStarted parameter of the template. Three effects to know: