Clustering

Nodes of a cluster never talk to each other. Each node is a normal scheduler that uses the same tables, the same scheduler name (the SCHED_NAME column of every table) and the same configuration, and the database is the only coordinator. Three things make that safe: two row locks, a heartbeat row per node, and an instance id that is unique per node.

When a trigger is due, the first node to acquire it fires it; every fire time runs on exactly one node. "Acquire" is a state change in the database, not a lock held for the whole wait: in the default configuration a node selects the due triggers, issues UPDATE QRTZ_TRIGGERS SET TRIGGER_STATE = 'ACQUIRED' WHERE …​ AND TRIGGER_STATE = 'WAITING' for the one it wants, and inserts a QRTZ_FIRED_TRIGGERS row with its own instance id. The node whose update matches a row owns the trigger; the other node’s update matches nothing (it waits for the winner’s commit, then sees ACQUIRED) and it moves to the next trigger. Only when the node actually fires the trigger does it take TRIGGER_ACCESS (triggersFired), re-check that the state is still ACQUIRED, mark the fired row EXECUTING and compute the next fire time.

Source: JobStoreSupport source at v2.5.2 (acquireNextTrigger, triggerFired) and Quartz documentation: Configure Clustering with JDBC-JobStore

Quartz documents the result as more or less random: near-random for busy schedulers, favouring the node that was just active for quiet ones (Lesson 11 and the clustering page). That is a description, not a guarantee, and the measurements here show how loose it is:

Source: Quartz Tutorial: Lesson 11, Advanced (Enterprise) Features. The first row is the PostgreSQL three-node run of this section’s verification, the second row ran with the default acquisition settings; both are single runs on one laptop.

So a cluster is a high-availability device first. It does give you "exactly once per fire time" and it does keep working when a node dies; it does not spread the work evenly, and a design that needs the work spread (a heavy report on each of three nodes) must create several jobs or several triggers rather than rely on the cluster to split one. The Quartz documentation adds that clustering works best for scaling out long-running or CPU-intensive jobs; for thousands of short jobs it recommends partitioning the jobs over several distinct schedulers (see Scaling limits).

A node that dies (a crash, kill -9, a lost host, an evicted pod) leaves three kinds of evidence in the database: its row in QRTZ_SCHEDULER_STATE stops being refreshed, its QRTZ_FIRED_TRIGGERS rows stay, and the triggers it had acquired stay ACQUIRED or BLOCKED. Every surviving node’s ClusterManager looks for exactly that at each check-in.

Quartz states the requirement bluntly: never run a cluster on separate machines unless their clocks are synchronised by a time-service daemon that runs very regularly, and the clocks must be within one second of each other. The reason is visible in the source. The check-in time written to QRTZ_SCHEDULER_STATE is the node’s own System.currentTimeMillis(), and the failure rule compares that value with the observer’s own clock. Fire times are computed by each node from its own clock too. Reasoned from the source, not run (the clocks of a laptop cannot be skewed safely), the effects are:

Source: Quartz documentation: Configure Clustering with JDBC-JobStore and JobStoreSupport source at v2.5.2 (calcFailedIfAfter, clusterCheckIn)

In practice this means one time service (NTP or chrony) on every host or Kubernetes node, a monitor on the offset, and the same TZ in every container. Cluster nodes on one host or in one Kubernetes node share a clock and cannot disagree. The clocks of the database host do not matter to Quartz in the way the nodes' do, because the timestamps are written by the nodes, not by now() in SQL.

Quartz warns never to start a non-clustered scheduler against tables that another scheduler is using, and predicts serious data corruption and erratic behaviour. The source shows the mechanism. A non-clustered job store runs recoverJobs() when the scheduler starts; it assumes it is alone, so it resets the world: every ACQUIRED and BLOCKED trigger of the scheduler name goes back to WAITING, COMPLETE triggers are removed, and every row of QRTZ_FIRED_TRIGGERS of that scheduler name is deleted, whichever node it belonged to. It recovers only the fired rows that carry its own instance id, which is NON_CLUSTERED. This was run: a clustered node was executing the 25-second recoverable job, a second JVM with the same scheduler name and isClustered=false was started, and then the first node was killed.

Source: Quartz documentation: Configure Clustering with JDBC-JobStore and JobStoreSupport source at v2.5.2 (schedulerStarted, recoverJobs)

The same reset also moves the blocked triggers of a @DisallowConcurrentExecution job back to WAITING (the guard against a second concurrent run lives in those trigger states), so the cluster-wide "never overlaps" rule of Jobs & Job Data stops holding for as long as the damage lasts. The practical rules are: every process that starts a scheduler with a given scheduler name on shared tables must set isClustered=true; a one-off script, an integration test or a developer’s laptop must use another scheduler name, another table prefix or another database; and the Boot default of spring.quartz.job-store-type=memory (a RAM store never touches the tables) is the safe setting for processes that must not schedule. Giving a different scheduler name to a non-clustered scheduler is safe because every table carries SCHED_NAME; two clustered schedulers with different names simply form two independent logical clusters.

Three properties shape how triggers are acquired. They are all documented on the configuration pages, and all verified in QuartzSchedulerThread and JobStoreSupport:

Source: Quartz documentation: Configure the scheduler, Quartz documentation: Configure JDBCJobStoreTX and QuartzSchedulerThread source at v2.5.2

Source: Quartz documentation: Configure the scheduler

Two nodes, six one-shot triggers due at the same instant, six worker threads per node, three runs per setting, every run with the other Bookshelf jobs still firing:

Source: QuartzSchedulerThread source at v2.5.2. Single runs on one laptop; the scheduled time of the probe triggers was five seconds ahead and the early fires happened when a frequently firing Bookshelf trigger was acquired and the probes fell inside its four-second window.

That is the trade in numbers: a batch makes one node grab everything that is due, and a non-zero window makes triggers fire early, by more than three seconds here. Neither belongs in a service whose jobs are few and whose fire times matter (a nightly reminder at 02:00 must not run at 01:59:57); both exist for schedulers with thousands of triggers due at the same second, where acquisition itself is the bottleneck.

Every acquisition-to-completion cycle of every fire passes through TRIGGER_ACCESS (at the latest at triggersFired and triggeredJobComplete), so the lock is a single serialisation point for the whole cluster. The Quartz documentation says that a cluster-wide lock of this kind degrades performance as nodes are added, beyond about three nodes and depending on the database. Nothing here was benchmarked; the figure is Quartz’s, not a measurement of this section. What follows from it:

MongoDB and Redis have no official Quartz job store. Two community projects exist (quartz-mongodb, mirrored as io.fluidsonic.mirror:quartz-mongodb:2.2.0-rc2, and net.joelinn:quartz-redis-jobstore:1.2.0), both advertise clustering, and both are broken on Quartz 2.5.x, the version that Spring Boot 4.1 and 3.5 manage. This was tested, including two-node clustered runs, and the result is the same in the single-node and the clustered case:

Source: this section’s verification of both stores (the full matrix and the failure output are on Quartz with MongoDB and Quartz with Redis), quartz-mongodb (archived) and quartz-redis-jobstore

Even when they work they cluster differently from the JDBC store: they do not use QRTZ_LOCKS or QRTZ_SCHEDULER_STATE; the MongoDB store keeps its own lock and scheduler documents and the Redis store has no isClustered switch (the property setter does not exist, a cluster is formed by giving every node instanceId=AUTO), its heartbeat is written only when triggers are acquired, and it has no equivalent of requestsRecovery: a job running on a dead node is never re-fired, its trigger is only released by a later sweep. The honest advice is the one of the MongoDB page: keep Quartz on a small SQL database (clustered as on this page) and keep the application’s data in MongoDB or Redis, or use @Scheduled with ShedLock, whose lock providers cover both when only single execution is needed. The other NoSQL stores on this site (Couchbase, Neo4j, Elasticsearch) have no Quartz store at all.

Quartz once clustered through the TerracottaJobStore. It was removed in Quartz 2.4.0: the 2.3.2 jar contains eight Terracotta classes, the 2.4.1 and 2.5.2 jars none (the comparison is on Job Stores & Persistence), so JDBC is the only clustering mechanism in 2.5.2. Some pages of the official 2.5.x documentation still describe it (the Lesson 11 text that this page cites says clustering "currently works with the JDBC-Jobstore …​ and the TerracottaJobStore", and the configuration index still lists a Terracotta page); that is stale text, not a feature. The StdSchedulerFactory of 2.5.2 still contains a branch for job stores whose class name starts with org.terracotta.quartz, which can never be taken without that class. If you meet org.terracotta.quartz.TerracottaJobStore in an old configuration, the migration is to a JDBC job store: see Migration & What’s Changed.

A cluster on Kubernetes is a Deployment (or StatefulSet) of several replicas of the service, all pointing at one database. Nothing about Quartz needs to know about Kubernetes except two things: the instance id must be unique per pod, and a pod must be given time to finish its jobs. The workload objects themselves are on ReplicaSets & Deployments and StatefulSets, the termination sequence on Health Probes & Lifecycle. Whether the work should be a Quartz cluster at all, or a Kubernetes CronJob, is the question of Jobs & CronJobs: the CronJob is the better answer for a job that is a separate batch process; a Quartz cluster is the better answer for dynamic schedules, persistence in the application’s own database, and job code that is part of the service.

Quartz ships a clustering example in the repository under examples/src/main/java/org/quartz/examples/example13 (read at tag v2.5.2, not run as it stands because it needs a prepared database and two terminals). It is the closest thing to an official reference and worth reading next to this page:

It teaches the operational experiment: start both nodes, watch which node logs each job, kill one while a job sleeps, and watch the other log RECOVERING after the detection time. It differs from this page’s setup in three ways: plain Quartz instead of Spring (so the pool, the transactions and the job factory are configured by hand), fixed ids set per properties file instead of AUTO, and a job store (JobStoreTX) that manages its own transactions instead of Boot’s LocalDataSourceJobStore. The cluster behaviour is the same, and so is the lesson: mark the jobs that must survive a crash with requestRecovery. Source: ClusterExample.java at v2.5.2, the example13 resources at v2.5.2 and Quartz documentation: Configure Clustering with JDBC-JobStore

The weekly statistics report of Bookshelf is the job that most needs clustering: it takes a long time, must not run twice at once, and must be re-run if its node dies. Three Bookshelf nodes share one PostgreSQL database with the configuration of the Kubernetes section. The job is written so that a second run is harmless:

Source: Quartz API: JobExecutionContext and Spring Framework: Task Execution and Scheduling

The table is weekly_report (week_start date PRIMARY KEY, status varchar(10), attempts int, node varchar(200), finished_at timestamptz). The job is declared with requestRecovery(true):

Source: Quartz Tutorial: Lesson 3, More About Jobs and Job Details

The failure test ran the job (with an immediate one-shot trigger and 20 seconds of simulated work) on a node in its own JVM and killed that JVM with kill -9 in the middle, with a second node waiting:

Source: JobStoreSupport source at v2.5.2 (clusterRecover)

The two other checks of the three-node cluster were made with three separate JVMs against one container: the instance ids of bookshelf-0, bookshelf-1 and bookshelf-sysprop all appeared in QRTZ_SCHEDULER_STATE with their check-in interval, and both pod-name nodes held fired-trigger rows; and, from the section’s first cluster test, three schedulers of one JVM fired the one-second cache refresh 14 times and the two-second overdue-loan reminder 7 times with no scheduled time running twice. The monitoring of such a cluster (what to alert on: check-in age, recovered-job counts, QRTZ_FIRED_TRIGGERS rows that never disappear) is on Observability, and testing a cluster in CI on Testing.

Source: this page’s measured results and the Quartz Best Practices.

Source: Quartz releases, Spring Boot 4.0 Migration Guide and this page’s measured results. The jar comparisons were made by listing the 2.3.2, 2.4.1 and 2.5.2 jars.