Production & Best Practices
|
This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production. The reference material consulted while preparing these pages is listed in the section bibliography. |
The earlier pages of this section explain how each Quartz feature works. This page is the one to read before the first deployment and again after the first incident: it takes the official Quartz best-practices page, checks every item against the 2.5.2 source and a test run, adds what a Spring Boot service needs on top (shutdown, rolling deployments, pool sizing, long jobs, database upkeep, what to expose), and ends with a table of seven anti-patterns. It deliberately links to the page that owns each topic instead of repeating it: the mechanics of clustering are on Clustering, the tables on Job Stores & Persistence, the failure semantics of a job on Jobs & Job Data.
It was written against Quartz 2.5.2, Spring Boot 4.1.1, Spring Framework 7.0.9 and Java 21 (bytecode level; the build ran on JDK 25.0.2 with Maven 3.9.16) and verified on 5 October 2026. Every behaviour reported as measured was run in a test project: single-node cases on the in-memory store, everything about shutdown, clusters and the tables on PostgreSQL 17 (postgres:17, Testcontainers 2.0.5), with each node as a separate JVM where a process had to be signalled. The numbers show that a mechanism works and how its timing is composed; they come from one laptop and are not a benchmark. Spring Boot 3.5.16 manages the same Quartz 2.5.2, so every statement about Quartz is identical there; only the package of the Boot types (org.springframework.boot.autoconfigure.quartz instead of org.springframework.boot.quartz.autoconfigure) and Spring Framework 6.2.19 differ, and this page was not re-run on Boot 3.5.
The official page, item by item
The official page groups its tips under eight headings. The table says what each one means for 2.5.2 and where the rest of this page (or another page of the section) deals with it. One of them is less true than it reads, and is marked.
| Official tip | Verdict for 2.5.2 and Spring Boot 4.1 | Where |
|---|---|---|
Enable bulk job retrieval |
True and measurable: |
|
Only store primitive data types and Strings in the JobDataMap |
True; with the JDBC store a non-serializable value is rejected at scheduling time and a changed class breaks stored data |
|
Use the merged JobDataMap |
True, with one exception: write-back only goes through the |
|
Use |
True; the builders also make the misfire instruction and the time zone explicit |
|
Never write directly to Quartz’s tables |
True; reading is fine and needed |
|
Never point a non-clustered scheduler at a database used by another scheduler with the same name |
True; the mechanism is spelled out on the clustering page |
|
Ensure adequate datasource connection size |
Official rule: worker threads plus three. Not reproduced as a failure in short runs here |
|
Avoid scheduling jobs near daylight-saving transition hours |
True for cron triggers; the engines differ on the repeated hour |
|
Do not wait for conditions inside a job |
True; measured |
|
Throwing exceptions: "Quartz will typically immediately re-execute it" |
Not true as written. An exception does not refire a job; only |
|
Recoverability and idempotence |
True; the crash and the recovery were measured |
|
Keep code in listeners concise; handle exceptions |
True; measured |
|
Be careful of security when exposing the scheduler (the pre-made |
|
Source: Quartz documentation: Best Practices and Tips and the 2.5.2 sources; each row is backed by the section it links to.
Bulk retrieval
The official page recommends the bulk retrieval methods over single-item calls in a loop. In 2.5.2 the bulk methods on Scheduler are getJobKeys(GroupMatcher<JobKey>), getTriggerKeys(GroupMatcher<TriggerKey>) and getJobDetails(GroupMatcher<JobKey>); there is no method that returns the triggers of many jobs at once, so they still cost one call each. On the JDBC store, with 2000 jobs and 2000 triggers in PostgreSQL 17:
| Operation | One call per item | One bulk call |
|---|---|---|
Job keys of the group |
- |
5 |
Job details of the group |
778 ( |
31 ( |
Triggers of every job |
1695 ( |
no bulk method |
Schedule 1000 jobs with one trigger each |
1829 ( |
1724 (one |
Source: Quartz API: Scheduler; milliseconds measured by the test project.
Two lessons. Use getJobDetails for listings and dashboards. And do not expect scheduleJobs to speed up a bulk load: it saves nothing measurable on the JDBC store, because every job and trigger is still stored by its own statements; its value is that the whole set is stored in one call.
Set<JobKey> keys = scheduler.getJobKeys(GroupMatcher.jobGroupEquals("bookshelf")); // cheap: keys only
List<JobDetail> details = scheduler.getJobDetails(GroupMatcher.jobGroupEquals("bookshelf")); // one call, all details
JobDataMap contents
The rules are on Jobs & Job Data; in one paragraph: store identifiers, strings and primitives, never entities, clients or secrets; read from the merged map (job detail overlaid by trigger); write back only through the JobDetail map of a @PersistJobDataAfterExecution job. The consequence for a service is that the map is a contract between a stored row and your class, and the row outlives deployments. The Bookshelf reminder stores two numbers and nothing else:
public static JobDetail reminderJob(long loanId, long memberId) {
return JobBuilder.newJob(MemberReminderJob.class)
.withIdentity("loan-" + loanId, "reminders")
.usingJobData("memberId", memberId) // identifiers and primitives, never entities
.usingJobData("loanId", loanId)
.requestRecovery(true)
.build();
}
Everything else the job needs (the member’s e-mail address, the template) is read when it runs, from the database or the configuration, so a changed address is picked up and nothing sensitive sits in a Quartz table or in the Actuator output. A non-serializable value fails fast (Unable to serialize JobDataMap … is not serializable, measured on the jobs page) and a serializable class that changes shape fails later, when the old row is read after a deployment; that is the second anti-pattern in the table below.
Trigger construction
Build triggers with TriggerBuilder, DateBuilder and the schedule builders instead of instantiating SimpleTriggerImpl or CronTriggerImpl and calling setters. Besides being shorter, the builders force the three decisions that cause production surprises: the time zone, the misfire instruction and the start time. The official page also lists TriggerUtils for analysing triggers; its use is on Triggers & Schedules.
public static Trigger reminderTrigger(JobDetail job) {
return TriggerBuilder.newTrigger()
.withIdentity(job.getKey().getName(), job.getKey().getGroup())
.forJob(job)
.startAt(DateBuilder.futureDate(3, IntervalUnit.DAY))
.withSchedule(SimpleScheduleBuilder.simpleSchedule().withMisfireHandlingInstructionFireNow())
.build();
}
public static Trigger nightlyTrigger(JobDetail job) {
return TriggerBuilder.newTrigger()
.withIdentity("overdueLoanRemindersTrigger", "bookshelf")
.forJob(job)
.withSchedule(CronScheduleBuilder.cronSchedule("0 0 3 * * ?") // 03:00, outside the 02:00 DST window
.inTimeZone(TimeZone.getTimeZone("Europe/Madrid"))
.withMisfireHandlingInstructionFireAndProceed())
.build();
}
Source: Quartz API: TriggerBuilder
The first trigger reported misfire instruction 1 (MISFIRE_INSTRUCTION_FIRE_NOW) when read back. What each instruction means for each trigger type, and the smart policy you get when you set none, is on Misfires.
The tables are not yours
The official page lists the symptoms of writing to the QRTZ_ tables with SQL: corrupt data, jobs that vanish or sit there, deadlocks. The reason (several tables, row locks and the scheduler thread’s in-memory view must agree) is on Job Stores & Persistence. The practical rule has two halves. Change schedules only through the Scheduler API (or the endpoints you build on it, see Dynamic Scheduling & Management). Read the tables as much as you like, with a read-only database user: the monitoring queries of Database maintenance are exactly that. Even a stale row that looks like garbage, such as the fired-trigger row of a node that was killed, is removed by Quartz’s own recovery; it was measured below, and deleting it by hand is the thing the official page warns about.
Datasource size
The official rule is a datasource of at least the number of worker threads plus three, more if the application calls the scheduler API often, and at least four connections for the non-managed datasource of a JobStoreCMT. Under Boot the store is LocalDataSourceJobStore, a JobStoreCMT subclass that borrows connections from the application’s own pool, so the pool is shared with everything else in the service; the arithmetic is on Job Stores & Persistence.
A stress run tried to exhaust the pool. Six workers, six jobs, Hikari with a 1500 ms connection timeout, for 20 seconds, JDBC store:
| Scenario | Hikari maximum-pool-size |
Executions | Scheduler errors and triggers in ERROR |
|---|---|---|---|
Six 200 ms jobs every 2 s |
2 |
64 |
0 and 0 |
same |
4 |
64 |
0 and 0 |
same |
9 (threads plus three) |
62 |
0 and 0 |
Six jobs that each hold a pooled connection for 4.5 s, every 5 s, staggered by 0.8 s, 40 s |
6 |
48 |
0 and 0 |
same |
9 |
48 |
0 and 0 |
Nothing failed, so the finding is a negative one: in these short runs Quartz’s own use of connections is brief and retried, and a pool below the official figure did not cause errors, lost executions or ERROR triggers. That does not repeal the rule. It says the rule is a conservative sizing for the worst moment (every worker completing a job, the scheduler thread acquiring and the cluster check-in all at once) and that the failure, when it comes, is a slow degradation rather than a crash. Use the rule as the minimum, give the pool real headroom for the application’s own traffic (the worked example uses 20 for ten workers), and watch the pool’s wait time in the metrics of Observability rather than waiting for an error.
Daylight saving time
CronTriggers fire at a wall-clock time, so they meet the transition hours. The measured behaviour of both Quartz and Spring cron on the gap day and the repeated day is on Cron Expressions; the rule for production is short. Do not schedule anything that must run once a day between 01:00 and 03:00 local time (the exact hour depends on the zone), or schedule it in UTC; SimpleTrigger and any every-N-minutes schedule are not affected; and on the repeated hour Quartz fires a cron job once where Spring’s @Scheduled fires it twice, so a job that moves between the two must be idempotent either way. The nightly trigger above uses 03:00 for that reason. A CalendarIntervalTrigger or a daily-time-interval trigger slides along the calendar and may shift by an hour instead of skipping, as the official page says.
Waiting for conditions
A Thread.sleep inside a job holds a worker thread for as long as it sleeps, and with the default ten threads a handful of waiting jobs is an outage for everything else. The official advice is to return from the job and let it schedule itself again. Both versions were run against a pool of one thread, a heartbeat job firing every second and a condition that becomes true after 8 seconds:
| Poller | Heartbeat runs (ideal 11) | Poller calls | Condition seen after |
|---|---|---|---|
Sleeping in a loop |
3 |
1 |
8117 ms |
Returning and rescheduling a one-shot trigger 1 s later |
12 |
9 |
8032 ms |
Both saw the condition at the same moment; the sleeping one cost the heartbeat eight of its eleven runs. The rescheduling poller is a few lines:
@DisallowConcurrentExecution
public class ReschedulingPoller implements Job {
@Override
public void execute(JobExecutionContext context) throws JobExecutionException {
if (!conditionHolds()) {
try {
context.getScheduler().scheduleJob(TriggerBuilder.newTrigger()
.forJob(context.getJobDetail()) // the job must be durable or have other triggers
.startAt(new Date(System.currentTimeMillis() + 1000))
.build());
} catch (SchedulerException e) {
throw new JobExecutionException(e, false);
}
return; // the worker thread is free again
}
proceed();
}
}
A cap on the number of attempts (carried in the JobDataMap) keeps an unreachable condition from rescheduling forever. If the condition is "another system has finished", a callback or a message is a better trigger than polling at all.
Throwing exceptions
The official page says that when a job throws, Quartz will typically re-execute it immediately and it will likely throw again. That is not what 2.5.2 does. A job that throws an unchecked exception is logged, wrapped in a JobExecutionException and reported to the listeners, and the trigger fires at its next regular time; only setRefireImmediately(true) re-runs a job at once (see Jobs & Job Data for the flags). Measured again here with a plain job that always throws an IllegalStateException, every 1000 ms for 4.5 seconds:
| Job | Calls | Trigger state afterwards |
|---|---|---|
Uncaught |
5 |
|
|
5 |
|
|
1 |
|
The official advice to catch everything and decide is still right, because a swallowed or unclassified failure either hides a problem or repeats it forever. The classification belongs in the job, in three branches:
public class OverdueLoanRemindersJob extends QuartzJobBean {
private final ReminderService service;
public OverdueLoanRemindersJob(ReminderService service) { this.service = service; }
@Override
protected void executeInternal(JobExecutionContext context) throws JobExecutionException {
try {
service.remindOverdue();
} catch (TransientDataAccessException e) {
// the next trigger will try again; listeners and metrics still see a failure
throw new JobExecutionException("overdue reminders failed, will retry at the next fire time", e, false);
} catch (RuntimeException e) {
JobExecutionException failure = new JobExecutionException("overdue reminders can never succeed", e, false);
failure.setUnscheduleFiringTrigger(true); // stop this trigger, keep the others
throw failure;
}
}
}
For failures that deserve a bounded retry with spacing, use the retry tools of Retry inside the job; do not build a refire loop.
Recoverability and idempotence
A job with requestRecovery(true) is run again by a surviving node after a crash, from the start, and JobExecutionContext.isRecovering() is then true. That is the official "some of the job’s work will be executed twice". The failure was reproduced (measured, PostgreSQL 17, a 30-second recoverable report on one node and kill -9): the fired-trigger row stayed EXECUTING with REQUESTS_RECOVERY true, and a node started 5.5 seconds after the kill took over: the stale rows were gone 9.5 seconds after the kill (clusterCheckinInterval 2 seconds) and its log shows the report running again with recovering=true. The detection arithmetic and the other cases are on Clustering; how to make "run it again" harmless (key the work on the scheduled fire time, claim with a unique constraint, prefer upserts) is on Jobs & Job Data. Without requestRecovery a job that was running when its node died is not run again at all, which is the correct setting for jobs that are cheap to skip and the wrong one for a report that must exist.
Listener hygiene
Listeners run on the worker thread that is about to run, or has just run, the job, so anything slow in a listener is slow for the scheduler. The full list of listener rules is on Listeners & Plugins; the cost of ignoring the first one was measured with one worker thread, a heartbeat every second and a jobToBeExecuted callback that takes 1.5 seconds:
| Listener | Heartbeat runs (of 10) | Longest gap between two runs | Max lag | Misfire events |
|---|---|---|---|---|
none |
10 |
1003 ms |
7 ms |
0 |
slow, inline |
7 |
1520 ms |
3108 ms |
0 |
same work handed to a two-thread executor |
10 |
1004 ms |
7 ms |
0 |
The inline listener cost three of ten runs and delayed the rest by up to three seconds without a single error or misfire event, which is why it is an anti-pattern rather than a bug report. The second rule, never let a listener throw, has its own measurements on the listeners page (a throwing triggerFired silently stops the job). A listener that follows both hands the work to a small bounded executor, drops it when the queue is full, and catches everything:
public class AuditJobListener implements JobListener {
private final TaskExecutor audit; // small, bounded, separate from the Quartz workers
private final Consumer<String> auditClient;
public AuditJobListener(TaskExecutor audit, Consumer<String> auditClient) { this.audit = audit; this.auditClient = auditClient; }
@Override public String getName() { return "audit"; }
@Override public void jobToBeExecuted(JobExecutionContext context) { }
@Override public void jobExecutionVetoed(JobExecutionContext context) { }
@Override
public void jobWasExecuted(JobExecutionContext context, JobExecutionException failure) {
String line = context.getJobDetail().getKey() + (failure == null ? " ok" : " failed");
try {
audit.execute(() -> auditClient.accept(line)); // returns at once
} catch (TaskRejectedException e) {
// queue full: drop the audit line, never fail or delay the job
}
}
}
Run against an executor with one thread, a queue of two and a client that needs 1.5 seconds per line, the heartbeat ran 9 times in 8.5 seconds with a maximum lag of 11 ms while five audit lines were delivered; the others were queued or dropped, and the scheduler never noticed. Boot does not register listener beans by itself; hand the listener to a SchedulerFactoryBeanCustomizer or the ListenerManager as shown on the listeners page.
Exposing the scheduler
The official page warns against letting users define jobs of any type with any parameters, and names org.quartz.jobs.NativeJob, which runs an operating-system command. That class is gone: the 2.5.2 quartz-jobs module contains DirectoryScanJob, FileScanJob, NoOpJob, SendMailJob, the JMS jobs, JMXInvokerJob and the EJB invoker jobs, and no native job. The warning stands for the rest, because a class name from a request plus parameters from a request is remote code execution by configuration. There are three ways a scheduler gets exposed, and each has a rule.
Your own REST or UI. Never turn a request field into a job class, a cron expression you did not validate, or a JobDataMap you did not whitelist. Map a small enum to the classes you allow:
public enum ScheduledTask {
MEMBER_REMINDER(MemberReminderJob.class);
private final Class<? extends Job> jobClass;
ScheduledTask(Class<? extends Job> jobClass) { this.jobClass = jobClass; }
public Class<? extends Job> jobClass() { return jobClass; }
public static Class<? extends Job> resolve(String type) {
try {
return valueOf(type).jobClass();
} catch (IllegalArgumentException e) {
throw new IllegalArgumentException("unknown task type");
}
}
}
resolve("MEMBER_REMINDER") returned the job class and resolve("org.quartz.jobs.FileScanJob") was rejected. The authentication and authorization of the endpoints that call it are on Dynamic Scheduling & Management and Spring Security; method security that is not switched on is silently ignored.
RMI. Quartz can export the scheduler through RMI (org.quartz.scheduler.rmi.export, default false in StdSchedulerFactory). Do not. The exported interface has no authentication, and shutdown(), deleteJob and the rest of the remote surface are callable by whoever reaches the port. A test scheduler with it enabled and createRegistry=always showed:
| Observation | Result |
|---|---|
Sockets opened |
Two listeners, the registry (11099) and the server port (11100), on all interfaces ( |
Registry content |
One entry, the scheduler’s unique id (scheduler name, |
Client without any credentials |
|
Source: Quartz documentation: Configure RMI; the test is the one described in the table. A Java RMI endpoint is also a deserialization surface, which is a separate reason not to open it.
For remote control use your own authenticated endpoint on the service, or the Actuator endpoint below.
The Actuator quartz endpoint. It is not exposed over HTTP by default (measured: GET /actuator listed only health and GET /actuator/quartz returned 404), it masks job and trigger data as while management.endpoint.quartz.show-values is NEVER (the default), and its only write operation is POST /actuator/quartz/jobs/{group}/{name} with {"state":"running"}, which triggers the job once; pause, resume and delete do not exist there. Expose it only behind authentication, and prefer role-based unmasking:
management:
endpoints:
web:
exposure:
include: health,quartz
endpoint:
quartz:
show-values: when-authorized # NEVER (default), WHEN_AUTHORIZED or ALWAYS
roles: SCHEDULER_ADMIN # empty means every authenticated user may see the values
Source: Spring Boot: Quartz Endpoint (REST API) and the QuartzEndpointProperties source of Spring Boot 4.1.1.
The endpoint and its trigger-now operation are covered with their measured responses on Observability. show-values: always belongs on a developer’s laptop and nowhere else, because the job data are exactly what the earlier advice says should not contain secrets, and people put secrets there anyway.
Graceful shutdown
What happens to a running job when the service stops depends on four settings, and the defaults lose the job. A node was started with a recoverable 8-second job, sent SIGTERM (through kill -TERM, so the JVM’s own shutdown hook ran) 1.5 seconds into the job, and the exit time and the fired-trigger row were recorded:
| Configuration | JVM exit after SIGTERM | Job ran to its end | Completion recorded in QRTZ_FIRED_TRIGGERS |
|---|---|---|---|
|
58 ms |
no, cut off |
no: the row stays |
|
6495 ms |
yes |
yes: the row is gone, the job’s own end record was written |
Boot’s |
372 ms |
no, cut off |
no: the row stays |
same executor, wait true, |
6796 ms |
the job body reached its end |
no: the row stays |
Row 2 is graceful shutdown: wait-for-jobs-to-complete-on-shutdown: true makes Spring call Scheduler.shutdown(true), which stops firing, waits for the workers and lets each job record its completion. Row 4 is the trap. With a Spring executor as the thread pool, Quartz’s adapter does not wait (see Spring Boot Configuration): the scheduler reported "shutdown complete" immediately, the executor’s own await-termination kept the JVM alive for the remaining 6.8 seconds, and when the job finally returned the job store was already shut down. The log showed JobStore is shutdown - aborting retry, the job’s own JDBC call failed with CannotGetJdbcConnectionException because the pool was already closed, and the fired row stayed EXECUTING, so a survivor will run a requestRecovery job again although it had in fact finished. await-termination makes the JVM wait; it does not make Quartz record the result. The safe combinations are the default SimpleThreadPool with the wait flag, or an executor together with a requestRecovery job that is idempotent.
Waiting is the right answer for a job of seconds. For a job of minutes it is not, because the platform kills the process after its grace period (terminationGracePeriodSeconds, 30 s by default on Kubernetes, see Clustering and Health Probes & Lifecycle). The answer for those is a job that stops at a safe point when asked: implement InterruptableJob, work in chunks, keep the cursor in the JobDataMap, and let the scheduler call interrupt() on shutdown with org.quartz.scheduler.interruptJobsOnShutdownWithWait (the WithWait property applies when the wait flag is true; the plain interruptJobsOnShutdown applies when it is false; both default to false).
@DisallowConcurrentExecution
@PersistJobDataAfterExecution
public class ChunkJob implements InterruptableJob {
private volatile boolean stopRequested;
@Override
public void execute(JobExecutionContext context) {
JobDataMap data = context.getJobDetail().getJobDataMap();
int cursor = data.getInt("cursor");
int total = data.getInt("total");
while (cursor < total && !stopRequested) {
processChunk(cursor); // one bounded piece of work
cursor++;
data.put("cursor", cursor); // saved when execute returns, @PersistJobDataAfterExecution
}
}
@Override
public void interrupt() {
stopRequested = true; // checked between chunks
}
}
Source: Quartz API: InterruptableJob
A 10-second job in 40 chunks was sent SIGTERM two seconds in, once with only the wait flag and once with interruptJobsOnShutdownWithWait as well; a second node was then started and its first run recorded:
| Configuration | JVM exit after SIGTERM | Last run of node 1 ended with | First run of node 2 started at |
|---|---|---|---|
|
8139 ms |
cursor 40, done |
cursor 40 (nothing left) |
plus |
389 ms |
cursor 9, interrupted |
cursor 9: it resumed where node 1 stopped |
Source: Quartz documentation: Configure the scheduler; the interrupt hook was read in QuartzScheduler.shutdown of 2.5.2.
The interrupting configuration gives the quickest shutdown that still records everything: the job returned after its current chunk, @PersistJobDataAfterExecution stored the cursor, the fired row was cleaned, and the next node continued from chunk 9. That is the pattern the Kubernetes grace period wants, whatever the job’s total length.
Rolling deployments
A rolling deployment replaces nodes one at a time. For a Quartz cluster that works without special handling, provided each stop is graceful and each new node joins with its own instance id (the pod-name pattern is on Clustering). The rule that matters is the order: never take the last node down before a replacement is up, because while no node is alive nothing fires and the triggers misfire later.
Two nodes were started against one PostgreSQL schema (wait-for-jobs-to-complete-on-shutdown true, instanceId AUTO, clusterCheckinInterval 2000, four workers each) running a heartbeat every 2 seconds and a 6-second recoverable report every 15 seconds. The node that was executing the report was stopped with SIGTERM and replaced, then the other node was stopped and replaced. Every execution wrote a row to a shared table, so the result is exact:
| Observation | Result |
|---|---|
Node that was running the report: exit after SIGTERM |
4980 ms (the job finished first and its end was recorded) |
Second node: exit after SIGTERM |
477 ms (it was idle) |
Heartbeats started |
30 distinct scheduled fire times, each started exactly once, longest gap between consecutive fire times 2000 ms: none skipped, none duplicated |
Report runs |
4 starts, 4 ends, no run flagged as recovering, no scheduled time started twice |
Misfire or |
0 |
|
2 rows: the stopped nodes' rows had been removed by the survivors |
Four practical points follow. Keep maxUnavailable at 0 (or one fewer than the replica count) so a node is always alive; a pod’s grace period must cover the longest job that does not stop at a chunk boundary; set readiness so a new pod receives traffic only after the scheduler started (the rollout strategy is on Deployment Strategies); and make schema changes backward compatible with the old nodes that are still running, because during the rollout the two versions share the same Quartz tables and the same job classes may even be serialized by one and read by the other. A job class that is renamed or removed leaves rows that name a class the new version cannot load (not run here), so rename in two releases, never in one. The behaviour of a node that crashed instead of stopping is on the clustering page and above under recoverability.
Thread pool sizing
org.quartz.threadPool.threadCount (default 10 under Boot, measured) is the number of jobs that can run at the same moment, and it is a real cap: the scheduler thread waits for a free worker before it fires the next trigger. A long job therefore does not only take a thread, it moves the others later. Two jobs of 6 seconds were fired together with a heartbeat every second, with two, three and four workers:
threadCount |
Heartbeat runs (of 14) | Longest gap between two runs | Heartbeat misfire events |
|---|---|---|---|
2 |
8 |
6996 ms |
1 |
3 |
14 |
1006 ms |
0 |
4 |
14 |
1004 ms |
0 |
With two workers the heartbeat was shut out for seven seconds and one misfire event covered the lost ticks (the run count varied between 8 and 9 across repeated runs). Because the in-memory store’s misfireThreshold is 5000 ms, a pool that is saturated for longer than that turns on-time triggers into misfires; the JDBC store’s default is 60000 ms (see Misfires).
The second way a pool is too small is a burst. The pool must drain the largest burst of triggers that share a fire time inside the misfire window: the time needed is the number of triggers times the job duration divided by the thread count. Four hundred one-shot triggers of 200 milliseconds (think of the per-member reminders at 08:00) were due at the same instant:
threadCount |
Time until the last job started (arithmetic: 400 x 0.2 s / threads) | Misfire events |
|---|---|---|
10 |
8022 ms (8.0 s) |
150 |
20 |
3908 ms (4.0 s) |
0 |
50 |
1447 ms (1.6 s) |
0 |
The measured times match the arithmetic closely. With ten threads the last triggers started eight seconds after they were due, so 150 of them were counted as misfires (a one-shot SimpleTrigger under the smart policy still fires them, only late, see the misfires page). The sizing method that follows: list the jobs that can be due together, work out the number of workers that drains the peak inside a delay you accept, add the long jobs that hold a worker for their whole run, and keep the database pool above that number plus three. Two hard limits apply: every extra thread uses a database connection when its job completes, and more nodes do not add throughput past about three, as explained on Clustering. Raise the thread count before adding nodes. Jobs that mostly wait for HTTP or database calls are the case for virtual threads and an executor, with the shutdown trap of the previous section and the other traps (no cap unless concurrency-limit is set, a rejecting executor sends triggers to ERROR) described on Spring Boot Configuration.
Long-running jobs and Spring Batch
A Quartz job is a good trigger and a poor batch engine. Quartz knows when to start a job and whether it ran; it does not know how far it got, cannot restart from the middle, keeps no per-item state and offers no skip or retry policy. Three situations, three answers:
| The work | Use | Why |
|---|---|---|
Seconds, a few hundred rows, safe to repeat |
A plain job with an idempotent service |
Nothing to resume |
Minutes, one pass over a large set that can be cut into bounded pieces, a cursor is enough to resume |
A chunked |
It stops at a safe point on shutdown and on a failover continues from the stored cursor; no extra infrastructure |
Hours, several steps, skip and retry, restart after failure, an audit trail, parallel partitions |
Spring Batch, launched by a thin Quartz job |
Spring Batch stores step and item state in its own job repository and restarts a failed job instance; see Stopping, Restart and Recovery and Scaling and Parallel Processing |
Source: Spring Batch: Running a Job
When a Quartz job launches a Spring Batch job, the three rules of Jobs & Job Data apply (identifying parameters from the scheduled fire time, treat "already complete" as success, @DisallowConcurrentExecution), and so does the pool arithmetic: a synchronous launch holds a Quartz worker for the whole run. Launch the batch asynchronously, or give it a scheduler with its own small pool, when runs take hours, and see Running a job for the launcher side. Whichever you choose, give the job a deadline and an alert on it (see Observability): a job that never finishes does not fail, its trigger just stays BLOCKED (as in Q2 above), because a @DisallowConcurrentExecution job cannot start while it runs, and nothing alerts.
Database maintenance
Quartz needs little upkeep, and the upkeep is mostly reading. The four queries below answer the questions an on-call engineer asks (who is alive, what is stuck, what is late, are the churn tables healthy). They were run on PostgreSQL 17 against a node executing a 30-second report, so the output is real.
-- Q1: which nodes are alive? (checkin_age_s must stay below a few intervals)
select instance_name,
round((extract(epoch from now())*1000 - last_checkin_time)/1000) as checkin_age_s,
checkin_interval/1000 as interval_s
from qrtz_scheduler_state order by last_checkin_time;
-- Q2: how many triggers per state? (anything in ERROR needs a person)
select trigger_state, count(*) as n from qrtz_triggers group by trigger_state order by trigger_state;
-- Q3: waiting triggers that are overdue (a scheduler that is not keeping up)
select trigger_group, trigger_name,
round((extract(epoch from now())*1000 - next_fire_time)/1000) as late_s
from qrtz_triggers
where trigger_state = 'WAITING' and next_fire_time < (extract(epoch from now())*1000 - 60000);
-- Q4: fired rows and the check-in of their node (the stale ones are of a node that stopped checking in)
select f.instance_name, f.job_name, f.state, f.requests_recovery,
round((extract(epoch from now())*1000 - f.fired_time)/1000) as fired_s_ago,
round((extract(epoch from now())*1000 - s.last_checkin_time)/1000) as checkin_s_ago
from qrtz_fired_triggers f
left join qrtz_scheduler_state s on s.sched_name = f.sched_name and s.instance_name = f.instance_name
order by f.fired_time;
Source: PostgreSQL: The Cumulative Statistics System; the table and column names come from tables_postgres.sql of Quartz 2.5.2.
On the healthy node Q1 returned one row with a check-in age of 1 s and an interval of 2 s, Q2 returned ACQUIRED 1 and BLOCKED 1 (the report’s own trigger is blocked while the report runs, because the job is @DisallowConcurrentExecution), Q3 returned nothing, and Q4 returned the running report as EXECUTING with REQUESTS_RECOVERY true plus an ACQUIRED row whose JOB_NAME is null (Quartz fills the job columns of a fired row only when the job starts executing).
Then the node was killed with kill -9 and the same Q4 was run four seconds later, with no node alive:
job_name |
state |
requests_recovery |
fired_s_ago |
checkin_s_ago |
Reading |
|---|---|---|---|---|---|
|
|
true |
7 |
5 |
stale: the node stopped checking in (interval 2 s) |
null |
|
false |
5 |
5 |
stale too |
The right reaction is to do nothing to the table. A node was started 5.5 seconds after the kill; its first check-ins found the dead node, recovered its fired rows and removed them, and the report ran again with recovering=true. The stale rows were gone 9.5 seconds after the kill and QRTZ_SCHEDULER_STATE held only the new node. Deleting the rows by hand would be a write to Quartz’s tables; the way to clear them is to have a node running. If no node can run, restart the service, which recovers its predecessors' rows on start (see Clustering for the arithmetic and for why a non-clustered start would be the wrong way to do it).
What does need regular care is the churn. QRTZ_TRIGGERS and QRTZ_FIRED_TRIGGERS are small tables with a constant stream of updates and deletes. On PostgreSQL that leaves dead row versions behind, and the measurements on Quartz on PostgreSQL are the reference: with autovacuum off, 975 executions left about 1957 dead tuples in qrtz_triggers; with the default settings it stayed around a hundred; and one idle transaction anywhere in the database kept autovacuum from removing anything (about 2700 dead tuples after a minute even while it ran), so the real maintenance task is to keep sessions from sitting in a transaction (idle_in_transaction_session_timeout, discussed there). The table statistics are the last check:
select relname, n_live_tup, n_dead_tup, last_autovacuum is not null as autovacuumed
from pg_stat_user_tables where relname like 'qrtz_%' order by relname;
The healthy node had a handful of dead tuples in qrtz_triggers (8 against 2 live) and none that mattered elsewhere, on a minute-old schema where autovacuum had not run yet; alert on the dead tuples growing over time, not on a single reading. The other databases have their own housekeeping (index and statistics maintenance on Oracle and SQL Server, purge of old row versions in InnoDB), which was not measured for this section; their pages say what was and was not run. Two last items: the Quartz tables belong in the same backup as the application data, because a restored application with a schedule from a different moment is a source of duplicate or lost runs, and a Quartz database that is not used any more (a migration finished, a scheduler name retired) is cleaned through the scheduler API (clear(), deleteJobs) of a node that runs it, not by dropping rows.
Worked example: the Bookshelf production profile
Everything above comes together in one profile. The Bookshelf service runs two or three pods against PostgreSQL, with tables created by a migration tool, the recoverable jobs marked as such, the long retention clean-up written as a chunked job, and shutdown that records its results. The file below was started as a node against PostgreSQL 17 with the credentials passed as environment variables (the tables were created by a first start with initialize-schema=always as a command-line override, standing in for the migration tool); it logged the settings it applied.
spring:
datasource:
url: ${DB_URL}
username: ${DB_USER}
password: ${DB_PASSWORD}
hikari:
maximum-pool-size: 20 # ten workers + 3 for Quartz, the rest for application traffic
quartz:
job-store-type: jdbc
scheduler-name: BookshelfScheduler
wait-for-jobs-to-complete-on-shutdown: true
jdbc:
initialize-schema: never # Flyway or Liquibase owns the tables
properties:
org.quartz.scheduler.instanceId: AUTO
org.quartz.scheduler.interruptJobsOnShutdownWithWait: "true"
org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
org.quartz.jobStore.isClustered: "true"
org.quartz.jobStore.clusterCheckinInterval: "5000"
org.quartz.threadPool.threadCount: "10"
Source: Spring Boot: Quartz Scheduler
Started with the prod profile, the log reported Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads. and Using job-store 'org.springframework.scheduling.quartz.LocalDataSourceJobStore' - which supports persistence. and is clustered., the scheduler took the host-name-based instance id, and a SIGTERM during the chunked job ended the JVM after 395 ms (the behaviour of the chunked job is the measurement of the shutdown section). Each line of the file answers an item of this page, and the table maps them:
| Setting | Answers |
|---|---|
|
Anti-pattern 1 (the default |
|
Graceful shutdown: short jobs finish, long jobs stop at a chunk boundary |
|
Rolling deployments without a lost or doubled tick; every node clustered (anti-pattern 3) |
|
|
|
Fail-over between 7.5 s and 2 x 5 s + 7.5 s, depending on when in the interval the node died; synchronise the clocks (anti-pattern 4) |
Anti-patterns
Seven mistakes cover most of the Quartz incidents a team can cause. Each row says what goes wrong (with the evidence of this section), what to do instead and where it is explained.
| Anti-pattern | What goes wrong | Do instead | Read more |
|---|---|---|---|
Anti-pattern 1: |
The script begins with |
|
|
Anti-pattern 2: Non-serializable (or class-coupled) objects in the |
With the JDBC store the map is a serialized BLOB: a |
Identifiers and primitives only, resolve everything else inside the job |
|
Anti-pattern 3: Clustered and non-clustered schedulers on the same tables |
A scheduler without |
Every node of a scheduler name clustered, one scheduler name per environment, never a non-clustered test run on a production schema |
|
Anti-pattern 4: Unsynchronised clocks |
Fire times and check-in times come from each node’s own clock and failure detection compares the observer’s clock with the dead node’s last check-in. An observer whose clock is ahead by more than the 7.5 s grace period takes a live node for dead and "recovers" it while it still runs its jobs, so a recoverable job can run twice; one that is behind only delays fail-over. Reasoned from the 2.5.2 source on the clustering page, not run (a laptop clock cannot be skewed safely) |
A time service on every node (NTP or the platform’s), clocks within one second, monitored |
|
Anti-pattern 5: Heavy work (or an exception) in a listener |
Listeners run on the worker thread. A 1.5-second |
Hand the work to a bounded executor, catch everything, alert on |
|
Anti-pattern 6: Relying on an unmaintained NoSQL store without tests |
|
PostgreSQL (or another JDBC database) for the schedule and the NoSQL database for application data through |
|
Anti-pattern 7: Quartz where |
A Quartz scheduler adds a datasource, eleven tables, a thread pool of its own, a cluster protocol and an upgrade path to carry for what could be an annotation. A fixed-rate cache refresh or a nightly clean-up that runs in every instance, or once with a lock, needs none of it |
|
Source: Quartz documentation: Best Practices and Tips and Spring Boot: Quartz Scheduler; the measurements are the ones of the linked pages and of this page.
What changed
-
Quartz 2.5.2 against the official page.
NativeJobis gone fromquartz-jobs; the "throwing exceptions re-executes immediately" sentence is wrong for 2.5.2 (an exception does not refire a job, measured above); the bulk methods exist asgetJobKeys,getTriggerKeysandgetJobDetails, and the last one is the one that saves time. The official clustering default of 15000 ms for the check-in interval is 7500 ms in the code (see Clustering). The other discrepancies between the Quartz documentation and 2.5.2 are collected on Migration & What’s Changed. -
Spring Boot 3.5 against 4.1. The same Quartz 2.5.2, the same
spring.quartz.*keys and the same shutdown behaviour ofSchedulerFactoryBean, and the same Actuator keys. The Boot types live inorg.springframework.boot.autoconfigure.quartz(and the endpoint inorg.springframework.boot.actuate.quartz) on 3.5.16, andspring.quartz.jdbc.continue-on-errordoes not exist there (the initializer hard-codes true). This page’s measurements were not repeated on Boot 3.5. -
Shutdown with a Spring executor.
spring.task.execution.shutdown.await-terminationkeeps the JVM alive for the running jobs but does not make Quartz record their completion (the fourth row of the shutdown table above).
Where to go next
-
Observability for the metrics, health checks and alerts that turn this page’s thresholds into signals, and Testing Quartz Jobs for tests that catch the silent failures of the anti-pattern table.
-
Spring Boot Configuration for every property used in the profile above.
-
Scheduling Cheat Sheet for the one-page summary.