Observability

Before building anything it helps to know which source answers which question, and for which part of the system. The first diagram shows the sources; the metrics pipeline further down shows what is done with the numbers. Boxes in green are provided by Spring Boot or Quartz itself, boxes in orange are the code written on this page.

Two scope rules recur through the page and are worth stating once. Everything that comes from a listener or from SchedulerMetaData describes the node it ran on, so a metric must be aggregated over the instances at query time, and getCurrentlyExecutingJobs (like Scheduler.interrupt) never sees a job running on another node. Everything that is stored in the job store (triggers, fire times, check-ins) is visible to every node, which is why the endpoint and the SQL queries are the right tools for questions about the cluster.

Spring Boot registers a QuartzEndpoint with the id quartz when an Actuator and a Scheduler bean are both present. In Boot 4.1.1 it lives in org.springframework.boot.quartz.actuate.endpoint, its auto-configuration in org.springframework.boot.quartz.autoconfigure, and the properties in the management.endpoint.quartz namespace.

Plain @Scheduled methods have their own, smaller set of signals, and it is worth seeing them next to Quartz’s because they answer the same questions with less. The test project carries two @Scheduled methods next to the Quartz jobs: a fixed-delay re-index and a cron session expiry (@EnableScheduling is on the application class).

Source: Spring Boot Actuator API: Scheduled Tasks

The response is grouped by the kind of task and carries, for the tasks that have run, the status and time of the last execution. The cron entry above is the 05:00 Europe/Madrid expression, reported one millisecond before 03:00 UTC, which is how Spring computes the next instant. The endpoint lists only what Spring’s ScheduledTaskHolder beans know about: the Quartz jobs are not in it (they are in the quartz endpoint), and, unlike the Quartz endpoint, it has no path to run a task and no per-task detail beyond what is shown.

For metrics, Spring Framework 6.1 introduced an observation named tasks.scheduled.execution, and Spring Boot’s observation auto-configuration enables it when an ObservationRegistry exists: it sets the registry on the ScheduledTaskRegistrar through a SchedulingConfigurer bean. A test run produced these series for the re-index method (Prometheus format, abbreviated):

Source: Spring Framework: Observability Support

The low-cardinality tags are code.function, code.namespace, exception and outcome (SUCCESS, ERROR or UNKNOWN), read from DefaultScheduledTaskObservationConvention. So @Scheduled gets duration, outcome and exception for free, and nothing of the kind exists for Quartz jobs: the tasks.scheduled.execution observation applies only to tasks registered with Spring’s own registrar and never fires for a Quartz job (the tasks_scheduled_execution series of the same run contain only the @Scheduled method and none of the Quartz job classes). The listener-based meters of the next section close that gap, and they deliberately use a similar shape (a timer with an outcome tag) so that one dashboard can show both worlds. What @Scheduled still lacks (a persisted last fire time, a cluster-wide view, misfire accounting) is the content of Standard Scheduling vs Quartz; its locking companion is on Scheduling & ShedLock.

On a fresh application with the Actuator and Micrometer on the classpath, the registry contained no meter whose name contains quartz, and no bean related to a Quartz health indicator existed; the only Quartz beans were the auto-configuration, the quartzScheduler bean and the properties (measured, and confirmed by listing the Quartz classes of the Boot 3.5.16 and 4.1.1 Actuator and auto-configuration jars: only the endpoint and its properties). Nothing in Micrometer binds a Quartz scheduler either. The pipeline that the next sections build is this:

A health indicator answers a different question from a metric: should this instance receive traffic, or be restarted? For a scheduler there are three honest states: running (UP), deliberately not firing (standby, or not started yet: OUT_OF_SERVICE) and broken (shut down, or the store cannot be reached: DOWN). With a clustered JDBC store a fourth fact matters: whether this node still checks in. A node whose LAST_CHECKIN_TIME is older than the check-in interval plus the 7.5-second grace that JobStoreSupport adds is a node its peers will treat as failed and recover (see Clustering); the indicator reads its own row.

Source: Spring Boot API: HealthIndicator

The bean is named quartz in the configuration above, so Boot publishes it as the component quartz of /actuator/health. The imports are the Spring Boot 4 ones: Health and HealthIndicator live in org.springframework.boot.health.contributor, where Spring Boot 3.5 has them in org.springframework.boot.actuate.health. The four situations were produced and measured:

Two operational decisions come with the indicator. First, which probe it belongs to. A failing Quartz check on the liveness probe makes Kubernetes restart the pod, which is wrong when the cause is a database outage that the restart cannot fix; the readiness group is the right place, and management.endpoint.health.group.readiness.include=readinessState,quartz made /actuator/health/readiness report the component (measured: status UP, components quartz and readinessState). For the probe background see Health Probes & Lifecycle. Second, the indicator is not a replacement for the alert on a job that did not run: a scheduler that is started, not in standby and checking in can still have a trigger in the ERROR state or a thread pool that is saturated, which the metrics and the table queries below catch.

Quartz’s own log lines (scheduler start, misfire handling, cluster recovery) use SLF4J and appear in the application log under org.quartz. For a line per execution there are the two history plugins LoggingJobHistoryPlugin and LoggingTriggerHistoryPlugin; their properties, message formats and output are on Listeners & Plugins, and the verdict there stands: three INFO lines per execution are fine for a development or low-volume scheduler and too noisy for a job that fires every second. What the plugins cannot do is label the job’s own log lines. A log line such as "Sent overdue-loan reminders" says nothing about which trigger, which fire or which node produced it, and with several jobs running in parallel on a pool of worker threads the thread name is the only hint.

The MDC (the SLF4J mapped diagnostic context, a thread-local map that the log pattern can print) solves it, and a JobListener is the natural place to fill it, because jobToBeExecuted, the job and jobWasExecuted all run on the same worker thread (read from JobRunShell; the sequence is in the tracing section below).

Source: Logback manual: Mapped Diagnostic Context

The log pattern prints the context with %X; Spring Boot’s logging.pattern.level is the property to extend (see Logging for the rest of the logging set-up):

Source: Spring Boot: Logging

The measured log lines of the test project (the application name, thread and trace columns are Boot’s; the bracket after the level is the MDC). The lines of the Quartz jobs carry the job key and the @Scheduled line, which has no Quartz context, prints an empty bracket:

Source: Spring Boot: Logging

With structured logging no pattern is needed: Spring Boot’s structured formats (logging.structured.format.console) write the whole MDC as fields. With the logstash format the same job produced one JSON object per line, with the three MDC keys next to the trace and span ids (reformatted for reading):

Source: Spring Boot: Logging

Three traps belong to the MDC. It is a thread-local, so work the job hands to another thread (an executor, CompletableFuture) does not carry the context unless that pool copies it. The listener must clean up: jobExecutionVetoed and jobWasExecuted do, but if a listener registered after this one throws in jobToBeExecuted the job does not run and jobWasExecuted is never called (see "When a listener fails" on the listeners page), so the key stays on the worker thread until the next job on that thread overwrites it; that is visible in the lab: the error logged by the scheduler listener carried the key of the job that never ran. Wrap the body of every other listener in a try/catch. And the fire id is not a UUID: the RAM store hands out a counter seeded from the clock, and the JDBC store the instance id followed by such a counter (getFiredTriggerRecordId in both stores), so it identifies one firing of one scheduler and means little without the instance name.

A trace follows one unit of work across code and services. A Quartz job has a property that makes it special: nothing calls it. There is no incoming HTTP request to carry a trace context, so without help every job execution starts and ends without a trace, and the calls the job makes (a database, a mail service) cannot be tied to the execution that caused them. The fix is again a listener, this time one that opens a Micrometer Observation for each execution. An observation is Micrometer’s single instrument for timing and tracing: the same start and stop produce a timer through the metrics handler and a span through the tracing handler, so one listener gives both and replaces the Timer of the metrics listener if tracing is wanted.

Source: Micrometer: Observation

The scope matters. openScope makes the span current on this thread, so the log lines of the job get the trace id and any outgoing call the job makes through an instrumented client joins the trace; jobWasExecuted closes the scope on the same thread and stops the observation (with error(exception) when the job failed, which becomes the error tag and the span’s error). The lab added io.micrometer:micrometer-tracing-bridge-brave and org.springframework.boot:spring-boot-micrometer-tracing-brave (any bridge works; the OpenTelemetry route is described on Metrics & Observability) and set management.tracing.sampling.probability to 1.0, because the default samples only a tenth of the traces. Spring Boot then added the trace and span id to every log line (the bracket after the thread column in the log lines above) and, because both the metrics and the tracing handler are present, the observation also produced a timer and a long-task timer:

Source: Micrometer Tracing

The long-task timer (quartz_job_active_seconds_*) is the one that tells you a job is running now and for how long: its max is the age of the oldest running execution, which is the signal behind the "job running too long" alert below. With an 8-second job in the lab the series read quartz_job_active_seconds_count 1 and quartz_job_active_seconds_max 2.8185435 2.8 seconds into the run and 0.0 after it finished (measured). The lab ran both the metrics listener and the observation listener to show both outputs; in a real application pick one of the two timers (the observation is the better choice when tracing is on, the plain timer when it is not).

The sequence for one execution, in the order the listeners were registered (read from JobRunShell and the registration order above), is the following. The trigger listeners run before the job listeners, the job listeners are notified in registration order on both sides, and the trigger listeners are completed last:

Two limits of tracing deserve a sentence each. Every execution is a root span: Quartz does not propagate a trace from the code that scheduled the job, so a reminder scheduled in an HTTP request and fired a week later is not linked to that request. If the link matters, store the trace id of the scheduling request as a JobDataMap entry and attach it to the observation as a high-cardinality key-value in the listener; the span is then searchable by the id of the request that created the reminder. And high-cardinality values (the full job key, the fire id) go into highCardinalityKeyValue, which tracing keeps and the metrics handler drops, so the job key can be on the span without exploding the timer’s tags.

Quartz can register the scheduler as an MBean in the platform MBean server. It is off in the Quartz default and, under Spring Boot, is switched on through the raw Quartz properties (Boot’s spring.quartz.properties map, see Spring Boot Configuration):

Source: StdSchedulerFactory source at v2.5.2

The keys org.quartz.scheduler.jmx.export, org.quartz.scheduler.jmx.objectName and the two jmx.proxy keys are read from the 2.5.2 source of StdSchedulerFactory; the Quartz configuration reference does not describe them. The default object name is quartz:type=QuartzScheduler,name=<scheduler name>,instance=<instance id> (QuartzSchedulerResources.generateJMXObjectName), so a cluster exports one MBean per node, each with its own instance id. The lab read the MBean through the standard javax.management API in the same JVM, 12 seconds after start (the attributes of the interface QuartzSchedulerMBean; instance id and name are those of the RAM store):

Source: Quartz API: QuartzSchedulerMBean

Four properties of the MBean decide how useful it is. The MostRecentSample attributes and PerformanceMetrics are *sampled statistics and are empty until SampledStatisticsEnabled is set to true (it is false by default; the lab switched it on and waited for a sample period before reading). The MBean has operations, not only attributes: the lab listed triggerJob, pauseJobGroup, pauseAllTriggers, resumeJob, deleteJob, unscheduleJob, interruptJob, standby, shutdown and more, so unlike the Actuator endpoint JMX can pause and delete, which makes it a management interface that must be as protected as the application itself (never expose a JMX port without authentication and TLS). It is per node, like the listeners. And it is the right tool for an interactive look with jconsole or a JMX exporter, and the wrong one for dashboards and alerts, where the Micrometer meters above are easier to aggregate; Quartz also offers a JMX proxy scheduler (the jmx.proxy keys) that lets a client scheduler operate a remote one through these MBeans, which is a management tool and not part of observability.

With a JDBC store the QRTZ_ tables are the one place that knows the whole cluster, and four read-only queries answer most operational questions. They are reads: nothing on this page writes to the Quartz tables, because the scheduler owns them (see Job Stores & Persistence). The queries were run against the PostgreSQL 17 database of the verification project with the lab application (JDBC store, clustered, PostgreSQLDelegate); the time arithmetic is PostgreSQL’s, and the instance id is shortened.

Source: Quartz configuration: JDBC JobStore clustering

Source: Quartz configuration: JDBC JobStore clustering

Source: Constants source at v2.5.2 (the state names)

Source: Constants source at v2.5.2 (the state names)

Source: Constants source at v2.5.2 (the state names)

Source: Constants source at v2.5.2 (the state names)

Source: StdJDBCConstants source at v2.5.2

Source: StdJDBCConstants source at v2.5.2

The QRTZ_FIRED_TRIGGERS result shows what a healthy table looks like: a row in the ACQUIRED state (the trigger a node has claimed and will fire next, with a NULL job name) and, while a job runs, a row in the EXECUTING state for it. A row that stays EXECUTING for far longer than the job takes, or a row whose instance_name is no longer in QRTZ_SCHEDULER_STATE, is a job that hung or a node that died; after a crash the survivor’s recovery deletes the dead node’s rows once it declares the node failed. The four queries are the basis for a database exporter or for a MeterBinder that reads the store directly (preferable to the API-based trigger-state gauge when the store holds thousands of triggers). Do not run them against a production store without an index-friendly sched_name predicate, as written, and keep their frequency modest; they take no Quartz lock, but a heavy query on the store a scheduler depends on is not free.

An alert is useful when it fires for a condition a person can act on, and a scheduler has a characteristic failure that is silent: when it stops firing it produces no error at all. Two kinds of alert are needed, and the second is the one that gets forgotten. Something went wrong alerts fire on events (a failure, a misfire, a scheduler error). Something did not happen alerts fire on the absence of a success, a fire time or a check-in. The table lists the alerts of the Bookshelf and their sources; the rules that follow implement the Prometheus ones and were checked with promtool check rules (8 rules found, no errors), and their expressions were run against a live Prometheus.

Source: Prometheus: Alerting rules

Three of the alerts in the table depend on facts that were measured and are worth restating. The thread-pool case: with a bounded and rejecting executor the scheduler logged ThreadPool.runInThread() return false! and moved the triggers it could not hand over to the ERROR state (4 of 6 in the test of the Spring Boot configuration page), so a saturated pool shows up first as quartz_triggers with state="error", not as a failing job. The ERROR state is permanent until Scheduler.resetTriggerFromErrorState is called; no timeout resets it. And the misfire counter has the Prometheus subtlety stated in the diagram: a counter series appears only after the first increment, and increase() over a series that first shows up with the value 1 reports 0 for that first event, which is why the lab’s increase(quartz_trigger_misfired_total[10m]) returned 0 although the counter already showed 3 (Prometheus saw the series for the first time at its final value and had no earlier sample to subtract); registering the counter for each known trigger group at start-up (registry.counter(…​) with the group tag) makes the first event visible.

Putting the pieces together for the five jobs of the scenario. The "proof of life" is the signal that says the job ran when it should; it is chosen by the job’s period, because the window of the query must fit inside the retention of the metrics store.

The three health layers combine as follows: the health indicator in the readiness group takes a node out of rotation when its scheduler is shut down or has stopped checking in; the Prometheus rules page a person for a job that did not run; and the endpoint with previousFireTime is the thing the person opens first, because it shows in one request which triggers are NORMAL, which PAUSED and which ERROR, and when each fired last.

Compared with the older advice that circulates in articles: the Quartz endpoint is read-only up to Spring Boot 3.4 and has the trigger-now operation from 3.5; management.endpoint.<id>.enabled was replaced by access in 3.4 (and deprecated, not removed); health and security classes moved packages in Spring Boot 4; and the Quartz documentation’s configuration reference does not describe the JMX properties, which are in the source of StdSchedulerFactory. Nothing in Quartz 2.5.2 itself changed for this page compared with 2.4.x: the listener, scheduler-state and MBean interfaces used here are the same. Spring Boot 4.1 still ships no Quartz meters or health indicator, so the code on this page applies to both lines, with the import changes in the table.