Real-time Monitoring with Backend Listener

A CLI run (the way every load test should run, as explained in GUI & Test Plan Workflow) prints a summary + line every 30 seconds and writes a JTL file that becomes an HTML dashboard once the run ends. For a ten-minute Bookshelf load test that is fine. For a two-hour soak test, a stress test that climbs until something breaks, or a test against a shared environment, waiting until the end is expensive:

JMeter’s answer since 2.13 is the Backend Listener: an asynchronous listener that aggregates sample results in the load generator and pushes the aggregates to a time-series database, from which Grafana draws live graphs. The application under test exports its own metrics (Spring Boot Actuator with Micrometer, Quarkus Micrometer) to Prometheus, and Grafana reads both.

The solid path (InfluxDB) is the one this page builds; Graphite is the older alternative and is described in its own section. The JMeter side ends at the database: the HTML dashboard described in HTML Dashboard Report is still the record of the run, and the live graphs are a view on aggregates that are deliberately lossy (see How the aggregates are computed).

InfluxdbBackendListenerClient (since 3.2) aggregates results per sampler label and per status and writes them to InfluxDB in line protocol over HTTP, by default every 5 seconds. InfluxDB 2.x support (the influxdbToken parameter) arrived in 5.2.

Both aggregating clients (InfluxDB and Graphite) keep per-label statistics and send them every send interval. The counts (count, hit, sb, rb, errors) are reset after each send. The response-time statistics (avg, min, max and the percentiles) are computed over a sliding window whose behaviour is set by properties:

Source: Properties Reference: BackendListener configuration

Worked example of the default window. The Bookshelf load produces about 15.6 samples per second, or 78 per 5-second interval. In fixed mode the all percentiles describe the last 100 samples, about 6.4 seconds, which is close to the interval. TX Login heavy is 10 % of the journeys; at about 1.5 samples per second, its 100-sample window spans about 65 seconds, so its p95 line in Grafana is a one-minute moving percentile that reacts to a regression a minute late and still shows it a minute after it is over. In timed mode each point describes exactly the samples of its interval, but with 7 or 8 samples per 5 seconds for that transaction a "p99" is just the maximum. Neither mode gives exact percentiles over the run; the HTML dashboard, computed from the JTL, does.

To make the live graphs closer to the report, either raise the window (for example backend_metrics_window=1000) or switch to the time-boxed mode:

Source: Properties Reference: BackendListener configuration

Cost on the load generator. Each tracked label keeps up to three windows (all, ok, ko) of backend_metrics_window doubles, about 2.4 KB at the default of 100, or up to 3 × 5000 doubles (120 KB) in timed mode. Ten labels cost well under 2 MB either way. CPU is dominated by sorting the windows for percentiles once per interval. The aggregating clients are cheap; what costs is a high number of labels (do not let samplersRegex match dynamic labels such as GET /api/books/1234) and the raw client at high throughput.

GraphiteBackendListenerClient was the first implementation (2.13) and is still the default class in the GUI. It sends to Graphite (Carbon) or to an InfluxDB 1.x server with the Graphite input enabled.

Source: GraphiteBackendListenerClient parameters (Component Reference) and Graphite Sender (User Manual)

Installing Grafana and provisioning data sources are covered in Grafana: Dashboards & visualization; only the JMeter-specific part follows.

Community dashboard. The JMeter manual shows a Grafana dashboard without naming one. A widely used community dashboard is Apache JMeter Dashboard using Core InfluxdbBackendListenerClient on grafana.com (ID 5496, published by Philippe M. of Ubik Load Pack). It uses InfluxQL, asks for an InfluxDB data source pointing at the jmeter database and for two constants, the measurement name (jmeter) and the send interval (5, which must match backend_influxdb.send_interval). It is not part of JMeter and is not maintained by the Apache project: import it with Dashboards → New → Import, enter 5496, choose the data source, and review the queries before relying on them.

Your own panels. Writing the few panels that matter teaches the schema and avoids surprises. These InfluxQL queries assume the InfluxQL data source provisioned in the Compose stack below (database jmeter) and a dashboard variable runId built from SHOW TAG VALUES FROM "jmeter" WITH KEY = "runId":

Source: Grafana configuration (User Manual) and InfluxDB data source (Grafana documentation)

A sensible dashboard has one row per question: Is the load the one requested? (active threads, throughput), Is it working? (error %, errors by code), How fast? (p95 and p99 per transaction, all average), and What is the server doing? (the Prometheus panels below). Set the dashboard refresh to 5 s, the send interval; refreshing faster only redraws the same points.

The Backend Listener measures the client side. The causes of a slowdown (CPU, heap and garbage collection, a full connection pool, a saturated request thread pool) live in the server’s metrics. Bookshelf, as a Spring Boot application, exposes them through Actuator and Micrometer at /actuator/prometheus; the setup, the meters and the Prometheus scrape configuration are on Metrics and Observability: Micrometer, Actuator, and Distributed Tracing. A Quarkus service exposes the same kind of meters at /q/metrics, see Observability: Health, Logging, Metrics, Tracing. Prometheus itself is covered in Prometheus & Monitoring, starting with Installing & running Prometheus, Instrumenting applications and PromQL: Functions. The workflow for reading the two timelines together is on Performance Testing with Apache JMeter in the Spring Boot section; it is not repeated here.

What this page adds is putting both on one Grafana dashboard:

Source: Metrics (Spring Boot reference) and Querying Prometheus (Prometheus documentation)

The application label assumes that Bookshelf adds a common application tag to its meters (Spring Boot’s management.metrics.tags.application property, or a MeterRegistry customizer); adapt the selectors to your own labels. The server’s http_server_requests_seconds_count rate is a useful cross-check: it should match JMeter’s throughput panel. If JMeter reports 15.6 requests per second and the server counts 12, requests are being lost or queued in between (a proxy, a connection limit), or the load generator itself is saturated.

The stack below runs InfluxDB 2.9.1, Grafana 13.2.4 and Prometheus beside the Bookshelf API’s own Compose file (see Compose for Local Development). JMeter runs on the host, as in the rest of this section; Apache publishes no official JMeter image. Every port is bound to 127.0.0.1, so nothing is reachable from the network, and all credentials are random local values generated by a setup script, never typed into a file under version control.

The directory layout:

1. Generate local credentials.

The admin token is used for brevity on a local machine; on a shared InfluxDB create a token with write access to the jmeter bucket only for JMeter, and a read token for Grafana.

2. The Compose file.

Source: Install InfluxDB with Docker Compose (InfluxDB 2), influxdb (Docker Official Image) and Configure a Grafana Docker image (Grafana documentation)

The Prometheus tag pins the release current on the date above; running and configuring the server is covered in Installing & running Prometheus. The DOCKER_INFLUXDB_INIT_* variables only act on the first start, with empty volumes: they create the user, the bookshelf organisation, the jmeter bucket and the admin token. Delete the volumes (docker compose -f compose.monitoring.yaml down -v) to start over.

3. Prometheus scrapes Bookshelf. The service name and port must match the Bookshelf Compose file; here the API service is assumed to be called bookshelf-api and to listen on 8080 inside the Compose network. The 5-second interval matches the JMeter send interval.

Source: Configuration (Prometheus documentation)

4. Grafana data sources. Two InfluxDB data sources (InfluxQL for the community dashboard and the queries above, Flux for ad-hoc exploration) and Prometheus. Grafana replaces $INFLUXDB_TOKEN with the container’s environment variable when it reads the file. InfluxQL works on InfluxDB 2.9 without extra setup, because a bucket gets a virtual DBRP mapping with its own name, so the database is jmeter.

Source: Configure the InfluxDB data source (Grafana documentation) and Provision data sources (Grafana documentation)

5. Run it. The monitoring stack has to share a network with Bookshelf for Prometheus to reach bookshelf-api. The include element at the top of compose.monitoring.yaml (Docker Compose 2.20 or later) pulls the Bookshelf file into the same project and network, and, unlike passing both files with two -f options (where every relative path is resolved from the first file’s directory), keeps each file’s relative paths working:

Source: Include (Docker Compose documentation)

Then run the load test from the host with the generated monitoring.properties, exactly as in Adding it to the Bookshelf plan, and open http://localhost:3000 (user admin, password in secrets/grafana-admin-password). Import dashboard 5496 or build the panels above, add the Prometheus panels, and set the time range to "Last 15 minutes" with a 5 s refresh.

Source: CLI Mode (User Manual)

The run feeds both outputs: the live dashboard while it runs, and the HTML report from the JTL afterwards. Keep the monitoring stack off the load generator for real tests. On one laptop, InfluxDB, Grafana, Prometheus, Bookshelf and JMeter compete for the same CPUs, which is fine to learn the tooling and wrong for measuring Bookshelf.

In a distributed test every JMeter server runs the whole plan, Backend Listener included, and each server’s client writes its own aggregates. Two consequences:

Distributed testing itself is covered on a later page of this section.

Source: Functions (User Manual) and Handle duplicate data points (InfluxDB 2)