CI/CD with Maven, Docker & GitHub Actions

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

A load test that only runs when someone remembers to start it finds regressions weeks after they were merged. This page puts the Bookshelf plan into a pipeline: a short performance smoke test on every pull request, run through Maven, and a longer load test on a schedule, run from a Docker image, both against a Bookshelf API started with Docker Compose on the CI runner, both uploading the results file and the HTML dashboard, and both failing the job when the results break a threshold.

Versions this page was written against, re-checked on 2026-10-14:

  • Apache JMeter 5.6.3, the latest release (Java 8 or later, Java 17 or later recommended; the examples use Java 21). JMeter 6.0 is not released; the changes planned for it on the apache/jmeter master branch are described as planned, not as released behaviour.

  • jmeter-maven-plugin 3.8.0 (com.lazerycode.jmeter:jmeter-maven-plugin), the latest release (July 2023). Its default JMeter version is 5.6.2, so every example sets 5.6.3 explicitly.

  • JMeter Plugins Manager 2.0 with cmdrunner 2.3, the Custom Thread Groups (jpgc-casutg 3.1.1), Throughput Shaping Timer (jpgc-tst 2.6) and Auto-Stop Listener (jpgc-autostop 0.3) plugins, and the gRPC Request plugin 2.0.2 (jmeter-grpc-request).

  • JMeter Java DSL 2.3 (Abstracta, third-party) for the Gradle option.

  • GitHub Actions actions/checkout@v7.0.1, actions/setup-java@v6.0.1 and actions/upload-artifact@v7.0.2, on the ubuntu-24.04 runner image (which ships Docker Compose 2.38 and jq 1.7).

The Maven output, the exit codes and the Auto-Stop behaviour shown below were measured with these versions on Java 21.

Two things the official projects do not provide, so the page builds them: there is no official Apache JMeter Docker image, and there is no official Gradle plugin. The JMeter user manual describes only CLI mode for automation; the Maven plugin, the community Docker images and the Gradle plugins are all third-party projects.

This page builds on Running Tests in CLI Mode (the command line, the profile properties and the measured exit codes), HTML Dashboard Report and Analyzing Results & Finding Bottlenecks (what to gate on), and Plugins & Plugins Manager (the plugin installation the Dockerfile and the POM reuse). The Spring-Boot-specific wiring of the same Maven plugin, with Actuator and Prometheus correlation, is on Performance Testing with JMeter in the Spring Boot reference; it is linked rather than repeated here. Image builds in GitHub Actions in general are covered by Docker CI/CD with GitHub Actions, and the Bookshelf stack itself by Compose for Local Development.

Where load tests fit in a pipeline

A full load test of the Bookshelf journey takes 11 minutes for the load profile and hours for a soak, needs a stable environment and produces numbers that only mean something against a baseline. A pull request needs an answer in a few minutes. The two needs are met by two kinds of run:

Aspect Performance smoke test Scheduled load test

When

Every pull request (and every push to the main branch)

Nightly or weekly on a schedule, before a release, and on demand

Profile

smoke: 1 thread, 60 s

load: 50 threads, 60 s ramp-up, 10-minute hold; soak and stress less often

Question it answers

Does the journey still work end to end, and did something become grossly slower or start failing?

Does the build still meet its response-time and error criteria at the expected load?

Thresholds

Generous: no errors (or under 1 %), p95 far below the timeout

The acceptance criteria: per-transaction p95 and error rate, compared with the baseline

Where it runs

On the CI runner, next to the system under test

Ideally a dedicated performance environment; on a CI runner only as a trend signal

Both ideas come from the books. Pro Apache JMeter (ch. 2) describes a performance smoke test, a few common and essential use cases plus those touched by the change, run as part of the build, with the full suites run only when the smoke test passes and scheduled for when they are needed (its example is a scalability suite run before a busy season); it also argues for a dedicated performance environment so that performance tests can be part of continuous integration. The JMeter Cookbook (ch. 8) runs a JMeter test on every check-in from a Jenkins job, with the Jenkins Performance plugin marking the build unstable or failed on error-percentage thresholds. Jenkins and its plugin are not covered here; the same ideas map onto GitHub Actions jobs and the jmeter-maven-plugin’s results goal.

The pipeline built on this page:

flowchart TD PR["Pull request"] --> UP1["Start Bookshelf with Docker Compose"] UP1 --> SMOKE["mvn verify, smoke profile: 1 thread, 60 s"] SMOKE --> RES{"results goal: error rate within threshold?"} RES -- yes --> OK1["Job passes"] RES -- no --> KO1["Job fails"] OK1 --> ART1["Upload results CSV, dashboard and log"] KO1 --> ART1 NIGHT["Nightly schedule"] --> IMG["Build JMeter image: official binary and pinned plugins"] IMG --> UP2["Start Bookshelf with Docker Compose"] UP2 --> LOAD["jmeter -n in the container: 50 threads, 60 s ramp-up, 10 min hold"] LOAD --> AUTO{"Auto-Stop: errors above 5 % for 30 s?"} AUTO -- yes --> EARLY["Test stopped early, exit code still 0"] AUTO -- no --> FULL["Run completes"] EARLY --> GATE{"perf-gate.sh: samples, error rate, p95"} FULL --> GATE GATE -- pass --> OK2["Job passes"] GATE -- fail --> KO2["Job fails"] OK2 --> ART2["Upload JTL, dashboard and log"] KO2 --> ART2

The diagram has one rule that the whole page follows: the job’s verdict never comes from JMeter’s exit code. JMeter 5.6.3 exits with 0 when every sample failed and when a test was stopped early (see Exit codes), so the verdict comes from the results: the Maven plugin’s results goal, a script that reads statistics.json and the JTL, or both.

Three ways to run JMeter in CI

Route What runs JMeter Strengths Weaknesses

jmeter-maven-plugin

Maven downloads the JMeter artifacts from Maven Central, assembles an installation under target and forks it in CLI mode

One mvn verify; JMeter version and plugins pinned in the POM; built-in error-rate gate and dashboard; works on any runner with a JDK

Plugins must be Maven artifacts; first run downloads JMeter; third-party plugin, last released in 2023

Docker image with the official binary

jmeter -n inside a container built from the Apache binary archive

Exactly the distribution the manual describes; same image on a laptop, in CI and on load generators; plugins installed with PluginsManagerCMD

The image must be built and maintained; thresholds need a script

JMeter Java DSL (Gradle or Maven test)

JMeter engine embedded in a JUnit test

Plans as code reviewed like code; assertions in Java; works from Gradle

Not a .jmx plan; third-party project; covered on the programmatic test plans page

The rest of the page builds the first two in full and shows the third for Gradle users.

The jmeter-maven-plugin

The jmeter-maven-plugin (com.lazerycode.jmeter, Apache License 2.0) is a community project hosted at github.com/jmeter-maven-plugin. It is not part of Apache JMeter. The books used version 1.10 (JMeter Cookbook ch. 8, Pro Apache JMeter ch. 7), with a single jmeter goal bound to verify; version 2.x split the work into Maven phases and 3.x made the JMeter version configurable. Only the 3.x behaviour is described here.

Project layout

The Bookshelf performance tests live in their own Maven module, perf, so that a normal mvn verify of the application does not start a load test. The layout below also holds the Docker route used later on the page:

bookshelf/
├── compose.yaml                   # Bookshelf API (service bookshelf-api) and its database
├── compose.perf.yaml              # adds the JMeter container (Docker route)
├── .github/workflows/performance.yml
└── perf/
    ├── pom.xml                    # jmeter-maven-plugin (Maven route)
    ├── src/test/jmeter/           # plans and their data: the plugin's default test folder
    │   ├── bookshelf-load.jmx
    │   ├── bookshelf-grpc.jmx
    │   ├── users.csv              # placeholder credentials only
    │   └── search-terms.csv
    ├── profiles/                  # -q property files for the Docker route
    │   ├── smoke.properties
    │   └── load.properties
    ├── docker/Dockerfile          # JMeter image (Docker route)
    ├── perf-gate.sh               # threshold check on statistics.json and the JTL
    └── results/                   # written by the Docker route; in .gitignore

What the plugin does with src/test/jmeter, from its wiki:

  • Plans. Every .jmx file in the folder and its sub-folders is copied to target/jmeter/testFiles and run. Files in sub-folders are flattened, and their path becomes part of the test file name.

  • Data files. The other files and folders are copied as well, so users.csv and search-terms.csv end up next to the plan. A CSV Data Set Config with a relative file name resolves it against the plan’s folder, so the same relative name works in the GUI, in the plain CLI and under Maven.

  • Properties files. jmeter.properties, saveservice.properties, upgrade.properties, system.properties, user.properties and global.properties found in the folder are used instead of JMeter’s own. Do not name a profile file user.properties by accident.

  • Other configuration such as log4j2.xml goes in src/test/conf.

Source: Basic Configuration (jmeter-maven-plugin wiki), CSV Data Set Config (Component Reference)

Goals and phases

Goal Default phase What it does

configure

compile

Downloads JMeter and the configured libraries and assembles a JMeter installation in a unique (UUID-named) folder under target. The configuration is stored under the execution id, configuration by default.

jmeter

integration-test

Runs every selected plan with that installation in a separate JVM, in CLI mode.

results

verify

Scans the results files and fails the build if they contain failures (or more than the accepted error rate).

gui

none (run from the command line)

Starts the JMeter GUI on the configured installation, to edit a plan with exactly the plugins the build uses.

remote-server

integration-test

Starts JMeter in server mode. It appears in the 3.8.0 plugin descriptor but not in the wiki, which documents distributed runs only through <remoteConfig>; distributed testing is covered on its own page of this section.

The jmeter and results goals find the installation through <selectedConfiguration>, which defaults to configuration. If the configure execution gets another id, the other executions must name it. Configuration placed inside the jmeter execution is ignored: it belongs in the plugin’s top-level <configuration> or in the configure execution, as the wiki’s FAQ explains.

Useful command lines, all from the wiki:

mvn clean verify                               # configure, run every plan, check the results
mvn jmeter:jmeter                              # run again on the existing installation, no reconfiguration
mvn jmeter:jmeter -DselectConfiguration=configuration
mvn jmeter:configure jmeter:gui -DguiTestFile=src/test/jmeter/bookshelf-load.jmx   # open a plan in the GUI

Source: Basic Configuration (jmeter-maven-plugin wiki), FAQ (jmeter-maven-plugin wiki)

The Bookshelf POM

The complete perf/pom.xml. The defaults in <properties> are the smoke profile; the perf-load Maven profile switches to the load profile. The plan reads threads, rampup, duration, host and port with ${__P(...)}, exactly as in the CLI examples of this section.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.example.bookshelf</groupId>
  <artifactId>bookshelf-perf</artifactId>
  <version>1.0.0-SNAPSHOT</version>
  <packaging>pom</packaging>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <!-- smoke profile by default -->
    <perf.threads>1</perf.threads>
    <perf.rampup>1</perf.rampup>
    <perf.duration>60</perf.duration>
    <perf.host>localhost</perf.host>
    <perf.port>8080</perf.port>
    <perf.maxErrorPct>1</perf.maxErrorPct>
  </properties>

  <profiles>
    <profile>
      <id>perf-load</id>
      <properties>
        <perf.threads>50</perf.threads>
        <perf.rampup>60</perf.rampup>
        <perf.duration>660</perf.duration>
      </properties>
    </profile>
  </profiles>

  <build>
    <plugins>
      <plugin>
        <groupId>com.lazerycode.jmeter</groupId>
        <artifactId>jmeter-maven-plugin</artifactId>
        <version>3.8.0</version>
        <executions>
          <!-- Generate JMeter configuration -->
          <execution>
            <id>configuration</id>
            <goals>
              <goal>configure</goal>
            </goals>
          </execution>
          <!-- Run JMeter tests -->
          <execution>
            <id>jmeter-tests</id>
            <goals>
              <goal>jmeter</goal>
            </goals>
          </execution>
          <!-- Fail build on errors in test -->
          <execution>
            <id>jmeter-check-results</id>
            <goals>
              <goal>results</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <jmeterVersion>5.6.3</jmeterVersion>

          <!-- plugins, copied into lib/ext -->
          <jmeterExtensions>
            <artifact>kg.apc:jmeter-plugins-casutg:3.1.1</artifact>
            <artifact>kg.apc:jmeter-plugins-tst:2.6</artifact>
            <artifact>kg.apc:jmeter-plugins-autostop:0.3</artifact>
            <artifact>kg.apc:jmeter-plugins-cmn-jmeter:0.7</artifact>
          </jmeterExtensions>
          <downloadExtensionDependencies>false</downloadExtensionDependencies>

          <!-- plain libraries such as JDBC drivers, copied into lib -->
          <testPlanLibraries>
            <artifact>org.postgresql:postgresql:42.7.14</artifact>
          </testPlanLibraries>

          <!-- run only the HTTP plan here; the gRPC plan has its own job -->
          <testFilesIncluded>
            <jMeterTestFile>bookshelf-load.jmx</jMeterTestFile>
          </testFilesIncluded>

          <!-- merged into user.properties, read in the plan with __P -->
          <propertiesUser>
            <threads>${perf.threads}</threads>
            <rampup>${perf.rampup}</rampup>
            <duration>${perf.duration}</duration>
            <host>${perf.host}</host>
            <port>${perf.port}</port>
          </propertiesUser>

          <!-- results and dashboard -->
          <testResultsTimestamp>false</testResultsTimestamp>
          <generateReports>true</generateReports>

          <!-- gate: fail only above this error percentage -->
          <errorRateThresholdInPercent>${perf.maxErrorPct}</errorRateThresholdInPercent>

          <!-- the forked JMeter JVM -->
          <jMeterProcessJVMSettings>
            <xms>1024</xms>
            <xmx>1024</xmx>
          </jMeterProcessJVMSettings>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

Run it against a Bookshelf API on localhost:8080:

cd perf
mvn -B clean verify                    # smoke profile
mvn -B clean verify -Pperf-load        # load profile
mvn -B clean verify -Dperf.port=18080  # any property can be overridden with -D

The interesting part of the output of a passing smoke run, with this exact POM (paths shortened, the four Log4j "package scanning" warnings that every 5.6.3 start prints removed; a machine with a German or Spanish locale prints 0,3/s instead of 0.3/s):

[INFO] --- jmeter:3.8.0:configure (configuration) @ bookshelf-perf ---
...
[INFO] --- jmeter:3.8.0:jmeter (jmeter-tests) @ bookshelf-perf ---
[INFO] -------------------------------------------------------
[INFO]  P E R F O R M A N C E    T E S T S
[INFO] -------------------------------------------------------
[INFO] Will generate HTML report in .../perf/target/jmeter/reports/bookshelf-load
[INFO] Executing test: bookshelf-load.jmx
[INFO] Arguments for forked JMeter JVM: [java, -Xms1024M, -Xmx1024M, -Djava.awt.headless=true, -jar, ApacheJMeter-5.6.3.jar, -d, .../target/<uuid>/jmeter, -e, -j, .../target/jmeter/logs/bookshelf-load.jmx.log, -l, .../target/jmeter/results/bookshelf-load.csv, -n, -o, .../target/jmeter/reports/bookshelf-load, -t, .../target/jmeter/testFiles/bookshelf-load.jmx, -Dsun.net.http.allowRestrictedHeaders, true]
[INFO] Creating summariser <summary>
[INFO] Starting standalone test @ 2026 Oct 14 13:54:46 CEST (1791978886357)
[INFO] Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
[INFO] summary =     15 in 00:00:45 =    0.3/s Avg:     6 Min:     4 Max:    18 Err:     0 (0.00%)
[INFO] summary =     19 in 00:00:57 =    0.3/s Avg:     6 Min:     4 Max:    18 Err:     0 (0.00%)
[INFO] Tidying up ...    @ 2026 Oct 14 13:55:43 CEST (1791978943729)
[INFO] ... end of run
[INFO] --- jmeter:3.8.0:results (jmeter-check-results) @ bookshelf-perf ---
[INFO] -------------------------------------------------------
[INFO] S C A N N I N G    F O R    R E S U L T S
[INFO] -------------------------------------------------------
[INFO] Will scan results using format: CSV
[INFO] -------------------------------------------------------
[INFO] P E R F O R M A N C E    T E S T    R E S U L T S
[INFO] -------------------------------------------------------
[INFO] Result (.csv) files scanned: 1
[INFO] Successful requests:         19
[INFO] Failed requests:             0
[INFO] Failures:                    0.0% (1.0% accepted)
[INFO] BUILD SUCCESS

The forked command line is the plain CLI run of Running Tests in CLI Mode: -n -t -l -j -e -o, plus -d for the generated JMeter home. The results file is CSV (named .csv, not .jtl; the content is the same CSV JTL format), the dashboard folder is named after the plan, and the log goes to target/jmeter/logs.

Worked numbers. The smoke profile is one thread for 60 s. With the Bookshelf think time of 3 s and a response time of about 0.2 s, one thread completes a request every 3.2 s, so the run produces about 60 / 3.2 ≈ 19 samples (the run above, against a local stand-in answering in a few milliseconds, produced exactly 19): enough to prove the journey works, far too few for a meaningful percentile. The perf-load profile gives about 15.6 requests per second once 50 threads are running, roughly 9,400 samples in the 600 s hold plus about 470 during the ramp-up, which is the volume the dashboard needs for stable p95 values.

Source: Basic Configuration (jmeter-maven-plugin wiki), Modifying Properties (jmeter-maven-plugin wiki), Test Results (jmeter-maven-plugin wiki)

Configuration reference

The elements below are the ones the plugin’s wiki documents. Defaults come from the wiki where it states them, and otherwise from the plugin descriptor of 3.8.0 (marked "descriptor").

Element Default Meaning

JMeter and libraries

<jmeterVersion>

5.6.2 (descriptor)

JMeter version to assemble. Also settable with -Djmeter.version=.... The wiki notes that JMeter 5.2 cannot be supported because of problems in its POM files.

<jmeterArtifacts>

derived from the version

Explicit list of org.apache.jmeter artifacts, for versions whose module list differs.

<jmeterExtensions>

empty

groupId:artifactId:version artifacts copied into lib/ext (plugins).

<downloadExtensionDependencies>

true (descriptor)

Also resolve and copy the extensions' transitive dependencies. The wiki’s own example sets it to false because some plugins have broken dependencies.

<testPlanLibraries>

empty

Artifacts copied into lib (JDBC drivers, JMS clients, utility jars).

<junitLibraries>

empty

Artifacts copied into lib/junit, for the JUnit Request sampler.

<excludedArtifacts>

empty

groupId:artifactId entries to drop during resolution, to fix jar conflicts.

Plans

<testFilesDirectory>

src/test/jmeter

Where the plans are read from.

<testFilesIncluded> / <jMeterTestFile>

all .jmx files

Plans to run, by name or wildcard pattern (foo*.jmx).

<testFilesExcluded> / <excludeJMeterTestFile>

none

Plans to skip.

<skipTests>

${skipTests}

Skip the performance tests; with <skipTests>${skipTests}</skipTests>, mvn verify -DskipTests=true skips them.

Properties

<propertiesUser>

empty

Merged into user.properties: the values the plan reads with __P.

<propertiesJMeter>

empty

Merged into jmeter.properties (for example log_level.jmeter).

<propertiesGlobal>

empty

Merged into global.properties, the properties sent to remote servers.

<propertiesSystem>

empty

Merged into system.properties (JVM system properties).

<propertiesSaveService>, <propertiesUpgrade>

empty

Merged into saveservice.properties and upgrade.properties.

<propertiesReplacedByCustomFiles>

false

Replace the properties files instead of merging into them. The wiki advises against it.

<customPropertiesFiles> / <file>

none

Extra properties files, the equivalent of --addprop (-q).

<propertiesFilesDirectory>

src/test/jmeter

Where the replacement .properties files are looked for.

Results and reports

<resultsDirectory>

target/jmeter/results

Where the results files are written.

<resultsFileFormat>

csv (since 2.6.0)

csv or xml. Forced to CSV when reports are generated.

<testResultsTimestamp>

true

Prefix each results file name (and dashboard folder) with a timestamp.

<appendResultsTimestamp>

false

Put the timestamp at the end of the name instead.

<resultsFileNameDateFormat>

basic ISO 8601 date

Joda-Time pattern for the timestamp.

<generateReports>

false

Generate the HTML dashboard (-e -o) after each plan.

<reportDirectory>

target/jmeter/reports

Parent folder of the dashboards.

<suppressJMeterOutput>

false

Do not copy JMeter’s console output into the Maven log.

Failure handling

<ignoreResultFailures>

false

When true, the results goal reports failures but does not fail the build.

<errorRateThresholdInPercent>

0 (descriptor)

Fail only when the percentage of failed samples is above this value (since 2.8.1). With the default, one failed sample fails the build.

Process

<jMeterProcessJVMSettings>

-Xms512M -Xmx512M (measured)

<xms>, <xmx> in MB and <arguments> for the forked JMeter JVM. The wiki suggests equal xms and xmx.

<selectedConfiguration>

configuration

Execution id of the configure run that jmeter and results use (-DselectConfiguration).

<proxyConfig>, <remoteConfig>

none

Outbound proxy and distributed-run settings (wiki pages Proxy Configuration and Remote Server Configuration).

Source: Advanced Configuration (jmeter-maven-plugin wiki) and the pages it links

<jmeterVersion>: pin it, and clean after changing it

Without <jmeterVersion>, plugin 3.8.0 assembles JMeter 5.6.2, not the current 5.6.3. The Spring Boot reference’s example (Performance Testing with JMeter) uses the same plugin version, 3.8.0, and also pins <jmeterVersion>5.6.3</jmeterVersion>, so both pages run the same JMeter. JMeter 5.6.3 is a bug-fix release over 5.6.2, so plans move between them unchanged.

The wiki’s caveat is that the version switch assumes the list of JMeter modules has not changed. That will matter for 6.0: the master branch removes the MongoDB elements (the ApacheJMeter_mongodb module appears in the wiki’s artifact list) and requires Java 17. Expect a new plugin release, or an explicit <jmeterArtifacts> list, before pointing <jmeterVersion> at 6.0.

After changing the plugin version or <jmeterVersion>, run mvn clean verify. The plugin reuses target, and the wiki’s FAQ shows the errors that a mix of jars from two JMeter versions produces.

Source: Specifying JMeter Version (jmeter-maven-plugin wiki), FAQ (jmeter-maven-plugin wiki)

<jmeterExtensions>: plugins and the gRPC sampler

The plugin does not use the Plugins Manager: every plugin jar is a Maven artifact listed in <jmeterExtensions>. The POM above lists the Custom Thread Groups, the Throughput Shaping Timer and the Auto-Stop Listener, plus jmeter-plugins-cmn-jmeter, the library those kg.apc plugins declare in the Plugins Manager catalogue, with <downloadExtensionDependencies> set to false so that the old ApacheJMeter_core versions in the plugins' POMs are not copied next to 5.6.3. Why that is necessary, and how to read the catalogue to find each plugin’s libraries, is explained in Plugins & Plugins Manager: With the jmeter-maven-plugin.

Not every plugin is on Maven Central. The gRPC Request plugin 2.0.x (jmeter-grpc-request) and the BlazeMeter HTTP plugin (bzm-http2) 3.x are published only as GitHub release assets. Install the jar into an internal repository (or the local one with mvn install:install-file) and list those coordinates, as shown on gRPC Load Testing. On a CI runner the local repository is empty, so a job that relies on install:install-file must run it before verify, from a jar stored in the repository or downloaded with a checksum check. An internal repository manager is the cleaner option for a team.

A separate job for the gRPC plan, in the same module, selects the plan and adds the extension:

<profile>
  <id>perf-grpc</id>
  <build>
    <plugins>
      <plugin>
        <groupId>com.lazerycode.jmeter</groupId>
        <artifactId>jmeter-maven-plugin</artifactId>
        <configuration>
          <jmeterExtensions combine.children="append">
            <!-- installed from the GitHub release jar into the internal repository -->
            <artifact>vn.zalopay.benchmark:jmeter-grpc-request:2.0.2</artifact>
          </jmeterExtensions>
          <testFilesIncluded combine.self="override">
            <jMeterTestFile>bookshelf-grpc.jmx</jMeterTestFile>
          </testFilesIncluded>
        </configuration>
      </plugin>
    </plugins>
  </build>
</profile>

combine.children and combine.self are standard Maven POM merge attributes, not plugin features: the first appends the gRPC jar to the extensions declared in the main build, the second replaces the list of plans.

Source: Adding/Excluding libraries to/from the classpath (jmeter-maven-plugin wiki), POM Reference: Plugins (Apache Maven)

<testPlanLibraries>: JDBC drivers

Plain libraries go into lib, not lib/ext. The JDBC examples of this section use the PostgreSQL driver; with the plugin it is one <artifact> element, resolved from Maven Central with its dependencies:

<testPlanLibraries>
  <artifact>org.postgresql:postgresql:42.7.14</artifact>
</testPlanLibraries>

The connection settings stay run properties (db_host, db_port, db_name, db_user, db_password), passed through <propertiesUser> from Maven properties; in CI the password comes from a secret mapped to an environment variable (-Dperf.dbPassword="$DB_PASSWORD"), never from the POM. The JDBC elements themselves are covered on Database Testing with JDBC.

A library that is not in any Maven repository can be put in src/test/jmeter/lib and added on the Test Plan’s Add directory or jar to classpath field with a path relative to the generated installation; the wiki gives the ../../../jmeter/testFiles/lib/... form. An internal repository is simpler to keep correct.

Source: Adding/Excluding libraries to/from the classpath (jmeter-maven-plugin wiki)

<propertiesUser> and load profiles

<propertiesUser> is merged into the generated user.properties, so every entry is a JMeter property and the plan reads it exactly as it reads -Jthreads=50 on the command line. That gives a three-level chain that keeps one plan and many profiles:

Level Example Changed by

Plan default

${__P(threads,1)} in the Thread Group

Editing the plan (rarely)

Maven property

<threads>${perf.threads}</threads> with perf.threads set by a POM profile

-Pperf-load

Command line

-Dperf.threads=20

The CI job or a developer, for one run

Because the values are merged into a properties file, they are properties, not variables: read them with ${__P(name,default)}, not with ${name}. A __P call without a default returns 1 when the property is not set, which is harmless for a thread count but wrong for a host name, so give every __P an explicit default. The section’s property names are threads, rampup, duration, host and port; the Spring Boot reference page’s own example uses rampUp and loops for its simpler loop-count plan, so keep each POM consistent with the plan it drives.

For values that should reach remote servers in a distributed run, use <propertiesGlobal>, the equivalent of -G.

Source: Modifying Properties (jmeter-maven-plugin wiki), __P (Functions)

Failing the build: the results goal

The results goal reads every results file of the run and counts successful and failed samples. By default one failed sample fails the build. Two settings change that, both documented in the wiki:

  • <errorRateThresholdInPercent>: fail only when the failure percentage is above the value. A float; 1 means more than 1 % failed samples fails the build.

  • <ignoreResultFailures>true</ignoreResultFailures>: report, never fail. Useful for an exploratory job whose verdict comes from a later script.

Output of a 10-second run (-Dperf.duration=10) with the Bookshelf API down, so every request gets "connection refused":

[INFO] summary =      3 in 00:00:09 =    0.3/s Avg:     6 Min:     2 Max:    14 Err:     3 (100.00%)
...
[INFO] P E R F O R M A N C E    T E S T    R E S U L T S
[INFO] -------------------------------------------------------
[INFO] Result (.csv) files scanned: 1
[INFO] Successful requests:         0
[INFO] Failed requests:             3
[INFO] Failures:                    100.0% (1.0% accepted)
[INFO] BUILD FAILURE
[ERROR] Failed to execute goal com.lazerycode.jmeter:jmeter-maven-plugin:3.8.0:results (jmeter-check-results) on project bookshelf-perf: Failing build because error percentage .../target/jmeter/logs is above accepted threshold 100.0. JMeter logs are available at: '1.0' -> [Help 1]

Maven exits with 1, so the CI step fails. The error line in 3.8.0 has its arguments swapped (it prints the log folder where the percentage belongs and the threshold where the log folder belongs); the summary lines above it are the ones to read.

What the results goal does not check, and how to cover it:

Not checked Cover it with

Response times and percentiles

Duration Assertions in the plan (a slow sample becomes a failed sample, see Assertions), or a script on statistics.json (below)

Per-transaction error rates

A script on statistics.json, which has one entry per label

A run that produced no samples

The JMeter log and a sample-count check in the script; with zero samples there is nothing to fail

Whether the load was actually applied

Compare sampleCount and throughput with the expected numbers from the profile

A JMeter start-up failure is caught one step earlier: when the forked JMeter exits with a non-zero code (missing or unparsable plan, non-empty dashboard folder), the jmeter goal itself fails with Test failed with exit code:1.

Source: Test Results: ignoreResultFailures and errorRateThresholdInPercent (jmeter-maven-plugin wiki)

<generateReports>: the dashboard

With <generateReports>true</generateReports> the plugin adds -e -o to the JMeter command line and the dashboard of each plan is written to target/jmeter/reports/<plan name> (or under <reportDirectory>):

target/jmeter/
├── logs/bookshelf-load.jmx.log
├── reports/bookshelf-load/
│   ├── index.html
│   ├── statistics.json
│   ├── content/
│   └── sbadmin2-1.0.7/
├── results/bookshelf-load.csv
└── testFiles/bookshelf-load.jmx

Two rules from the wiki and from measurement:

  • Reports force CSV results. The dashboard needs a CSV results file, so enabling reports switches <resultsFileFormat> to CSV even if it is set to xml.

  • The dashboard folder must be empty. JMeter refuses to write a dashboard into a non-empty folder. With <testResultsTimestamp>false</testResultsTimestamp>, a second mvn verify without clean fails in the jmeter goal with An error occurred: Cannot write to '.../reports/bookshelf-load' as folder is not empty. Always run mvn clean verify (CI does, because each job starts from a fresh checkout), or keep the default timestamps, which give every run its own folder.

The dashboard’s properties (APDEX thresholds, percentiles, granularity) are JMeter properties; set them with <propertiesUser> or in a user.properties in src/test/jmeter. They are described on HTML Dashboard Report.

Source: Generating Reports (jmeter-maven-plugin wiki), Generating Report Dashboard (User Manual)

The forked JVM

The plugin runs JMeter in its own JVM; the forked command line starts with java, which in CI is the JDK that actions/setup-java installs and puts on the PATH. Without <jMeterProcessJVMSettings> the measured command line uses -Xms512M -Xmx512M, half of JMeter’s own bin/jmeter default of 1 GB; the POM above restores 1 GB. For the heap a given load needs, see the sizing section of Running Tests in CLI Mode. The HEAP and JVM_ARGS environment variables of the bin/jmeter script do not apply here, because the plugin starts ApacheJMeter-5.6.3.jar directly.

Source: Configuring the JVM that the JMeter process runs in (jmeter-maven-plugin wiki)

Gradle: no official plugin

Neither Apache JMeter nor the jmeter-maven-plugin project publishes a Gradle plugin. The Gradle Plugin Portal lists community plugins that wrap JMeter; they are maintained by individuals, several have not followed recent JMeter releases, and none is endorsed by the Apache project. Check the JMeter version a plugin assembles and its last release date before adopting one. Two options avoid the question:

1. Call the CLI from a Gradle task. With JMeter installed on the runner (or in the Docker image below), an Exec task is enough and keeps the plan, the properties and the thresholds identical to the Docker route:

// build.gradle.kts
tasks.register<Exec>("perfSmoke") {
    group = "verification"
    description = "Runs the Bookshelf smoke profile in JMeter CLI mode"
    workingDir = file("perf")
    commandLine(
        "jmeter", "-n", "-t", "src/test/jmeter/bookshelf-load.jmx",
        "-q", "profiles/smoke.properties",
        "-l", "results/smoke.jtl", "-j", "results/smoke.log", "-f",
        "-e", "-o", "results/smoke-report"
    )
}

tasks.register<Exec>("perfGate") {
    group = "verification"
    dependsOn("perfSmoke")
    workingDir = file("perf")
    commandLine("./perf-gate.sh", "results/smoke.jtl", "results/smoke-report")
}

Exec fails the task when the command exits with a non-zero code, which is why the gate is a separate task: JMeter itself returns 0 even for a run where everything failed.

Source: Exec (Gradle DSL Reference), Getting Started: CLI Mode (User Manual)

2. Write the test with the JMeter Java DSL. The DSL (Abstracta, Apache License 2.0, not an Apache project) embeds the JMeter engine in a JUnit test, so Gradle’s test task runs it and JUnit assertions are the thresholds. Its setup guide gives the Gradle dependency with an exclusion of the org.apache.jmeter:bom module:

// build.gradle.kts
dependencies {
    testImplementation("us.abstracta.jmeter:jmeter-java-dsl:2.3") {
        exclude("org.apache.jmeter", "bom")
    }
    testImplementation("org.junit.jupiter:junit-jupiter:5.13.4")
    testImplementation("org.assertj:assertj-core:3.27.6")
    testRuntimeOnly("org.junit.platform:junit-platform-launcher")
}

tasks.test {
    useJUnitPlatform()
}
import static org.assertj.core.api.Assertions.assertThat;
import static us.abstracta.jmeter.javadsl.JmeterDsl.*;

import java.io.IOException;
import java.time.Duration;
import org.junit.jupiter.api.Test;
import us.abstracta.jmeter.javadsl.core.TestPlanStats;

class BookshelfSmokeTest {

  @Test
  void searchStaysFastAndCorrect() throws IOException {
    TestPlanStats stats = testPlan(
        threadGroup(1, 10,
            httpSampler("http://localhost:8080/api/books?query=java")),
        jtlWriter("build/jtls")
    ).run();

    assertThat(stats.overall().samplesCount()).isEqualTo(10);
    assertThat(stats.overall().errorsCount()).isZero();
    assertThat(stats.overall().sampleTimePercentile99()).isLessThan(Duration.ofMillis(800));
  }
}

The DSL plan is Java, not a .jmx, so it does not reuse the Bookshelf plan; it suits teams that prefer plans as code. Its capabilities and limits (including the absence of gRPC support) are compared on the programmatic test plans page of this section. The JUnit and AssertJ versions are examples; use the ones the project already has.

Source: JMeter Java DSL User Guide, abstracta/jmeter-java-dsl README (GitHub)

Docker

No official image

The Apache JMeter project publishes no Docker image. Docker Hub has community images such as justb4/jmeter and alpine/jmeter; they are third-party, their JMeter and Java versions and update cadence vary, and they come without the plugins a plan needs. The approach used in this section is a small Dockerfile of your own: the official binary archive, checked against its published SHA-512, plus the Plugins Manager’s command-line tool to install pinned plugin versions. It combines the image from Installation & Configuration with the plugin installation from Plugins & Plugins Manager.

The Dockerfile

# perf/docker/Dockerfile: Apache JMeter 5.6.3 from the official binary, plugins pinned
FROM eclipse-temurin:21-jre

ARG JMETER_VERSION=5.6.3
ARG PMGR_VERSION=2.0
ARG CMDRUNNER_VERSION=2.3
ARG JMETER_PLUGINS="jpgc-casutg=3.1.1,jpgc-tst=2.6,jpgc-autostop=0.3,jmeter-grpc-request=2.0.2"
ENV JMETER_HOME=/opt/apache-jmeter-${JMETER_VERSION}
ENV PATH="${JMETER_HOME}/bin:${PATH}"

RUN apt-get update \
 && apt-get install -y --no-install-recommends curl \
 && rm -rf /var/lib/apt/lists/* \
 && curl -fL -o /tmp/jmeter.tgz \
      https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-${JMETER_VERSION}.tgz \
 && curl -fL -o /tmp/jmeter.tgz.sha512 \
      https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-${JMETER_VERSION}.tgz.sha512 \
 && echo "$(cut -d' ' -f1 /tmp/jmeter.tgz.sha512)  /tmp/jmeter.tgz" | sha512sum -c - \
 && tar -xzf /tmp/jmeter.tgz -C /opt \
 && rm /tmp/jmeter.tgz /tmp/jmeter.tgz.sha512 \
 && curl -fL -o ${JMETER_HOME}/lib/ext/jmeter-plugins-manager-${PMGR_VERSION}.jar \
      https://repo1.maven.org/maven2/kg/apc/jmeter-plugins-manager/${PMGR_VERSION}/jmeter-plugins-manager-${PMGR_VERSION}.jar \
 && curl -fL -o ${JMETER_HOME}/lib/cmdrunner-${CMDRUNNER_VERSION}.jar \
      https://repo1.maven.org/maven2/kg/apc/cmdrunner/${CMDRUNNER_VERSION}/cmdrunner-${CMDRUNNER_VERSION}.jar \
 && java -cp ${JMETER_HOME}/lib/ext/jmeter-plugins-manager-${PMGR_VERSION}.jar \
      org.jmeterplugins.repository.PluginManagerCMDInstaller \
 && echo "jpgc.repo.sendstats=false" >> ${JMETER_HOME}/bin/jmeter.properties \
 && PluginsManagerCMD.sh install ${JMETER_PLUGINS} \
 && sleep 5 && PluginsManagerCMD.sh status

WORKDIR /work
ENTRYPOINT ["jmeter"]

What each part does, and why:

Part Purpose

eclipse-temurin:21-jre

A JRE is enough to run JMeter; Java 21 is the version this section uses and the one 6.0 recommends.

archive.apache.org and sha512sum -c

The archive host keeps every release, so the URL keeps working after 5.6.3 stops being the current release (the dlcdn.apache.org mirror network carries only current releases). The build fails if the checksum does not match. The checksum file comes from the same host, which protects against corruption more than against a compromised host; add the PGP signature check from the download page for stronger assurance.

Plugins Manager, cmdrunner, PluginManagerCMDInstaller

Installs the command-line tool PluginsManagerCMD.sh into bin.

jpgc.repo.sendstats=false in jmeter.properties

Turns off the Plugins Manager’s usage statistics; the command-line tool reads only jmeter.properties before it reads this setting, as explained on the plugins page.

install id=version,...

Pins every plugin. Never use a bare id or install-for-jmx in an image: both install the latest version.

sleep 5 and status

Gives the helper process that moves the jars time to finish, and records the installed list in the build log.

WORKDIR /work, ENTRYPOINT ["jmeter"]

The plans are mounted at /work, and the container’s arguments are the JMeter command line.

Build it and check what it contains:

docker build -t bookshelf-jmeter:5.6.3 perf/docker
docker run --rm bookshelf-jmeter:5.6.3 -v

The status line in the build log is the record of the plugin versions, to be kept with each run’s results. From a build of this Dockerfile without the gRPC plugin (with it, jmeter-grpc-request=2.0.2 is listed as well):

[jpgc-autostop=0.3, jpgc-casutg=3.1.1, jpgc-plugins-manager=2.0, jpgc-tst=2.6, jmeter-core=5.6.3, jmeter-ftp=5.6.3, jmeter-http=5.6.3, jmeter-jdbc=5.6.3, ...]

docker run --rm bookshelf-jmeter:5.6.3 -v prints the JMeter banner ending in 5.6.3.

Build time. The first build downloads the 83 MB JMeter archive. archive.apache.org can be slow (a build measured for this page took about 20 minutes, almost all of it for that download), so in CI cache the image layers (the docker/build-push-action cache described on Docker CI/CD with GitHub Actions) or publish the image to a registry once and pull it in the test jobs.

Source: Download Apache JMeter, Plugins Manager from Command-Line (JMeter Plugins wiki)

Running the plan against Bookshelf with Compose

From inside a container, localhost is the container itself, not the Bookshelf API. The simplest way to reach the API is to put JMeter on the same Compose network and address the API by its service name. A second Compose file adds the JMeter service to the Bookshelf project; the API service is assumed to be called bookshelf-api and to listen on 8080 inside the network, as on the real-time monitoring page:

# compose.perf.yaml: used together with the Bookshelf compose.yaml
services:
  jmeter:
    image: bookshelf-jmeter:5.6.3
    build: ./perf/docker
    profiles: ["perf"]              # never started by a plain "docker compose up"
    working_dir: /work
    volumes:
      - ./perf:/work                # plans, profiles and results
    environment:
      HEAP: "-Xms1g -Xmx1g -XX:MaxMetaspaceSize=256m"
    depends_on:
      bookshelf-api:
        condition: service_started
# 1. start Bookshelf and wait until it answers
docker compose up -d --wait
curl -fsS --retry 30 --retry-delay 2 --retry-all-errors http://localhost:8080/actuator/health

# 2. run the smoke profile in the JMeter container on the Compose network
mkdir -p perf/results
docker compose -f compose.yaml -f compose.perf.yaml run --rm jmeter \
  -n -t src/test/jmeter/bookshelf-load.jmx -q profiles/smoke.properties \
  -Jhost=bookshelf-api -Jport=8080 \
  -l results/smoke.jtl -j results/smoke.log -f -e -o results/smoke-report

# 3. check the thresholds (the script is shown in the GitHub Actions section)
perf/perf-gate.sh perf/results/smoke.jtl perf/results/smoke-report

Output of steps 2 and 3 against a local stand-in for the API (container creation lines and the four Log4j warnings removed, summariser lines abridged):

Created the tree successfully using src/test/jmeter/bookshelf-load.jmx
Starting standalone test @ 2026 Oct 14 11:52:18 UTC (1791978738507)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
summary +      1 in 00:00:03 =    0.3/s Avg:    32 Min:    32 Max:    32 Err:     0 (0.00%) Active: 1 Started: 1 Finished: 0
...
summary =     19 in 00:00:57 =    0.3/s Avg:     6 Min:     2 Max:    32 Err:     0 (0.00%)
Tidying up ...    @ 2026 Oct 14 11:53:15 UTC (1791978795925)
... end of run
samples=19 errors=0.0% p95=32 ms
PASS

The profile files are the ones from Running Tests in CLI Mode (smoke: threads=1, rampup=1, duration=60; load: 50, 60 and 660). -Jhost and -Jport on the command line override anything in them.

Details that matter:

  • docker compose run starts a service that has a profile because the service is named explicitly, and it starts the depends_on services if they are not running. service_started only means the container started; the curl loop is what makes sure the API answers before load begins. If the Bookshelf service defines a healthcheck, use condition: service_healthy instead.

  • HEAP is read by the bin/jmeter script, so it works for the container (unlike the Maven route). Size it from the profile, as described on the CLI page.

  • Results land in perf/results on the host. On a Linux host they are owned by root, because the container runs as root. CI runners do not mind; on a Linux workstation, add a user: entry with your numeric user and group ids to the service if root-owned files are a nuisance.

  • -f deletes the previous results file and dashboard folder before the run, so a second run does not fail on a non-empty -o folder.

  • The system under test and the load generator share one machine. On a laptop or a CI runner, JMeter’s CPU use slows the API down and vice versa. Treat the numbers as a regression signal, not as capacity figures.

Source: Compose file reference: services, depends_on (Docker Docs), Using profiles with Compose (Docker Docs), Getting Started: CLI Mode (User Manual)

GitHub Actions

The threshold script

The Docker route needs its own verdict. perf/perf-gate.sh checks three things: that the JTL holds at least one sample (a CSV JTL with only its header line is not empty, but has no data), that the dashboard’s statistics.json exists, and that the overall error rate and 95th percentile are within the limits. It also writes a one-line summary to the job summary page when it runs in GitHub Actions.

#!/usr/bin/env bash
# perf/perf-gate.sh JTL REPORT_DIR [MAX_ERROR_PCT] [MAX_P95_MS]
# Fails (exit 1) when the run has no samples, too many errors or a slow 95th percentile.
set -euo pipefail
jtl="$1"; report="$2"; max_err="${3:-1}"; max_p95="${4:-800}"
stats="$report/statistics.json"

# a JTL with only its CSV header line is "non-empty" but holds no samples
samples=$(($(wc -l < "$jtl") - 1))
if [ "$samples" -le 0 ] || [ ! -s "$stats" ]; then
  echo "FAIL: no samples in $jtl or no $stats"; exit 1
fi

summary=$(jq -r '.Total | "samples=\(.sampleCount) errors=\(.errorPct)% p95=\(.pct2ResTime | floor) ms"' "$stats")
echo "$summary"
[ -n "${GITHUB_STEP_SUMMARY:-}" ] && echo "JMeter: $summary" >> "$GITHUB_STEP_SUMMARY"

jq -e --argjson e "$max_err" --argjson p "$max_p95" \
  '.Total.errorPct <= $e and .Total.pct2ResTime <= $p' "$stats" > /dev/null \
  || { echo "FAIL: thresholds are error rate <= $max_err %, p95 <= $max_p95 ms"; exit 1; }
echo "PASS"

Commit the script with its executable bit (git add --chmod=+x perf/perf-gate.sh), or call it through bash; a checkout keeps the mode stored in Git. Output on a run where the API was unreachable, and on a JTL with no samples:

$ perf/perf-gate.sh perf/results/load.jtl perf/results/load-report
samples=40 errors=100.0% p95=5 ms
FAIL: thresholds are error rate <= 1 %, p95 <= 800 ms
$ echo $?
1
$ perf/perf-gate.sh perf/results/empty.jtl perf/results/load-report
FAIL: no samples in perf/results/empty.jtl or no perf/results/load-report/statistics.json

The run above is also a reminder of why a percentile alone is not a gate: requests refused in 5 ms give an excellent p95. pct2ResTime is the 95th percentile because jmeter.reportgenerator.aggregate_rpt_pct2 defaults to 95. The script gates on Total for brevity; a per-transaction version, which is what real acceptance criteria need, is on Analyzing Results & Finding Bottlenecks.

Source: Generating Report Dashboard (User Manual), Adding a job summary (GitHub Docs)

The workflow

One workflow with two jobs: smoke runs the Maven route on every pull request, and load runs the Docker route on a weekday-night schedule. Both can be started by hand. Action versions are pinned to exact releases; for stricter supply- chain control pin them to the commit SHA of those releases instead.

# .github/workflows/performance.yml
name: performance

on:
  pull_request:
    branches: [main]
  schedule:
    - cron: "30 2 * * 1-5"         # 02:30 UTC, Monday to Friday
  workflow_dispatch:

permissions:
  contents: read

concurrency:
  group: performance-${{ github.ref }}
  cancel-in-progress: true

jobs:
  smoke:
    name: Performance smoke (Maven)
    if: github.event_name != 'schedule'
    runs-on: ubuntu-24.04
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v7.0.1

      - uses: actions/setup-java@v6.0.1
        with:
          distribution: temurin
          java-version: "21"
          cache: maven

      - name: Start Bookshelf
        run: |
          docker compose up -d --build --wait
          curl -fsS --retry 30 --retry-delay 2 --retry-all-errors http://localhost:8080/actuator/health

      - name: Smoke profile with the results gate
        run: mvn -B -ntp -f perf/pom.xml clean verify

      - name: Upload results, dashboard and log
        if: always()
        uses: actions/upload-artifact@v7.0.2
        with:
          name: jmeter-smoke-${{ github.run_id }}
          path: |
            perf/target/jmeter/results/
            perf/target/jmeter/reports/
            perf/target/jmeter/logs/
          if-no-files-found: warn
          retention-days: 14

      - name: Stop Bookshelf
        if: always()
        run: docker compose down -v

  load:
    name: Load test (Docker)
    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-24.04
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v7.0.1

      - name: Build the JMeter image
        run: docker compose -f compose.yaml -f compose.perf.yaml build jmeter

      - name: Start Bookshelf
        run: |
          docker compose up -d --build --wait
          curl -fsS --retry 30 --retry-delay 2 --retry-all-errors http://localhost:8080/actuator/health

      - name: Load profile in CLI mode
        run: |
          mkdir -p perf/results
          docker compose -f compose.yaml -f compose.perf.yaml run --rm jmeter \
            -n -t src/test/jmeter/bookshelf-load.jmx -q profiles/load.properties \
            -Jhost=bookshelf-api -Jport=8080 \
            -l results/load.jtl -j results/load.log -f -e -o results/load-report

      - name: Check thresholds
        run: perf/perf-gate.sh perf/results/load.jtl perf/results/load-report 1 800

      - name: Upload JTL, dashboard and log
        if: always()
        uses: actions/upload-artifact@v7.0.2
        with:
          name: jmeter-load-${{ github.run_id }}
          path: perf/results/
          if-no-files-found: warn
          retention-days: 30

      - name: Stop Bookshelf
        if: always()
        run: docker compose down -v

How the pieces map onto the pipeline:

Step Notes

on and the job if conditions

Pull requests get the smoke job only; the schedule gets the load job only; a manual run gets both.

concurrency

A new push to the same pull request cancels the running smoke test instead of queuing a second one.

actions/setup-java with cache: maven

Caches the local Maven repository, including the JMeter artifacts the configure goal downloads, so later runs skip most of the download.

docker compose up -d --build --wait

Builds and starts the Bookshelf stack; --wait waits until the services are running (or healthy, for services with a health check). The curl retry loop waits for the API’s Actuator health endpoint, which is what "ready for load" means.

mvn ... clean verify

configure, jmeter and results in one step: the step, and so the job, fails when the error rate is above the threshold.

perf-gate.sh

The Docker route’s verdict. The docker compose run step fails only when JMeter cannot start the test (exit code 1); whether the test passed is decided here.

actions/upload-artifact with if: always()

Runs even when an earlier step failed, so a failing run still leaves its JTL (or Maven’s CSV), the dashboard and the log behind. The dashboard is uploaded as a folder and downloaded as a zip; open index.html after unpacking it.

docker compose down -v with if: always()

Removes the containers and their volumes, so a rerun starts from the same data.

The jobs use only GITHUB_TOKEN with read access. If the plan needs credentials beyond the placeholder test users (for example a database password for JDBC samplers), store them as repository secrets, map them to environment variables in the step (env: DB_PASSWORD: ${{ secrets.BOOKSHELF_DB_PASSWORD }}) and pass them as -J or -D properties. Never write them into the POM, a profile file or the plan.

Source: Workflow syntax for GitHub Actions (GitHub Docs), actions/upload-artifact (GitHub), actions/setup-java (GitHub), docker compose up (Docker Docs)

Three ways to fail the job on thresholds

Method What it checks When it acts Exit code behaviour

results goal (Maven route)

Overall percentage of failed samples, against <errorRateThresholdInPercent>

After the run

Maven exits 1; the step fails

perf-gate.sh (any route)

Sample count, error rate and p95 from statistics.json; extendable per transaction

After the run and the dashboard

The script exits 1; the step fails

Auto-Stop Listener (jpgc-autostop)

Error rate, average response time or latency, a response-time percentile, sustained for N seconds

During the run: stops a test that is already failing

JMeter still exits 0, so a gate is still needed

The Auto-Stop Listener is not a gate on its own, it saves time. A 10-minute load test against a broken build should not run for 10 minutes. Add it in the GUI under Add → Listener → jp@gc - AutoStop Listener, at Test Plan level, and set the criteria; leave unused criteria at 0, which disables them. The fragment the GUI writes for "stop when more than 5 % of samples fail for 30 seconds" (with the unused criteria at 0):

<kg.apc.jmeter.reporters.AutoStop guiclass="kg.apc.jmeter.reporters.AutoStopGui"
    testclass="kg.apc.jmeter.reporters.AutoStop" testname="Stop on sustained errors">
  <stringProp name="avg_response_time">0</stringProp>
  <stringProp name="avg_response_time_length">0</stringProp>
  <stringProp name="error_rate">5</stringProp>
  <stringProp name="error_rate_length">30</stringProp>
  <stringProp name="time_setting">response time</stringProp>
  <stringProp name="avg_response_latency">0</stringProp>
  <stringProp name="avg_response_latency_length">0</stringProp>
  <stringProp name="percentile_value">0</stringProp>
  <stringProp name="percentile_response_time">0</stringProp>
  <stringProp name="percentile_response_time_secs">0</stringProp>
  <stringProp name="custom_validation_duration">0</stringProp>
</kg.apc.jmeter.reporters.AutoStop>
<hashTree/>

A run with a 10-second window, 2 threads and the API down, measured on 5.6.3 with Auto-Stop 0.3:

Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
summary +      4 in 00:00:02 =    2.6/s Avg:     4 Min:     1 Max:    13 Err:     4 (100.00%) Active: 2 Started: 2 Finished: 0
AutoStop - Error rate more than 5 for 10s. Auto-shutdown test...
Command: StopTestNow received from /127.0.0.1
summary =     38 in 00:00:10 =    3.7/s Avg:     2 Min:     1 Max:    13 Err:    38 (100.00%)
Tidying up ...    @ 2026 Oct 14 13:41:09 CEST (1791978069046)
... end of run
$ echo $?
0

The test planned for 120 s stopped after 10 s, the dashboard was still generated (-e -o runs after a stop received on the UDP port), and the exit code was 0. Three practical notes:

  • It talks to the shutdown port. In CLI mode the listener stops the test through the UDP port on 127.0.0.1 (4445 by default). Two JMeter processes in the same network namespace move to 4446 and so on; that is fine inside separate containers, but keep it in mind for parallel runs on one host.

  • Empty fields log errors. An empty value in an unused criterion (as left by hand-edited XML) makes version 0.3 log Wrong response time errors at the start of the test; set unused criteria to 0, as above.

  • The auto_stopped flag. The wiki says the listener sets auto_stopped to true when it starts the shutdown; in 0.3 it is a Java system property of the JMeter JVM, readable from a tearDown Thread Group with ${__groovy(System.getProperty('auto_stopped'))}. It is not visible to the CI step, which is another reason to let perf-gate.sh decide.

The listener and its plugin version are listed on Plugins & Plugins Manager; the Maven route needs kg.apc:jmeter-plugins-autostop:0.3 in <jmeterExtensions>, as in the POM above.

Source: AutoStop Listener (JMeter Plugins wiki), Getting Started: CLI Mode (User Manual)

Load-generator cost on CI runners

GitHub’s standard Linux runners have 4 vCPUs and 16 GB of memory for public repositories and 2 vCPUs and 8 GB for private repositories (14 GB of SSD in both cases). On such a runner the Bookshelf stack (API and database) and JMeter share the machine. Worked numbers for the two profiles:

Profile Threads Request rate Fit on a 2-vCPU runner

smoke

1

about 0.3 requests/s

Trivial for JMeter. The job’s time is dominated by starting the stack and, the first time, downloading JMeter.

load

50

about 15.6 requests/s

JMeter needs little CPU at this rate (each thread sleeps 3 s per request), and 1 GB of heap is plenty. The API and its database compete for the same 2 vCPUs, so percentiles are inflated and vary from run to run.

stress, soak

hundreds, or hours

depends on the profile

Not on a shared runner. Run them on a dedicated performance environment or larger runners, as the books recommend.

The consequences for thresholds: gate the CI runs on errors and on gross regressions (a p95 several times the expected value), and keep tight percentile criteria for the dedicated environment. Record the runner type with the results, because a 2-vCPU and a 4-vCPU run are not comparable. The analysis of run-to-run variance is on Analyzing Results & Finding Bottlenecks, and generator sizing on Running Tests in CLI Mode.

Source: GitHub-hosted runners (GitHub Docs), Ubuntu 24.04 runner image (actions/runner-images)

Pitfalls

  • Trusting the exit code. JMeter exits 0 with 100 % failed samples, after an Auto-Stop and after a usage error. Every job needs a results-based gate.

  • Forgetting <jmeterVersion>. Plugin 3.8.0 silently runs JMeter 5.6.2.

  • Re-running without clean. With <testResultsTimestamp>false</testResultsTimestamp>, the second run fails because the dashboard folder is not empty; after changing the JMeter or plugin version, old jars in target break the run.

  • Transitive plugin dependencies. downloadExtensionDependencies left at true copies old JMeter core jars from the kg.apc plugins' POMs next to 5.6.3.

  • Plugins that are not Maven artifacts. The gRPC and HTTP/2 plugins need an internal repository (or an install:install-file step) before the Maven route can use them.

  • Reading properties as variables. <propertiesUser> values are properties: ${__P(threads,1)}, not ${threads}.

  • Gating a smoke test on percentiles. Nineteen samples do not make a p95. Gate smoke runs on errors and use Duration Assertions for hard limits.

  • Aiming at localhost from a container. Use the Compose service name (-Jhost=bookshelf-api) on the Compose network.

  • Starting the load before the API is ready. A container that is "started" is not an application that answers; wait for the health endpoint.

  • No artifacts for a failed run. Without if: always(), the upload step is skipped exactly when the results are needed.

  • Unpinned versions. An unpinned action, base image, JMeter archive or plugin changes the test without a change in the repository. Pin all of them and record the PluginsManagerCMD.sh status output with the results.

  • Load against systems you do not own. CI makes it easy to run a load test against a shared environment by accident. Keep the target configurable and aimed at the stack the job started.

From the books to 5.6 to 6.0

In the books (JMeter 2.11 to 3.0) Today

jmeter-maven-plugin 1.10.1 with one jmeter goal bound to verify, plans in src/test/jmeter (Pro Apache JMeter ch. 7; JMeter Cookbook ch. 8)

Plugin 3.8.0: separate configure, jmeter and results goals, a selectable JMeter version (default 5.6.2, set 5.6.3), plugins and libraries as Maven artifacts, dashboard generation and an error-rate threshold.

Jenkins with the Performance plugin, a parameterised build and error-percentage thresholds marking the build unstable or failed (JMeter Cookbook ch. 8)

Any CI system that can run Maven or Docker; this page uses GitHub Actions, with the results goal or a script on statistics.json as the gate and the HTML dashboard as an artifact. The Jenkins Performance plugin still exists but is not covered here.

Performance smoke tests in the build, full suites when needed, a dedicated performance environment for CI (Pro Apache JMeter ch. 2)

Unchanged as practice: a smoke job per pull request, scheduled load jobs, and dedicated environments for capacity numbers.

Plugins installed by unzipping the Standard and Extras sets (JMeter Cookbook appendix)

PluginsManagerCMD.sh install id=version in a Dockerfile, or <jmeterExtensions> in a POM.

"Non-GUI mode" (-n)

Called CLI mode since 5.1; same option.

JMeter 6.0 (planned, not released)

Requires Java 17 (21 recommended), removes the MongoDB elements and moves to Groovy 5, among other changes listed in xdocs/changes.xml on master. For CI this means a Java 17+ runner, a jmeter-maven-plugin release that knows the 6.0 module list, and re-checked plugins. Re-check when 6.0 is released.

References