Running Tests in CLI Mode

Source: Getting Started: Logging and error messages (User Manual)

The command line is processed after jmeter.properties, user.properties and system.properties have been loaded. -q, -S, -J, -D, -G and -L are then applied in the order they appear on the command line, so a later option overrides an earlier one. That makes this pattern safe: a profile file with defaults, then individual overrides.

Two details matter in scripts. An empty value (-Jthreads=) removes the property, so the plan falls back to the default in ${__P(threads,1)}. And in 5.6.3 numeric properties are parsed as written, so a value with a trailing space copied into a properties file can fail to parse; trimming whitespace when parsing numeric properties is a planned 6.0 change in changes.xml on master.

-D and -S set Java system properties, which configure the JVM and libraries rather than the plan: TLS keystores, java.net.preferIPv4Stack, proxy settings. -G is the distributed-testing counterpart of -J: the controller sends the properties to every server, which is how one command line sets threads on four generators at once (the page on distributed testing covers it).

Source: Getting Started: Overriding Properties Via The Command Line (User Manual)

-L changes log levels for one run without editing log4j2.xml:

-i points to another log4j2.xml entirely. Never leave DEBUG logging on in a load run: every sample then writes several log lines, and the disk becomes part of the measurement.

Source: Getting Started: Logging and error messages (User Manual)

These options are for a load generator that reaches the system under test through a corporate proxy. The manual’s example is jmeter -E https -H my.proxy.server -P 8000 -u username -a password -N localhost; the non-proxy list uses a vertical bar between entries and accepts wildcards such as *.apache.org. Testing through a shared proxy measures the proxy as much as the application, so for a load test the generator should normally sit where it reaches the target directly. The HTTP(S) Test Script Recorder is a different thing: it is JMeter acting as a proxy for a browser while recording.

Source: Getting Started: Using JMeter behind a proxy (User Manual)

-e -o generates the dashboard at the end of the run. -g generates it later from any results file, which is the way to rebuild a report after a run was killed, after changing reportgenerator.properties, or from a file merged from several generators:

Source: Generating Report Dashboard (User Manual)

-r and -R turn the CLI process into the controller of a distributed test; -X shuts the servers down at the end; -s (or the jmeter-server script) starts a server. The manual’s form is jmeter -n -t testplan.jmx -r [-Gprop=val] [-Gglobal.properties] [-X]. RMI, keystores and the "every server runs the whole plan" rule are the subject of the page on distributed testing.

Source: Getting Started: Server Mode (User Manual) and Remote Testing (User Manual)

The output below is genuine: a 5.6.3 installation on Java 21, running a one-request plan (cli-smoke.jmx, a Thread Group as above with one HTTP Request and a 500 ms Constant Timer) against a static file served locally by python3 -m http.server. The summariser interval was lowered to 10 s so that a 40-second run shows several lines.

Source: Properties Reference: Summariser

The process exited with code 0 and left run1.jtl, run1.log and a report1 folder with index.html, statistics.json, content/ and sbadmin2-1.0.7/. The console also started with a few warnings that are not JMeter errors: Log4j StatusConsoleListener lines saying that package scanning for plugins is deprecated, and on macOS a JVM warning about sun.awt.X11. Removing the Log4j package scanning is a planned 6.0 change.

Checking the numbers. Five threads, each waiting 500 ms after a request of about 3 ms, should produce 5 / 0.503 ≈ 9.9 requests per second, which is what the full-load intervals report. The first line covers only 4 s and 4 started threads because ramp-up starts one thread per second (5 threads over 5 s) and the summariser aligns its lines to multiples of the interval on the wall clock, not to the start of the test. The last line has Active: 0 and Finished: 5: the scheduler’s 40 s duration ended every thread.

By default (since 3.1) the summariser ignores the parent samples generated by Transaction Controllers, so its counts are real requests only.

Source: Properties Reference: Summariser and Generate Summary Results (Component Reference)

What to watch while a run is live. The summariser is the first warning system. A rate that stops rising while Active keeps growing means the target (or the generator) is saturated; a rate that falls while Active is constant means response times are growing, which in a closed model throttles the load by itself; a non-zero Err early in the run usually means a data or correlation problem rather than load. The numbers are averages, so they are a trigger to look at the dashboard or a live monitoring system, not a result.

Locale pitfall. The launcher sets -Duser.language=en -Duser.region=EN through JMETER_LANGUAGE. Starting JMeter directly with java -jar ApacheJMeter.jar skips that, and on a machine with a comma decimal separator the summariser then prints 1,7/s and (0,00%), which breaks scripts that parse these lines. Always use the launcher.

Source: Properties Reference (JMeter engine section)

The documentation disagrees with itself on two of these defaults. Getting Started says jmeterengine.stopfail.system.exit defaults to false, while the properties reference, the commented line in jmeter.properties and the 5.6.3 source all use true. For jmeter.exit.check.pause the properties reference gives 2000, while Getting Started, jmeter.properties and the source use 0 (no check). The values in the table are the ones the 5.6.3 code uses.

Two CLI instances were started on the same machine against the same local target; the second one found 4445 busy and took 4446. shutdown.sh then stopped the first gracefully and stoptest.sh 4446 stopped the second abruptly (genuine 5.6.3 output, summariser interval 10 s):

The first instance:

The second instance:

Source: Properties Reference (jmeterengine.nongui.port) and JMeter.java (5.6.3 source)

Both processes exited with code 0. When several instances share one generator, give each a fixed port so that the operator does not have to read the console to find it:

In containers. The command has to originate inside the container’s network namespace, so run the script there: docker exec <container> /opt/apache-jmeter-5.6.3/bin/shutdown.sh (or kubectl exec for a pod), with the path of the image’s JMeter installation. A docker stop instead sends SIGTERM, which is covered next.

JMeter registers a JVM shutdown hook that flushes and closes the results files, so Ctrl+C (SIGINT), kill (SIGTERM) and docker stop keep the samples recorded so far. They do not run the graceful shutdown: in-flight samples are lost, tearDown Thread Groups do not run, and the -e dashboard is not generated. Measured on 5.6.3: a run killed with SIGTERM after 8 s exited with code 143 (the shell convention for 128 plus signal 15) and left a complete, readable JTL file and no report folder. Rebuild the report with jmeter -g results.jtl -o report. SIGKILL (kill -9) runs no hook at all, and the last buffered samples can be lost.

Order of preference: shutdown.sh, then stoptest.sh if threads are stuck in long timeouts, then SIGTERM, and SIGKILL only if the JVM does not react.

Source: Changes History (2.10: shutdown hook for kill and Ctrl+C) and Getting Started: CLI mode shutdown (User Manual)

Since 2.5.1 JMeter no longer calls System.exit() when a CLI test ends normally; it ends its own non-daemon threads and lets the JVM exit. A plugin or script that leaves a non-daemon thread running keeps the JVM alive after ... end of run, and a CI job then hangs. Two properties address this: jmeter.exit.check.pause (for example -Jjmeter.exit.check.pause=2000) makes JMeter print the threads that are still alive after that pause, which names the culprit; jmeterengine.force.system.exit=true forces the exit. Fix the plugin or script where possible and use the second only as a workaround.

Source: Getting Started: CLI mode shutdown (User Manual)

The dashboard writes statistics.json, whose Total entry holds the overall error percentage and response-time percentiles (pct1ResTime, pct2ResTime and pct3ResTime are the 90th, 95th and 99th percentiles by default). A gate after the run turns them into an exit code. The thresholds below are the Bookshelf search SLA: under 1 % errors and a 95th percentile under 800 ms.

Source: Generating Report Dashboard (User Manual)

The jmeter-maven-plugin’s results goal does a similar check inside a Maven build; it and GitHub Actions are covered in the page on CI/CD with Maven, Docker and GitHub Actions.

The launchers set the heap through the HEAP variable, -Xms1g -Xmx1g -XX:MaxMetaspaceSize=256m by default, and always add -XX:+HeapDumpOnOutOfMemoryError, so an out-of-memory error leaves a .hprof file to analyse.

Source: Getting Started: Configuring JMeter (User Manual)

What uses the heap in a CLI run:

What uses memory outside the heap, and therefore outside HEAP: metaspace (bounded by MaxMetaspaceSize), thread stacks, direct buffers and the JVM itself. Thread stacks are the one that grows with load: the Java 21 java reference gives a default stack size (-Xss) of 1024 KB on Linux/x64 and macOS/x64 and 2048 KB on Linux/AArch64 and macOS/AArch64. That is reserved address space, committed only as the stack is used, but it sets the ceiling.

Worked example. 500 threads on a Linux/x64 generator reserve up to 500 × 1 MB ≈ 0.5 GB of stack outside the heap. With -Xmx4g and 256 MB of metaspace, plan for about 5 GB of process memory plus the operating system and any agent on the machine, so an 8 GB machine is the minimum and the heap should not be raised further without more RAM: a generator that swaps measures its own disk. Keep -Xms equal to -Xmx so the heap does not resize during the test.

In containers, modern JDKs size the default heap from the container memory limit (-XX:MaxRAMPercentage, default 25 %). The JMeter launcher sets an explicit -Xmx through HEAP, which takes precedence, so set HEAP to fit the container limit (for example 75 % of it). The commented-out RUN_IN_DOCKER line in bin/jmeter uses -XX:+UseCGroupMemoryLimitForHeap, an experimental option for Java 8u131 and 9 that current JDKs no longer accept; leave it commented out.

Source: The java Command (Java SE 21)

The launchers select G1 through GC_ALGO. The shipped 5.6.3 scripts (bin/jmeter and bin/jmeter.bat) use -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:G1ReservePercent=20; the Getting Started page lists the same options with MaxGCPauseMillis=250. The script is what actually runs. Either way, a GC pause stops every thread of the generator, and a pause in the middle of a sample is added to that sample’s response time.

To see whether GC is a problem, enable a GC log for the run. bin/jmeter contains a ready-made VERBOSE_GC line for Java 9 and later; the same thing through JVM_ARGS:

or watch a running generator with the JDK tools:

Source: The java Command (Java SE 21) and Getting Started: Configuring JMeter (User Manual)

Healthy: young collections of a few milliseconds, old generation that drops back after collections, no full GC. Undersized: old generation climbing towards 100 %, back-to-back collections, full GCs, rising response times on every sampler at once. The fix is more heap or less retained data, not a different collector.

CPU cost grows with requests per second and with the work per response, not with the number of threads: a thread sleeping in a timer costs almost nothing. The expensive parts are TLS handshakes (especially without keep-alive), compression, extractors and assertions on large bodies (regular expressions and XPath are the costly ones), JSR223 scripts that are not cached, and writing results. Keep the generator’s CPU well below saturation during the hold; a common rule of thumb is to stay under about 80 %, the utilisation figure Pro Apache JMeter (ch. 2) works with. Above that, scheduling delays inflate response times and the summariser’s rate flattens while the server is idle.

To find the ceiling of a machine without a network, the best-practices chapter suggests the Java Request sampler with its JavaTest class, which needs no network access and shows the maximum throughput achievable on that platform.

The generator’s network link carries every request and every response. Worked example for the Bookshelf mix: at 150 requests per second with an average response of 20 KB, the generator receives 150 × 20 KB = 3 MB/s, about 24 Mbit/s, comfortable on a 1 Gbit/s link. A catalogue export of 2 MB requested 50 times a second is 100 MB/s, 800 Mbit/s, and saturates the same link before the server. The JTL’s bytes and sentBytes columns and the dashboard’s receivedKBytesPerSec give the real figure after a smoke run; compare it with the link capacity.

Also check:

Every open socket is a file descriptor, as are the results files, the run log and every jar on JMeter’s classpath. A thread with keep-alive holds one connection per target host; with embedded resources and parallel downloads it can hold up to the pool size (default 6) more. The per-process limit is the shell’s ulimit -n, which on many Linux distributions is 1024 for interactive sessions.

Worked example. 500 threads with keep-alive to one host: about 500 sockets, plus roughly 150 for jars, logs and results, already over half of 1024. With parallel downloads of embedded resources (up to 6 per thread) the need rises to about 3,500 and the run fails with java.net.SocketException: Too many open files. Raise the limit before the run:

For services started by systemd, the limit is LimitNOFILE= in the unit; for containers, the runtime’s --ulimit nofile= option. macOS uses ulimit -n and kern.maxfilesperproc in the same way.

Every outgoing TCP connection uses a local (ephemeral) port. When a connection is closed by the client, the socket stays in TIME_WAIT for 60 seconds on Linux (the kernel constant TCP_TIMEWAIT_LEN) before its port can be reused for the same destination. The Linux range is net.ipv4.ip_local_port_range, by default 32768 to 60999, which is 28,232 ports per destination address and port; Windows uses 49152 to 65535 (16,384 ports) by default.

Worked example. A plan that opens a new connection per request (keep-alive off, or a fresh user on each iteration that makes one request) at 500 requests per second leaves 500 × 60 = 30,000 sockets in TIME_WAIT, more than the 28,232 ports available: after about a minute connections fail with java.net.BindException: Cannot assign requested address (Linux) or Address already in use: connect (Windows), and the errors look like server failures. The Bookshelf load profile, by contrast, keeps one connection per thread: 50 connections for the whole run.

Ways to stay clear, in order of preference:

Count TIME_WAIT sockets during a run with ss -tan state time-wait | wc -l on Linux.

Source: IP Sysctl (Linux kernel documentation), The default dynamic port range for TCP/IP (Microsoft Learn) and Properties Reference: HttpClient4

Record the generator’s own metrics for the duration of every run, next to the server’s, and keep them with the report. A test whose generator was saturated is not a valid result, whatever the dashboard says.

Real-time dashboards for the JMeter side (Backend Listener to InfluxDB or Graphite, viewed in Grafana) are the subject of Real-time Monitoring with Backend Listener; the generator’s operating-system metrics can go to the same Prometheus and Grafana as the server’s, see Grafana: Dashboards & Visualization. For CPU and memory limits of generators running in Kubernetes see Resource Management.

Target: the Bookshelf journey at 150 requests per second, one generator, Linux/x64.

If any line does not fit one machine, use two generators of half the load each and merge the results files, or run in distributed mode.