Troubleshooting & Debugging

Validate (right-click a Thread Group, Validate) runs that Thread Group with one thread, one iteration, no timers and no Backend Listener, so a journey that takes minutes under its real settings runs in seconds. The properties behind it (testplan_validation.*) and a worked timing example are on GUI & Test Plan Workflow.

Validate exists only in the GUI. When the problem appears in CLI runs (a CI agent, a container, a distributed node), reproduce it in CLI mode with the run properties the Bookshelf plan already reads (threads, rampup, duration), a log of its own and DEBUG for the package under suspicion:

The console lines are those of a 5.6.3 CLI run; the summary figures are illustrative. With the Bookshelf 3 s think time and about 0.2 s per request, one thread makes roughly 1 / 3.2 ≈ 0.31 requests per second, so 30 s give about 10 requests: few enough to read every line of the log, enough to cover the journey twice. Never use these logging settings in a measured run: DEBUG writes several lines per sample and the disk becomes part of the measurement.

Source: Getting Started: Command-line options (User Manual)

Both elements make variables and properties visible in the Response data pane of the View Results Tree. Where to place them and how they help with correlation is on Correlation & Extractors; this section lists their fields with the defaults a new element gets in the 5.6.3 GUI.

Debug Sampler (Add → Sampler → Debug Sampler) produces a sample of its own whose response data lists the current values. It is a sampler, so it appears in the results and counts in the summariser: disable it before a load run.

Debug PostProcessor (Add → Post Processors → Debug PostProcessor) adds a sub-sample to the sampler it is attached to, so the values appear under the request that set them instead of as a separate sample.

Sampler Properties is the field that answers "what did JMeter actually send": a ${bookId} that was never set appears there literally. As a fragment, attached to the Bookshelf login request:

The variables listing includes password and, after the login, token. Treat a View Results Tree or an XML results file that contains Debug output as a secret, and remove the element (or set enabled="false") before the plan is committed.

Source: Debug Sampler (Component Reference) and Debug PostProcessor (Component Reference)

The View Results Tree is the debugging listener; its tabs, every renderer and its properties are described on Listeners & Results Files. For troubleshooting, the renderer to pick depends on the question:

Two limits mislead people when they debug with the tree. It keeps only the last 500 samples (view.results.tree.max_results), so the first iterations of a longer debug run disappear; and it does not display responses above view.results.tree.max_size (10485760 bytes in the 5.6.3 jmeter.properties; the component reference still mentions 200 KB). For a bigger body, use Save Responses to a file (below). The Request tab shows only the headers JMeter added, not the ones the HTTP client adds later, such as Host; wire logging shows those. The Browser renderer is absent on Java 9 and later unless JavaFX is added to the launcher, as the Hints and Tips chapter describes.

Source: View Results Tree (Component Reference) and Hints and Tips: Browser renderer (User Manual)

In a CLI run there is no View Results Tree to look at, and the CSV results file has no bodies. Add → Listener → Save Responses to a file with Save Failed Responses only writes one file per failed sample: the body the server actually sent, which is usually the fastest way to the cause of a 500 or a failed assertion. The options to put error bodies into an XML results file instead are on Listeners & Results Files.

It costs nothing while samples succeed and one file write per failure. In a run where everything fails (the server is down) it writes one file per sample, so keep it for smoke and debug runs or watch the disk.

Source: Save Responses to a file (Component Reference)

Since 3.2 JMeter logs through SLF4J and Log4j 2, configured by log4j2.xml (in bin/, or the directory JMeter is started from). The log file is jmeter.log in the directory JMeter was launched from, unless -j names another one. Installation details (the configuration file, the GUI log viewer) are on Installation & Configuration and the command-line options on Running Tests in CLI Mode. The tools at a glance:

The categories worth knowing, all commented examples in the 5.6.3 log4j2.xml:

A log line has the layout date level category: message, with the category shortened (o.a.j.t.JMeterThread). An exception that stops a thread is logged at ERROR by o.a.j.t.JMeterThread with its stack trace, whose at lines name the element class that failed (for example CSVDataSet). Groovy scripts write to the same file through their log binding, which is often the quickest way to print an intermediate value (see Scripting with JSR223 & Groovy).

Source: Getting Started: Logging and error messages (User Manual), Hints and Tips: Enabling Debug logging (User Manual) and log4j2.xml at rel/v5.6.3 (apache/jmeter)

When the question is what went over the network (which headers, which body, which status line), the View Results Tree’s Request tab is not enough, because it shows only what JMeter added. The default HTTP implementation, Apache HttpClient 4, logs the real exchange through two categories:

The shipped log4j2.xml has commented examples for "header wire + context logging" (org.apache.http at DEBUG with org.apache.http.wire at ERROR) and "full wire + context logging" (org.apache.http at DEBUG). For one run, pass the categories with -L:

The output has this shape (the line format of a 5.6.3 run against a local server; abridged, with the Bookshelf search as the request):

>> is sent, << received, and http-outgoing-N identifies the connection, so a new number for every request shows that connections are not being reused. For connection-pool questions (reuse, stale connections, time_to_live) use org.apache.http.impl.conn at DEBUG instead.

For TLS, the HTTP logs stop at the socket; the JVM’s own -Djavax.net.debug=ssl:handshake prints the handshake, as shown on HTTP Requests.

Wire logs contain everything: the Authorization: Bearer header, the login body with the password, cookies and every response body. Use them with one thread and placeholder credentials, never commit the log, and never enable them in a load run, where they multiply the disk I/O by the size of every request and response.

Source: Getting Started: Logging and error messages (User Manual) and HttpClient Logging Practices (Apache HttpComponents)

Symptom. java.lang.OutOfMemoryError with a reason: Java heap space (the heap is full), GC overhead limit exceeded (the JVM spends nearly all its time collecting), Metaspace (too many classes, typically JSR223 scripts that are not cached), or unable to create native thread (the operating system refused another thread). Before the error the summariser shows throughput falling while response times rise for every request at once, which is the generator, not the server.

Cause. The 5.6.3 launcher starts JMeter with HEAP="-Xms1g -Xmx1g -XX:MaxMetaspaceSize=256m" unless HEAP is set, and the Getting Started guide warns that 1 GB may not be enough for a plan and its thread count. The usual consumers are listeners that keep results in memory (View Results Tree, View Results in Table, Aggregate Report, the plugin graphs), large responses held by many threads at once, and XML results with response data.

Fix, in this order:

Evidence. The launcher always adds -XX:+HeapDumpOnOutOfMemoryError, so the JVM writes a .hprof file (named after the process id) into the working directory when the heap is exhausted; open it in a heap analyser to see which objects filled it. During a run, bin/heapdump.sh asks a CLI run for a dump through the shutdown port without waiting for the error. Sizing the heap, thread stacks and containers is worked through on Running Tests in CLI Mode.

Source: Getting Started: Running JMeter (User Manual) and Best Practices (User Manual)

Two different problems share this exception.

At load, inside samples: Cannot assign requested address (Linux; Windows reports Address already in use: connect). Every new TCP connection needs a local ephemeral port, and a port stays in TIME_WAIT for 60 s after the client closes it. A plan that opens a new connection per request exhausts the default Linux range (32768 to 60999, 28,232 ports) at about 470 new connections per second sustained for a minute. The errors look like server failures, but they appear on the generator, at a constant rate once the threshold is crossed, and the server’s logs show no corresponding requests. The fix is to keep connections alive (Use KeepAlive, Same user on each iteration), then to spread the load, then to widen the port range; the arithmetic, the properties and the operating system commands are on Running Tests in CLI Mode.

At start-up of a listener: Address already in use. The HTTP(S) Test Script Recorder (default port 8888) or a jmeter-server (registry port 1099, or server_port) cannot open its port because another process already holds it, usually a previous recorder or server that was not stopped. Find the process (lsof -i :8888 on Linux and macOS, netstat -ano on Windows), stop it or choose another port. The UDP shutdown port of a CLI run (4445) is the exception: when it is taken, JMeter tries the next port, up to jmeterengine.nongui.maxport, and prints the one it used.

Source: Properties Reference: HttpClient4 and HTTP(S) Test Script Recorder (Component Reference)

When the HTTP sampler gets no HTTP response at all, it records the exception class as the response code, prefixed with Non HTTP response code:, and the exception message as the response message, prefixed with Non HTTP response message:. The class name is the diagnosis. These rows were reproduced on 5.6.3 (HttpClient4 implementation, Java 21) against local endpoints:

A row that is the same for every sample from the first second is a configuration problem (address, protocol, URL). A row that appears only after load has risen (timeouts, refused connections, NoHttpResponseException) is a capacity signal for the server or a resource limit on the generator. The dashboard’s Errors table groups samples by these strings; the classification of HTTP status codes is on Analyzing Results & Finding Bottlenecks.

Retries hide these errors: httpclient4.retrycount defaults to 0 in 5.6.3, and should stay there for measurement.

Source: HTTP Request (Component Reference) and Properties Reference: HttpClient4

JMeter’s HTTP samplers accept every server certificate, valid or not, so an untrusted or expired server certificate is not a cause of failure (and a load test does not detect one). What remains:

To see the handshake, add -Djavax.net.debug=ssl:handshake to a one-thread run; the JVM prints the protocol versions and cipher suites offered and the alert that ended the handshake. The TLS properties and the client-certificate setup are on HTTP Requests. gRPC over TLS has its own settings and fails with UNAVAILABLE when TLS is on one side only; see gRPC Load Testing.

Source: Getting Started: SSL Encryption (User Manual) and Properties Reference: SSL configuration

While recording HTTPS traffic, the browser rejects the certificate the recorder presents for each site and jmeter.log shows unknown_ca: the browser does not trust the root CA that JMeter generates (ApacheJMeterTemporaryRootCA.crt in bin/). Install it as a trusted authority in the store the browser actually reads and restart the browser. The certificate is valid for 7 days by default and is regenerated with proxyserver.jks, so a recording that worked last week can fail today for the same reason. The full table of certificate symptoms (expiry, dynamic keys, keytool not found, a browser that refuses outright) is on Recording & cURL Import.

Source: HTTP(S) Test Script Recorder (Component Reference)

Distributed testing has its own failure modes, all of which happen before the first sample. These were reproduced on 5.6.3 with a client and a server on the same machine:

The server cannot start: no keystore. Since 4.0, RMI uses SSL unless server.rmi.ssl.disable=true, and the keystore defaults to rmi_keystore.jks (type JKS, alias rmi, password changeit). Without it, jmeter-server stops at once:

The client cannot connect: no keystore on the client. The client needs the same keystore. Without it, -R fails for each server and the run exits with status 1:

The client cannot connect: nothing listening. With a keystore (or with SSL disabled on both sides), a server that is not running, a wrong port or a firewall gives:

The fixes, which the distributed testing page of this section covers step by step:

Source: Remote Testing (User Manual) and Properties Reference: Remote hosts and RMI configuration

Missing file. A CSV Data Set Config whose file does not exist does not produce failed samples: the thread stops at its first iteration, before any request. A 5.6.3 run of a plan whose Filename was data/users.csv with no such file printed a summary of zero samples, and jmeter.log held the reason:

A zero-sample summary with Err: 0 therefore means "read the log", not "all good". Relative paths are resolved against the folder of the .jmx, not the working directory, and in a distributed run every server needs its own copy; see Parameterization & Test Data.

Empty or wrong values. The file is found but the values are wrong. Use a Debug PostProcessor on the first request and compare the variables with the file:

Source: CSV Data Set Config (Component Reference)

JMeter does not report an undefined variable or a misspelt function: the Getting Started guide states that the reference "is just used as is". What happens next depends on where it is used. Both cases were reproduced on 5.6.3 with a variable term that was never set:

The causes, in the order they occur in practice:

To find every element that uses a variable, use the Search menu with the variable name: matching elements are highlighted in the tree.

Source: Functions and Variables (User Manual) and Hints and Tips: Searching (User Manual)

When the Bookshelf plan sends Authorization: Bearer TOKEN_NOT_FOUND or calls /api/loans/LOANID_NOT_FOUND, the extractor ran and did not match. That is the purpose of a loud default value; the alternative is the silent literal of the previous section. The usual causes:

Make the failure fail loudly: a Response Assertion on the variable that fails when it equals the default, and a Result Status Action Handler that starts the next iteration. The defaults, the -1 rules and the assertion are on Correlation & Extractors and Assertions.

Source: JSON Extractor (Component Reference)

Symptom: the plan does not load. A .jmx that uses a plugin element names the plugin’s class. When that class is not on JMeter’s classpath, the plan cannot be read at all. Reproduced on 5.6.3 with the Concurrency Thread Group of jpgc-casutg not installed:

The class name tells which plugin is missing; path and line number point at the element in the .jmx. In the GUI the same error appears in a dialog when opening the file.

Symptom: the plugin is installed but its elements are not in the Add menu, or a sample fails with NoClassDefFoundError.

bin/PluginsManagerCMD.sh install-for-jmx bookshelf-stress.jmx reads the class names from the plan and installs the plugins that provide them, which fixes the first symptom on a CI agent or in a Docker image. All of this, including the discovery mechanism in 5.6 and the lib/lib/ext rules, is on Plugins & Plugins Manager.

Source: Getting Started: Classpath configuration (User Manual) and JMeter Plugins Manager (JMeter Plugins wiki)

The gRPC Request plugin (2.0.2) fails in two places, and its response code says which:

The messages below are those of the 2.0.2 source (ServiceResolver, ProtoMethodName, ProtocInvoker); they appear in the stack trace of the sample’s response data and in jmeter.log.

Two gRPC failures look like server problems but are not: 14 UNAVAILABLE on every call after one earlier error (in 2.0.2 a failed call closes the thread’s channel and it is not rebuilt), and 4 DEADLINE_EXCEEDED under load because the Deadline field kept its GUI default of 1000 ms. The plugin’s fields, its error mapping and these pitfalls are on gRPC Load Testing.

Source: bakthava/jmeter-grpc-request README and source at v2.0.2 (GitHub) and Status Codes (gRPC)