Distributed Testing

Source: Remote Testing (User Manual)

The sequence below follows what 5.6.3 does when the controller is started with -R (from DistributedRunner and ClientJMeterEngine in the source code).

Two consequences follow from this sequence. First, the connection is opened in both directions: the controller calls the worker, and the worker calls back to listener objects exported by the controller. A firewall that only allows controller-to-worker traffic lets the test start but no results arrive. Second, the -G properties are set after the plan is configured and before it runs, which is why ${__P(...)} in Thread Group fields picks them up.

Source: Remote Testing: Tips (User Manual) and DistributedRunner.java (5.6.3 source)

On the controller and on every worker:

If every worker needs different data (unique user accounts, for example), split the file and give each worker its own part. The manual suggests exactly that: different property values in each worker’s user.properties, or different content in the data files.

Source: Remote Testing: Step 0 (User Manual)

Since 4.0, RMI between controller and workers uses TLS by default, and JMeter does not ship keys. The simplest setup is one key pair shared by all nodes. bin/create-rmi-keystore.sh (or .bat) is a thin wrapper around keytool: it generates a 2048-bit RSA key with the alias rmi, valid for 7 days, in rmi_keystore.jks with the password changeit, and passes any extra arguments to keytool. Run it interactively as in the manual, answering rmi as the first and last name, or non-interactively by passing -dname:

Output on 5.6.3 with Java 21:

Then copy the same rmi_keystore.jks to every worker and to the controller, either into bin/ or anywhere else with server.rmi.ssl.keystore.file pointing at it. Three details matter in practice:

Treat the keystore as a credential. Anyone holding it can connect to a worker and make it run an arbitrary plan, which includes OS Process Samplers and Groovy scripts. Never commit it to version control; create it per environment and distribute it like any other secret (for example as a Kubernetes Secret, see Secrets).

Source: Remote Testing: Setting up SSL (User Manual)

On each worker:

What the options do:

A worker that started correctly prints a "Created remote object" line. From the test run (loopback addresses, a non-default registry port and the keystore path shortened):

and its jmeter-server.log ends with:

Only one JMeter server can run per host unless each uses a different registry port (SERVER_PORT=1664 jmeter-server, then host:1664 in the controller’s host list). A worker keeps running after a test and accepts the next one, unless the controller passes -X or the worker sets server.exitaftertest=true. Its console prints one line when a test starts and one when it ends:

Source: Remote Testing: Step 1 (User Manual) and Remote Testing: Using a different port (User Manual)

Either list the workers in the controller’s user.properties, which the GUI’s Remote Start menu and -r use, or pass them on the command line with -R, which has the same effect as -r -Jremote_hosts=.... The default value of remote_hosts is 127.0.0.1, and a host without :port uses server.rmi.port (1099).

java.rmi.server.hostname is a Java system property, not a JMeter property: in user.properties it would have no effect. Put it in bin/system.properties, whose entries JMeter sets as system properties at start-up, or pass -Djava.rmi.server.hostname=10.0.1.10 on the command line:

Source: Remote Testing: Step 2 (User Manual) and Properties Reference: Remote hosts and RMI configuration

The manual is explicit that GUI mode is for debugging; real distributed tests start from a CLI controller:

A real run of the same command, scaled down to 5 threads per worker for 20 s against a local stub of the Bookshelf API (workers on ports 21099 and 21100 of the same machine):

Three things in this output are specific to distributed runs:

The prefix makes the dashboard’s Active Threads Over Time correct for distributed runs. The grpThreads and allThreads columns, however, are counted per worker (they were 5 in this run, never 10), so a step analysis that groups samples by allThreads must group by worker too, or count the workers' threads together. The dashboard’s Time Vs Threads graph has the same per-server limitation, which the manual documents.

Source: Remote Testing: Step 3b (User Manual) and Getting Started: Server Mode (User Manual)

For a first check of the network and the keystore, the GUI is convenient. With remote_hosts set, the Run menu gains these entries:

Do not use the GUI controller for the measured run: GUI listeners on the controller then process every sample from every worker.

Source: Distributed Testing Step-by-step (User Manual)

Every RMI connection between controller and workers, the registry included, is a TLS connection. The server side requires a client certificate (mutual TLS has been enforced since 5.1), and both sides validate the peer against the truststore. With the shared keystore from create-rmi-keystore.sh, each node presents the same self-signed certificate and trusts it because it is in its own truststore. A node with another certificate is rejected.

The test run produced each failure mode in turn:

In each case the controller then prints Failed to configure host:port and stops with Following remote engines could not be configured, unless client.continue_on_fail is set. An expired certificate (after the seven days) shows up as another TLS handshake failure; create a new keystore and copy it again.

Source: Remote Testing: Setting up SSL (User Manual) and Changes History (4.0: SSL for RMI; 5.1: client authentication enabled)

Setting server.rmi.ssl.disable=true on every node (controller and workers) turns RMI back into plain TCP, which is how distributed testing worked before 4.0 and how the books describe it:

The log then shows Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true'. The cost is security, not performance. A JMeter server is, in the manual’s words, basically a remote execution agent: anyone who can reach port 1099 and the engine port of an unprotected worker can send it a plan with an OS Process Sampler or a Groovy script and run commands on that machine, with the worker’s user rights. Disable SSL only on isolated, disposable networks (a private lab subnet, containers on a private network that is torn down after the test), never on a shared or reachable network. Mixing settings (SSL on one side, off on the other) fails at connection time.

The manual’s section 13.7 additionally describes running JMeter under a Java security manager with a policy file. The Java Security Manager is deprecated for removal since Java 17 (JEP 411) and permanently disabled since Java 24 (JEP 486), so on current Java versions network isolation and TLS are the controls that remain.

Source: Remote Testing: Using a security-manager (User Manual), JEP 411 and JEP 486

"Stripped" means that elements on the controller that need response bodies, such as a View Results Tree used for debugging, show empty responses. Assertions and extractors are not affected, because they run on the workers where the full response is still available.

The manual says that mode is resolved on the controller and the other properties on the server. In 5.6.3 this is governed by sample_sender_client_configured, which defaults to true, and then the thresholds too are taken from the controller. The test run confirmed it: with -Jnum_sample_threshold=50 -Jtime_threshold=5000 on the controller only, each worker logged:

So in practice: set every sender property on the controller with -J (or in its user.properties). Set sample_sender_client_configured=false on the controller only if each worker should use its own num_sample_threshold, time_threshold and asynch.batch.queue.size.

Source: Remote Testing: Using a different sample sender (User Manual) and Properties Reference: Remote batching configuration

Both runs below used two workers, 5 threads each, a 500 ms think time and a 5 s summariser interval against the same local stub. Only the sender changed.

-Jmode=Standard:

Default (StrippedBatch), same plan and duration:

Both runs made 190 requests at about 15 to 16 req/s, and the final summary = lines agree. The difference is only in the controller’s live view: with batching, almost every sample arrived in the last second, so the interval line shows a meaningless 1,106.5/s. For long tests that is harmless (batches of 100 arrive continuously), but in short tests, or when watching the run, lower time_threshold (for example 5000) or use the Backend Listener for live monitoring, as described in Real-time Monitoring with Backend Listener.

Worked example: result traffic. At 625 req/s in total across four workers, Standard means 625 RMI calls per second into the controller, each carrying one sample, and every sampler thread waits for its call. StrippedBatch with the defaults means about 625 / 100 ≈ 6 calls per second, each with 100 stripped samples. That is why the default is batched, and why the manual warns that a controller with many workers can still become overloaded: check its CPU and network like a generator’s.

Source: Remote Testing: Using a different sample sender (User Manual)

Instead of a controller, run the same plan in CLI mode on each generator at the same time, collect the results files and merge them. This avoids RMI, keystores, callback ports and the controller bottleneck completely, and it is the usual way to run JMeter in containers. What it gives up: a single live summary, a single start command, and coordinated stop (each instance stops on its own schedule, or with its own shutdown.sh).

The steps:

A real run of two instances on one machine (15 s, 5 threads each):

and jmeter -g produced report/index.html and report/statistics.json with exit code 0. The merge only works when every instance writes the same columns, so use the same jmeter.save.saveservice.* settings everywhere. Percentiles must be computed from the merged raw file, never averaged from per-instance reports.

Source: Generating Report Dashboard (User Manual) and AbstractThreadGroup.java (5.6.3 source)

There is no official Apache JMeter Docker image and the JMeter manual says nothing about containers or Kubernetes; the patterns below are general container practice applied to JMeter, not an official recipe. Build your own image from the official binary, verified against its published SHA-512:

The checksum is the one published for 5.6.3 on the download page; plugins can be added in further RUN steps with the Plugins Manager command line, which keeps every generator identical by construction.

Independent instances as an Indexed Job. The simplest Kubernetes form of the previous section: one Job, N pods, each running the plan in CLI mode with its index, writing to a shared volume. Indexed Jobs set JOB_COMPLETION_INDEX in every pod.

Four pods with 500 threads each give 2,000 users, the worked example at the top of this page. Kubernetes expands $(JOB_COMPLETION_INDEX) in args because the Job controller adds that variable to each container’s environment. The results volume must support ReadWriteMany if the pods land on different nodes; after the Job completes, a short pod (or a step in CI) merges the files and runs jmeter -g as shown above. Equal CPU requests and limits keep generators comparable and avoid CPU throttling skewing timings; pod anti-affinity spreads them over nodes. The Kubernetes pages of this site cover the building blocks: Jobs and CronJobs (Indexed Jobs), ConfigMaps, Persistent Volumes and Storage Classes, Resource Management and Scheduling and Disruptions. To watch the application scale under the load, see Autoscaling.

RMI workers in Kubernetes. Classic controller and worker mode also works in a cluster if both ends live inside it, since pod IPs are directly routable between pods but not from a laptop outside the cluster. A StatefulSet with a headless Service gives workers stable DNS names (jmeter-worker-0.jmeter-worker.perf.svc.cluster.local and so on, see StatefulSets and Services), and the controller runs as a Job in the same namespace with -R listing those names. Each pod must advertise its own pod IP and use fixed ports:

The controller pod needs -Djava.rmi.server.hostname=$(POD_IP) too, and the keystore comes from a Secret mounted on every pod. If NetworkPolicies are in force, allow controller-to-worker traffic on 1099 and 4000 and worker-to-controller traffic on the controller’s ports (see Network Policies). For most container setups the independent-instance Job is simpler and scales better; choose RMI only when a single live summary on one controller is worth the extra moving parts.

For running the system under test and JMeter side by side on one machine, see Compose for Local Development and Docker Networking; remember that inside a container localhost is the container itself.

Source: Jobs: Completion mode (Kubernetes documentation), Downward API (Kubernetes documentation) and Download Apache JMeter

Several commercial and cloud services run JMeter plans on managed generators and merge the results, among them BlazeMeter, OctoPerf and Azure Load Testing. They take care of provisioning, distribution and reporting; check each service’s own documentation for the JMeter versions and plugins it supports and for how it splits threads and CSV files between engines. The JMeter Cookbook (ch. 5) shows recipes for Flood.io, Vagrant and Amazon EC2 from 2014; the services and tooling have changed since then, so use the recipes for the ideas only.