Analyzing Results & Finding Bottlenecks

Analysis is the step between a run and the next change. PTwJ chapter 1 describes it as examining the results of each run, identifying the bottleneck, applying a change, preferably one at a time, and rerunning the same test to compare. Pro chapter 2 adds that results are compared with the requirements (a missed requirement is a defect) and with the baseline (any deviation is a trend to investigate).

The order matters. Response times of a run with many errors, or of a run whose load generator was overloaded, describe the failure or the generator, not the system. Validity comes first, then errors, then throughput, and only then latency.

Every analysis starts from the per-sample results file written with -l in CLI mode. The manual recommends the CSV format for it and lists the columns, in a fixed order, that appear when the corresponding save settings are enabled. The columns that matter for analysis are:

The glossary adds two definitions that shape every interpretation: elapsed time does not include rendering or client JavaScript (JMeter is not a browser), and connect time is not subtracted from latency. Connect time is computed only for the HTTP Request, TCP and JDBC samplers.

The dashboard needs a minimum set of save settings, which are the defaults in 5.6.3. If a team has customised jmeter.save.saveservice.*, check them against the list in the dashboard chapter. A setting that is useful for analysis and off by default is the generator’s host name:

Source: CSV Log format (Listeners, User Manual)

Source: Glossary (User Manual)

Three derived artefacts are read alongside the raw file:

A typical end of a 50-thread Bookshelf load run in CLI mode:

The overall rate (14.1/s) is lower than the steady-state rate (15.4/s) because the total includes the 60 s ramp-up; judge steady state from the summary + lines of the hold phase or from a dashboard filtered to that window (see Excluding warm-up and ramp-down). The output above is illustrative; the format is the one documented for Generate Summary Results.

Source: Generate Summary Results (Component Reference)

Source: Generating Report Dashboard (User Manual)

Performance Testing Fundamentals shows with a 100-sample example why the mean hides the tail and why requirements should be stated as percentiles. This section covers how JMeter reports each statistic and how to read them together.

Errors come first because they invalidate everything else. A failed request is often fast (a connection refused takes milliseconds), so a run with errors can show better response times than a clean one. A rule that keeps analysis honest: if the error rate of a transaction is above its acceptance threshold, report the failure and do not draw conclusions from that transaction’s response times.

The most useful single picture of a system’s capacity is throughput and response time plotted against the number of concurrent users. It comes from a stress test that raises the load in steps and holds each step long enough to reach a steady state: for Bookshelf, steps of 25 users held for two minutes each, built with the Concurrency Thread Group described in Custom Thread Groups & Load Shaping.

The curve has three regions:

The saturation point is the knee. Where the two asymptotes of the figure cross, the light-load line \(N / (R_0 + Z)\) meets the capacity line \(X_{\max}\):

With the Bookshelf values (\(X_{\max} \approx 60\) requests per second, \(R_0 = 0.2\) s, \(Z = 3\) s) the knee is at 60 x 3.2 = 192 users. Real curves bend before the knee, as the figure shows; the formula gives the order of magnitude and a sanity check for the measurements.

Little’s Law, \(N = X \times (R + Z)\) for a closed model, is derived and worked through on Performance Testing Fundamentals. In analysis it is a cross-check: three measured quantities must agree with the number of threads that were running. A mismatch means the test did not do what it was designed to do.

Performance Testing Fundamentals explains the mechanism and quotes the manual’s single warning about it. In analysis, the question is how to recognise it in a results file and how much it distorts the numbers.

A worked case: during the Bookshelf load test the server stalls for 10 s (a full garbage collection, a database failover). With the closed 50-thread plan, every thread that sends a request during the stall waits for it, and within one cycle time (3.2 s) all 50 are stuck. The results file records about 50 slow samples, one per thread, and almost no samples at all for the rest of the stall. Real users arriving at 15.6 requests per second would have issued about 156 requests during those 10 s, each delayed by between 0 and 10 s. The closed run therefore under-represents the stall roughly threefold in its sample count, and its percentiles are better than what an open stream of users would have experienced.

How to recognise it:

What to do with it: report the percentiles of a closed run as the experience of the modelled user population, not of all arriving traffic; rerun the scenario with an open model when the real traffic is open; and treat every throughput dip as a finding to explain, never as noise.

Source: Best Practices (User Manual)

A single run says little; the comparison with a baseline says whether a change helped. Pro chapter 2 describes comparing every report with the baseline to note trends, and PTwJ chapter 1 insists on changing one thing at a time so that each improvement can be attributed.

JMeter measures response time on the client. If the client is overloaded, threads wait for CPU before sending and after receiving, and that wait is counted as response time. The manual’s best-practices page warns that the number of threads must suit the hardware and the plan, and lists ways to reduce the resources a plan needs. Check the generator in every run, and keep its metrics in the report.

Symptoms of a saturated generator:

Collect the generator’s GC log and watch its CPU and GC while the test runs. JVM_ARGS is passed to the JMeter JVM by the start script; the GC options are standard JDK unified logging, and jcmd and jstat are standard JDK tools:

In the jstat output, an old-generation occupancy (O) that stays near 100 percent and a full-GC count (FGC) that keeps rising mean the heap is too small for the plan, usually because of listeners or large responses kept in memory. Operating-system tools (top, vmstat, sar) give CPU and network usage.

A decisive test: run the same total load from two generators with half the threads each. If response times improve, the single generator was the bottleneck. How to size and tune generators, and the resource-saving measures from the best-practices page, are covered on the best practices and load generator tuning page later in this section; how many threads one machine holds is measured on Thread Groups: Concurrent Users & Ramp-up.

Source: Best Practices (User Manual)

Source: Getting Started (User Manual)

JMeter sees only the client side. To find out why response time rose, put the server’s metrics on the same time axis. PTwJ chapter 5 lists what monitoring during the test should reveal: long-running queries, insufficient thread and data source pools, an insufficient heap, high I/O, capacity shortfalls and CPU usage. Cookbook chapter 8 makes the same case for CPU, memory, threads, thread pools and I/O. The tools in those books (the Monitor Results listener, PerfMon with its ServerAgent, VisualVM) are dated or removed; the current approach is to scrape application metrics with Prometheus and view them in Grafana, next to JMeter’s own live metrics from the Backend Listener.

The setup is documented elsewhere and not repeated here:

The books group bottlenecks into three families. PTwJ chapter 1: system-related (memory, CPU, thread pool and database pool sizes, network settings), database-related (I/O, top queries, SQL profiling, missing indexes, statistics, page sizes and locks) and application-related (component design, memory consumption, database round trips). Cookbook chapter 4 adds expensive joins, missing indexes and slow queries as reasons to test the database directly. The checklist below orders the common causes by how often they explain a knee in a typical Java service, with the evidence that confirms each one.

PTwJ chapter 1 also stresses that analysis is collaborative: developers, DBAs and system administrators each read the part of the evidence they own. Hand them the time window, the transaction and the metric that moved.

A report is read by people who did not watch the test. It must state the verdict first and contain enough context to reproduce the run. Pro chapter 2 describes reports as the input for defects (missed requirements) and for trend analysis against the baseline; Cookbook chapter 7 covers presenting results with graphs. A structure that works:

The central table, for the Bookshelf load test at 50 users (illustrative values; the criteria are those set on Performance Testing Fundamentals):

Keep the raw results file, the dashboard folder and the properties with the report. A later run can only be compared with this one if the inputs are kept.

An automated pipeline needs a machine-readable verdict. Do not rely on the exit status of the jmeter command as the verdict on performance: evaluate the results. Four mechanisms, from the earliest to the latest in the run:

Performance gates in CI need a stable environment. On shared CI runners the run-to-run variance is large, so gate on generous limits (errors, gross regressions) in CI and keep tight percentile criteria for the dedicated performance environment.