Performance Testing Fundamentals

A functional test asks whether the answer is right. A performance test asks how quickly, how often and for how many users at the same time the system answers, and what happens when it is pushed past what it was designed for. The result is not a pass or fail on one request but a set of measurements (see Performance criteria) compared with a target. Pro chapter 2 frames the same point as choosing a test type from the question you need answered, which is why the next section starts from the types.

Performance testing finds a different class of defect than unit tests: thread-pool exhaustion, connection-pool limits, lock contention, slow queries that only hurt under concurrency, memory leaks, cache stampedes, undersized instances and timeouts that cascade. It cannot prove the absence of defects, and a protocol-level tool such as JMeter does not measure what a browser feels. The JMeter manual states that JMeter is not a browser: it does not run JavaScript or render pages, so front-end performance (rendering, bundle size, Core Web Vitals) needs a browser-driven tool, covered by the Playwright, Cypress and Selenium section (tracked in issue #258).

The books use slightly different taxonomies (Pro chapter 2 lists the types and their goals; PTwJ chapter 1 contrasts load and stress testing). They agree on the five below, plus two that describe a goal rather than a shape.

The manual’s load-testing introduction suggests a similar escalation: start with a low-volume test, then a benchmark with average users, then a test at maximum users, then a destructive test to find the hard limit. Capacity and scalability tests are not different test plans; they are the same plan analysed differently, and the second needs the environment to be changed between runs.

The figure shows relative shapes only. A smoke test lasts minutes, a load test an hour, a stress test as long as it takes to reach the limit, and a soak test many hours.

Four families of measurements describe a run. Pro chapter 2 and PTwJ chapter 1 discuss them as the basis for acceptance criteria.

A fifth quantity is concurrency: how many users are active at once. It is both an input (how many threads you start) and an outcome (how many requests are in flight on the server), and Little’s Law ties it to throughput and response time.

Pro chapter 2 distinguishes requirements, which are mandatory limits, from goals, which are desirable targets. The distinction matters because it decides what a failed run means.

PTwJ chapter 1 recommends writing acceptance criteria before testing, because without them a test only produces numbers. A useful criterion names the load (users or rate), the duration, the measure (percentile, not just average), the threshold and the environment.

Criteria can be enforced during a JMeter run: a Duration Assertion fails a sample slower than a limit, and a Response Assertion checks the content. The assertions page of this section covers them. The dashboard’s APDEX table (see APDEX) and its percentile columns give the verdict afterwards.

PTwJ chapter 1 describes performance testing as a cycle of testing and tuning, and Pro chapter 2 expands it into a strategy document. The shape is the same in both: plan, build, check, run, analyse, change, repeat.

The steps in practice:

Two disciplines make the cycle trustworthy. First, repeatability: keep the plan, data set, environment and warm-up the same between runs and record the JMeter version, JVM and properties used. Second, run-to-run variance: repeat a run at least once before believing a small difference, because garbage collection, caches and noisy neighbours move the numbers.

Pro chapter 2 treats environment and data as part of the strategy. The points that matter most:

A workload model says who does what, how often and how many at a time. Pro chapter 11 describes deriving it from production data; the method is original to the books because the JMeter manual has no equivalent chapter.

An example for Bookshelf. Suppose the peak hour of an access log contains 36000 journeys, from which the analysis shows:

The mix is 36000 journeys per hour, which is 10 per second. The requests per second are 10 x (0.7 x 2 + 0.2 x 5 + 0.1 x 2) = 10 x 2.6 = 26 requests per second at peak. A test at the "expected peak" therefore needs a throughput of about 26 requests per second, not 26 users: the number of users depends on response and think time (see Little’s Law).

This is the most consequential modelling decision, and one the books do not draw because the JMeter of their time only had the first model.

JMeter supports both: the Thread Group and the Concurrency Thread Group plugin are closed (the second varies the number of active threads over time), the Open Model Thread Group and the Arrivals Thread Group plugin are open, and the Precise Throughput Timer and Throughput Shaping Timer bolt a target throughput onto a closed model. Many real systems mix the two: a closed population of users, each of whom arrives as part of an open stream. Choose by asking whether the rate or the population is the thing you know.

A schedule in the Open Model Thread Group is a sequence of commands. The manual lists rate(...), random_arrivals(...) and pause(...); the time units are ms, sec, min, hour and day, and the unit may be omitted for rate(0). The manual’s own examples are:

The first is a constant 10 requests per second for one minute; the second ramps from 0 to 10, holds for a minute and ramps back down; the third pauses for two seconds in the middle of a constant rate.

Source: Open Model Thread Group (Component Reference)

In JMeter 6.0 the delays of the Open Model Thread Group and the Precise Throughput Timer are planned to be measured from the Thread Group start (per changes.xml on master); that is planned behaviour, not released.

Little’s Law is a result from queueing theory: in a stable system, the average number of items inside the system equals the arrival rate multiplied by the average time each item spends in it. The JMeter manual does not state it. Applied to a closed-model test it becomes:

where \(N\) is the number of concurrent users (threads), \(X\) the throughput in requests per second, \(R\) the average response time in seconds and \(Z\) the average think time between requests in seconds. Turned around, a Thread Group of \(N\) users produces \(X = N / (R + Z)\). The law holds for averages in steady state; it says nothing about the distribution, and it does not hold during ramp-up, saturation or after errors change the cycle time.

The average (mean) response time hides the slow tail, and the slow tail is what users remember. A response time distribution is usually skewed: most requests are fast and a few are very slow.

A concrete example: 100 requests, of which 95 take 100 ms and 5 take 3000 ms.

Both the mean and the 95th percentile look fine, and only the 99th percentile shows the problem. The lesson is to report several percentiles together with the maximum and the error rate, and to choose the percentile that matches the requirement (95th or 99th for user-facing latency, 99.9th for strict services). Percentiles cannot be averaged across runs or across load generators; compute them from the raw samples.

The JMeter HTML dashboard reports three configurable percentiles. The manual gives their properties and defaults:

jmeter.reportgenerator.statistic_window sets the sliding-window size used to evaluate percentiles (default 20000, larger is more accurate and uses more memory). The manual also notes that dashboard percentiles can differ from the GUI Aggregate Report, and that setting the (unprefixed) property backend_metrics_percentile_estimator=R_3 brings them closer. Set these in user.properties or with -J on the command line, never by editing jmeter.properties.

Source: Generating Report Dashboard (User Manual)

APDEX (Application Performance Index) condenses response times into one number between 0 and 1. Each sample is classified as satisfied (fast enough), tolerating (slow but acceptable) or frustrated (too slow). The standard formula, from the APDEX specification that the JMeter manual links, is:

JMeter’s dashboard computes it per transaction. It has two thresholds: a sample at or under the satisfied threshold is satisfied, one at or under the tolerated threshold is tolerating, and a slower one is frustrated. The properties and their defaults are:

The first two lines state the documented defaults. The override format (sample_name:satisfaction|tolerance, separated by ;, values in milliseconds, names may be regular expressions) comes from the manual; the transaction names here are illustrative. The manual does not state the formula itself and refers to the APDEX article for it, which is why the formula above is attributed to the APDEX definition rather than quoted from the JMeter page.

Worked example with the defaults: of 100 samples, 80 take 500 ms or less, 15 take 501 to 1500 ms and 5 are slower. Apdex = (80 + 15 / 2) / 100 = 0.875. A rule of thumb used with APDEX is that values near 1 are good, but the scale and its qualitative labels are conventions, not JMeter features; thresholds should be derived from your own requirements (see Requirements, goals, SLAs and SLOs), not left at the defaults.

Source: Generating Report Dashboard (User Manual)

APDEX is a coarse summary. Use it as a headline next to percentiles, never instead of them.

Coordinated omission is a measurement bias that makes load tests look better than reality. It happens when a load generator that waits for a response before sending the next request (a closed model) stops generating load exactly while the system is slow. A stall of 10 seconds then contributes one slow sample instead of the ten seconds of requests that real users would have issued and experienced as slow, so the tail latency is understated.

The JMeter manual mentions the problem once, in its best-practices page, under the advice to use the correct number of threads: if you do not size the number of threads correctly you will face the "Coordinated Omission" problem, which can give wrong or inaccurate results. The page does not give a remedy beyond sizing the thread count and spreading very large tests over several machines in distributed mode. The practical measures below are consequences of the model description above, not statements by the manual:

Source: Best Practices (User Manual)

The best-practices page also covers other points relevant here: it advises GUI mode only for building and debugging, and notes that a very large total number of threads is possible provided that not too many are active concurrently because thread creation can be delayed until a thread starts sampling.

Each test type is a particular load shape, and JMeter offers an element for each. The Bookshelf plan bookshelf-load.jmx is reused for all of them; only the load driver changes. The figures are chosen to be easy to reason about and are worked through with Little’s Law.

The sections below show the minimum needed to configure each shape. The detailed fields of the thread groups, the open model and the timers have their own pages later in this section (thread groups and ramp-up, the open model and throughput control, timers and think time, custom thread groups and load shaping).