Build Performance: Cache, Parallel Builds & mvnd

The reactor graph decides what can run in parallel: a project starts as soon as every project it depends on has finished, and a free thread is available. In harbor the parent comes first, then the core, and after that the six quays are independent. The Maven guide on parallel builds describes the graph in levels: level 0 first, then a fan-out of everything whose dependencies are done. A thread count larger than the width of the graph buys nothing, and a count smaller than the width forces the projects to run in waves.

On Maven 3.10.0, mvn -T 1C verify printed the marker line that confirms that the threaded builder is in use, and the Building lines show the starts. The order of the start lines is no longer the reactor order, the [n/m] numbers are the positions in the sorted reactor and not the order of completion, and the test lines of different modules are interleaved:

(Lines in between were left out where the ellipsis is.) The line of the total time ends with (Wall Clock) in the threaded builder because the sum of the project times would be larger than the elapsed time. With -T 4 only four quays can be in progress, so the last two start when threads become free, and the build takes about two waves of two seconds:

Source: Parallel builds in Maven 3

Each cell is one run of mvn -B -o clean verify, with the mode in the first column, on a quiet 10-core machine with a warm local repository. Total time is the figure that Maven prints; the spread of three runs shows how noisy the figure is on a laptop.

The speed-up is real here (from about 18 s to about 5 s) because the shape is ideal: six independent modules and a slow test in each. A different shape gives a different result. In the tidepool build of the reactor page, a chain of three modules with fast tests took 3.373 s sequentially and 4.293 s with -T 1C, because starting the threads cost more than the one independent module saved. The Maven wiki says that a 20 to 50 percent improvement is common, which is a more usual figure than the factor of three above, and that putting all tests into one module makes parallel builds less useful, unless the tests themselves run in parallel (the Surefire page covers parallel test execution and forkCount). The rules of thumb that follow from the graph are:

Typing -T 1C every time is not necessary. Maven reads default arguments from several places, and all of them worked on harbor:

The details of these files, and what happens when the same option is set twice, are on Configuring Maven. Committing .mvn/maven.config with -T1C makes every developer and the CI server build in parallel, which is convenient and also forces every plugin of the build to be thread-safe, so do it only after a clean parallel run.

Two kinds of code run concurrently in a parallel build: Maven core, which is thread safe and well behaved according to the Maven wiki, and the plugins. A Mojo that is safe to run in several modules at once says so with @ThreadSafe on its class (the annotation is covered on Writing Plugins); Maven 3 was meant to warn about plugins that are not marked. I tried maven-antrun-plugin 1.8 with -T 2 on Maven 3.10.0, 3.9.16 and 4.0.0-rc-7 and did not get a warning from any of them, which is consistent with the wiki list that names the Antrun plugin as thread safe since 1.4. I did not run a build with a plugin that is really not thread safe, so I have no output for the warning or for a failure, and I do not claim that Maven prints one.

What the wiki does say, and what matters in practice:

When a build only fails with -T, the first test is -T 1: if it passes, the problem is a race. The second is to bisect with -pl module -am (see the reactor page) to find which two modules collide.

A separate setting controls how many artifacts Maven downloads at once, and it is independent of -T. By default Maven downloads up to five artifacts from different groups at the same time, and -Dmaven.artifact.threads changes the size of the pool (for example mvn -Dmaven.artifact.threads=1 verify). It matters on the first build with an empty local repository and not after that.

Source: Configuring parallel artifact resolution

The cache is key-based. For each module the extension computes a checksum from the content of its configured source files, its dependencies (including the checksums of the modules it depends on in the reactor), and the effective POM, including plugin parameters. If a build with that key is in the cache, the extension restores the output (classes, JARs, attached artifacts) and marks the plugin executions that produced it as "skipped (cached)". If it is not, the module is built as usual and its result is saved. Two consequences are important: a change in a module changes the key of every module that depends on it, and a change of the build configuration changes the key even when the sources are identical. The hash is computed from the raw bytes of the files, without any normalization of line endings.

The extension is declared either in the <build><extensions> of the POM or, preferably according to the documentation because it gives the extension better access to the Maven API, in .mvn/extensions.xml:

Core extensions in general, and what else can go into this file, are on Extensions. Nothing else is required: the extension works with a built-in default configuration. The first build with it printed that fact, and the next two lines are the whole activation message:

Source: Build Cache Extension: Declaring the extension

The first mvn verify (no clean, an empty cache) found no cache entry for the core, built it, and saved the result. The line that says where the cache entry is also shows the layout: the cache lives in a build-cache directory next to the local repository, with one directory per module and one per key.

On the JDK used here, the first build also printed four WARNING lines from the hashing library of the extension about sun.misc.Unsafe::arrayBaseOffset being terminally deprecated (JDK 26). They are harmless, and they come from the library, not from the build.

The second mvn verify, with nothing changed, restored every module. The log of one module says it all: the key was found, and every plugin execution that produced the output was skipped. The test did not run, which is why the build took no time:

Then I changed one line of harbor-quay-3. Only that module got a new key, so only that module was built (and its test ran); the other five quays and the core were restored. The whole build took 3.710 s instead of 15.831 s:

Finally I changed the core, which every quay depends on. All seven modules got new keys, because a dependency’s checksum is part of the key of its dependents, and the six tests ran (here with -T 1C, so the build took 4.330 s). A cache does not help a change at the bottom of the graph, only a change near the top.

Cache and parallel builds work together. The same -T 1C build with a warm cache took 0.795 s. The cache directory (112 KB for this project) holds one buildinfo.xml per entry plus the saved artifacts. buildinfo.xml is worth opening when a hit is not what you expect: it lists the goals that were run, the artifact with its hash, and for every plugin execution the properties that the extension recorded and whether each one is tracked (compared when deciding whether the entry can be reused).

(Hash and size elements and most properties were left out.) With Maven 3.9.16 and 4.0.0-rc-7 the same extension and the same configuration gave the same behavior: a first build of about 4 s with -T 1C and a second build of 1.169 s on 3.9.16 and 0.854 s on 4.0.0-rc-7 (0.473 s on 3.10.0), and the cache directory has the same v1.2 layout.

Source: Build Cache Extension: Usage

The cache key contains the effective POM, but the plugin parameters that you pass on the command line are not part of it unless the configuration says so. With the default configuration, this happened. I started with an empty cache and built with tests skipped; the cache stored that build. Then I built with tests enabled:

The second command, which asked for tests, reported success without running a single one: the entry was written by a build in which the tests were skipped, and it had the same key. This is the classic way to get a green build that never tested anything, and the documentation lists it among the common problems (the how-to has an entry for "I occasionally cached a build with -DskipTests=true, and tests do not run now"). The fix is to tell the extension which plugin parameters change the meaning of a cache entry. This is called reconciliation, and it is configured in maven-build-cache-config.xml, described next.

Source: Build Cache Extension: How-To

The optional file .mvn/maven-build-cache-config.xml customizes the cache. The documentation recommends starting from the template and doing three things: exclude unstable or environment-specific files, add critical plugin parameters to the runtime reconciliation, and adjust the source selectors for unusual layouts. The file used for the rest of the experiments is short:

The elements, in the order of the file:

With this file in place, the same two commands behaved correctly. The first one stored an entry with skipTests true. The second one found the entry by key, compared the tracked parameter, saw the mismatch, and built the module for real:

The same file also made mvn install work with the cache: after a restore the install:3.2.0:install execution still ran and copied the JAR into the (scratch) local repository, which is what runAlways is for. The other direction, a build that ran the tests being reused for -DskipTests, is harmless, and happened with the default configuration too.

Source: Build Cache Extension: Adding build cache config

The behavior is controlled from the command line as well, which is how CI jobs differ from developer machines. These are the flags that matter, from the parameters page; the ones marked "ran" were used in the experiments above:

There are also per-project properties (maven.build.cache.input.glob, maven.build.cache.exclude.xxx, maven.build.cache.skipCache, maven.build.cache.skipSave and others) that can be set as POM properties to override the global settings for one module. A cache that is wrong for a module is disabled for that module, not for the build.

Source: Build Cache Extension: Cache parameters

The performance page of the extension suggests four things when the cache itself is slow: a faster hash algorithm (the default is already the fastest), filtering huge artifacts (WARs, EARs, ZIPs) out of the cache with <output><exclude> patterns because assembling them locally is cheaper than storing and downloading them, lazy restore, and switching off the post-processing of META-INF in archives with <projectVersioning adjustMetaInf="false"/> when the manifest version does not matter.

In an IDE, the cache does not restore the whole project state: compiled classes and unpacked artifacts are not necessarily in target. The documentation recommends letting the IDE compile with its own output directories and using the cache for command-line builds.

Source: Build Cache Extension: Performance tuning

A local cache helps one machine. The remote cache shares entries between the CI server and the developers, so that a developer’s first build after pulling the main branch finds the entries that CI already built. This part was not run here: it needs shared storage, and no repository manager or web server with PUT support was installed, and I did not set one up. The following is taken from the documentation.

Source: Build Cache Extension: Remote cache setup

The remote cache page lists the five problems that account for most cache misses. They apply to the local cache too, as soon as two environments (a laptop and CI, Windows and Linux) share entries or one machine is configured differently.

Unlike a build tool that tracks task inputs and outputs by design, Maven plugins do not declare their inputs and outputs, so the extension has to guess: it introspects plugin configuration to find input directories. A plugin that reads a file the extension does not know about (a code generator reading a model from outside src/) does not make the key change, and a stale entry is restored. That is what the <input><plugins> section of the configuration, and the maven.build.cache.input project properties, are for. It is also why the documentation says to validate results iteratively, looking at buildinfo.xml, before trusting the cache on a release build. Nothing replaces an occasional -Dmaven.build.cache.skipCache=true build on CI to check that a clean rebuild gives the same result.

Source: Build Cache Extension: Common issues