Maven Integration Testing with Failsafe

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

This page is written against Apache Maven 3.10.0 with the Apache Maven Failsafe Plugin 3.6.0. Every example was run on JDK 26 on 2026-10-16 in a scratch copy of the tidepool project (a parent POM and the tidepool-model, tidepool-engine and tidepool-cli modules, with every build plugin pinned in pluginManagement). The same builds were also run on Maven 3.9.16 and on Maven 4.0.0-rc-7 (a release candidate, not a final release), and the differences are stated. Paths in the console output were shortened to /work/failsafe/..., the Loaded N auto-discovered prefixes and download lines were left out, and long output was filtered with grep and says so. Where an example was not run (a container, a real application server) the text says "not run here".

Unit tests run in the test phase, before the project is packaged, and they must not need anything outside the JVM. Integration tests check the packaged result: the JAR started as a process, a web application deployed to a container, a service talking to a database. They need the artifact, often need something started before and stopped after them, and they are slower. Maven keeps them apart with a second plugin, Failsafe, bound to later phases. This page covers that plugin. Unit-test configuration (JUnit Platform, includes and excludes, -Dtest, tags, forking, parallelism) is the same for Failsafe and is not repeated: see Testing with Surefire. The phases themselves are described in Maven Build Lifecycle.

Why a second plugin

Surefire runs in the test phase and fails the build as soon as a test fails. That is right for unit tests and wrong for integration tests, because the build would stop before the cleanup that has to follow them: a server started in pre-integration-test would be left running. Failsafe therefore splits its work in two goals:

Phase Goal What it does

pre-integration-test

(your plugins)

Prepare the environment: start a server or a container, create a database, reserve a port.

integration-test

failsafe:integration-test

Runs the tests and writes their results, including a summary file. A test failure does not fail this goal.

post-integration-test

(your plugins)

Tear the environment down. It runs whether or not the tests passed.

verify

failsafe:verify

Reads the summary file and fails the build if any test failed or errored.

Surefire and Failsafe come from the same project, share the test provider and most parameters, and differ in naming, phase and failure handling. Since Surefire 3.6.0 every test, JUnit 4, JUnit Jupiter or TestNG, runs through the JUnit Platform: the engine is chosen from the test dependencies, so the plugin needs no extra configuration.

graph LR PKG["package: jar:jar builds the artifact"] --> PRE["pre-integration-test: start the server"] PRE --> IT["integration-test: failsafe:integration-test runs the IT classes"] IT -->|"failures are only recorded"| POST["post-integration-test: stop the server"] POST --> VER["verify: failsafe:verify fails the build if the summary says so"]

Source: Failsafe plugin: Usage

The example project

The integration tests live in tidepool-cli, the module whose artifact is the command-line program. The layout is the usual Maven one; the integration tests sit next to the unit tests of the other modules, in src/test/java:

tidepool/
  pom.xml
  data/stations.csv
  tidepool-model/
  tidepool-engine/
    src/test/java/io/tidepool/engine/StationReaderTest.java   unit test (Surefire)
  tidepool-cli/
    pom.xml
    src/main/java/io/tidepool/cli/Main.java
    src/test/java/io/tidepool/cli/MainIT.java                 integration tests (Failsafe)
    src/test/java/io/tidepool/cli/UnknownStationIT.java
    src/test/java/io/tidepool/cli/StationServerIT.java

MainIT starts the program as a separate JVM and checks its output; the other two follow later. The program is run with the test classpath that Failsafe built, which contains the packaged tidepool-cli JAR:

package io.tidepool.cli;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;

import java.nio.file.Path;
import org.junit.jupiter.api.Test;

class MainIT {
    @Test
    void predictsForBoston() throws Exception {
        System.out.println("Main loaded from " + Path.of(Main.class.getProtectionDomain().getCodeSource().getLocation().toURI()).getFileName());
        var java = Path.of(System.getProperty("java.home"), "bin", "java").toString();
        var process = new ProcessBuilder(java, "-cp", System.getProperty("java.class.path"),
                "io.tidepool.cli.Main", System.getProperty("stations.file"), "BOS")
                .redirectErrorStream(true).start();
        var output = new String(process.getInputStream().readAllBytes()).trim();
        assertEquals(0, process.waitFor(), output);
        assertTrue(output.startsWith("Boston: "), output);
    }
}

This test class is this page’s own lab setup, not an example from the Maven documentation.

UnknownStationIT runs the same command for a station that does not exist and expects exit code 1 and the message Unknown station: XXX. StationReaderTest in tidepool-engine is an ordinary unit test with one method. The two modules need JUnit Jupiter as a test dependency, managed in the parent POM (see Dependency Management & BOMs).

Pinning the plugin does not run it

The parent POM pins Failsafe 3.6.0 in pluginManagement, and nothing else. Failsafe, unlike Surefire, is not part of the default lifecycle bindings of the jar packaging, so declaring the version is not enough. With only the managed version, mvn clean verify runs the unit test and silently ignores MainIT; this is the cut-down output (grep on the plugin lines):

[INFO] --- surefire:3.6.0:test (default-test) @ tidepool-model ---
[INFO] No tests to run.
[INFO] --- surefire:3.6.0:test (default-test) @ tidepool-engine ---
[INFO] Running io.tidepool.engine.StationReaderTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.026 s -- in io.tidepool.engine.StationReaderTest
[INFO] --- surefire:3.6.0:test (default-test) @ tidepool-cli ---
[INFO] --- jar:3.5.1:jar (default-jar) @ tidepool-cli ---
[INFO] BUILD SUCCESS

In tidepool-cli Surefire prints nothing at all: its default includes (**/Test*.java, **/*Test.java, **/*Tests.java and **/*TestCase.java) match none of the integration-test classes (only MainIT existed at that point), so there is no Running line, and no Failsafe line follows. A green build that did not run the integration tests is the classic mistake with this plugin.

Source: Failsafe plugin: Usage

Binding the goals

Binding is done by listing both goals in an <execution>. The Failsafe usage page recommends giving the execution an id when the declaration is shared in a parent POM, so that a child can override it. The tidepool parent therefore holds:

<pluginManagement>
  <plugins>
    <plugin>
      <artifactId>maven-failsafe-plugin</artifactId>
      <version>3.6.0</version>
      <executions>
        <execution>
          <id>integration-test</id>
          <goals>
            <goal>integration-test</goal>
            <goal>verify</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</pluginManagement>

and a module opts in by naming the plugin, here with the system properties its tests read:

<plugin>
  <artifactId>maven-failsafe-plugin</artifactId>
  <configuration>
    <systemPropertyVariables>
      <stations.file>${project.basedir}/../data/stations.csv</stations.file>
      <server.port>${server.port}</server.port>
    </systemPropertyVariables>
  </configuration>
</plugin>

Neither goal declares a phase; each has a default one (integration-test and verify, as help:describe prints for both goals), so the build log shows the two executions where the lifecycle puts them. The execution id appears in parentheses in the log, failsafe:3.6.0:integration-test (integration-test); an unnamed execution is called default, as the first run before the id was added showed (failsafe:3.6.0:integration-test (default)). A run from the command line uses the id default-cli.

Source: Failsafe plugin: Usage in multi-module projects

Source: Failsafe: integration-test, systemPropertyVariables

Running it: verify and the JAR

mvn clean verify, from the project root on Maven 3.10.0, with the server steps of the next section already in place (the antrun lines are explained there). The output for tidepool-cli, with the resource and compile lines filtered out:

[INFO] --- build-helper:3.6.2:reserve-network-port (reserve-server-port) @ tidepool-cli ---
[INFO] Reserved port 53394 for server.port
[INFO] --- surefire:3.6.0:test (default-test) @ tidepool-cli ---
[INFO] --- jar:3.5.1:jar (default-jar) @ tidepool-cli ---
[INFO] Building jar: /work/failsafe/tidepool-cli/target/tidepool-cli-1.0.0-SNAPSHOT.jar
[INFO] --- antrun:3.2.0:run (start-server) @ tidepool-cli ---
[WARNING]      [echo] Test server is listening on port 53394
[INFO] --- failsafe:3.6.0:integration-test (integration-test) @ tidepool-cli ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]  T E S T S
[INFO] Running io.tidepool.cli.StationServerIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.208 s -- in io.tidepool.cli.StationServerIT
[INFO] Running io.tidepool.cli.UnknownStationIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.143 s -- in io.tidepool.cli.UnknownStationIT
[INFO] Running io.tidepool.cli.MainIT
Main loaded from tidepool-cli-1.0.0-SNAPSHOT.jar
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.154 s -- in io.tidepool.cli.MainIT
[INFO] Results:
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] --- antrun:3.2.0:run (stop-server) @ tidepool-cli ---
[WARNING]      [echo] Test server on port 53394 stopped
[INFO] --- failsafe:3.6.0:verify (integration-test) @ tidepool-cli ---
[INFO] BUILD SUCCESS

Three things to read from it:

  • The order is package, then the pre-integration step, then the tests, then the post-integration step, then verify. Failsafe needs the packaged artifact, so a mvn test or mvn compile never reaches it.

  • The line Main loaded from tidepool-cli-1.0.0-SNAPSHOT.jar is printed by the test itself: the class under test came from the JAR, not from target/classes. The classesDirectory parameter of Failsafe defaults to the built artifact when there is one (the parameter text from help:describe: "Defaults to built artifact JAR file or ${project.build.outputDirectory}"). This is the point of the plugin: tests exercise what would be published, including the manifest and the resources that made it into the JAR.

  • The tests ran in file-system order (StationServerIT, UnknownStationIT, MainIT), not alphabetically. Do not rely on an order (runOrder can change it).

Running a Failsafe goal directly changes the last point. mvn clean test-compile failsafe:integration-test failsafe:verify -Dit.test=MainIT (with -Dfailsafe.failIfNoSpecifiedTests=false, so the modules without ITs do not fail) printed Main loaded from classes, because no JAR had been built, and the execution was failsafe:3.6.0:integration-test (default-cli). Direct calls are handy while writing a test, but they are not the same test as the one verify runs.

Source: Failsafe: integration-test, classesDirectory

Naming: which classes are integration tests

Failsafe selects test classes by name, as Surefire does, but with its own patterns. The default includes of failsafe:integration-test, copied from help:describe -Ddetail, are:

**/IT*.java  **/*IT.java  **/*ITCase.java

The default excludes are **/*$*, which leaves out inner classes. The two plugins therefore partition the test sources between them: StationReaderTest is a unit test (Surefire), MainIT is an integration test (Failsafe), and neither runs the other’s classes. Both includes and excludes can be replaced in the plugin configuration, or on the command line as -Dfailsafe.includes and -Dfailsafe.excludes, with the same pattern syntax as Surefire (comma-separated lists, %regex[...], ! negation). A class called MainIntegrationTest would end in Test, so Surefire would run it in the test phase, before the JAR exists and before any server is up; name it MainIT, or change both plugins' patterns.

Source: Failsafe plugin: Inclusions and exclusions

Why two goals: the failing test

A failing integration test shows what the split is for. UnknownStationIT was changed to expect exit code 0 (the program exits with 1), and then mvn integration-test -pl tidepool-cli -am was run. (The failure runs in this page were made before the execution id was added, so their log lines say default instead of integration-test.) The build succeeded and Maven exited with status 0:

[INFO] --- failsafe:3.6.0:integration-test (default) @ tidepool-cli ---
[INFO] Running io.tidepool.cli.UnknownStationIT
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.171 s <<< FAILURE! -- in io.tidepool.cli.UnknownStationIT
[ERROR] io.tidepool.cli.UnknownStationIT.unknownStationExitsWithOne -- Time elapsed: 0.161 s <<< FAILURE!
org.opentest4j.AssertionFailedError: Unknown station: XXX ==> expected: <0> but was: (1)
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:601)
	at io.tidepool.cli.UnknownStationIT.unknownStationExitsWithOne(UnknownStationIT.java:17)

[INFO] Running io.tidepool.cli.MainIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.163 s -- in io.tidepool.cli.MainIT
[INFO] Results:
[ERROR] Failures:
[ERROR]   UnknownStationIT.unknownStationExitsWithOne:17 Unknown station: XXX ==> expected: <0> but was: (1)
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
[INFO] Reactor Summary for tidepool-parent 1.0.0-SNAPSHOT:
[INFO] tidepool-cli ....................................... SUCCESS [  1.656 s]
[INFO] BUILD SUCCESS

The failure is printed in red and recorded, and the build is green. The same project with mvn verify reached failsafe:verify, which reads target/failsafe-reports/failsafe-summary.xml and stops the build, with exit status 1:

[INFO] --- failsafe:3.6.0:verify (default) @ tidepool-cli ---
[ERROR] tidepool-cli ....................................... FAILURE [  0.882 s]
[ERROR] BUILD FAILURE
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:3.6.0:verify (default) on project tidepool-cli: There are test failures.
[ERROR]
[ERROR] See /work/failsafe/tidepool-cli/target/failsafe-reports for the individual test results.
[ERROR] See dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :tidepool-cli

The consequences are practical. A pipeline step that runs mvn integration-test reports success whatever the tests do: run verify (or anything after it, install, deploy) and never the phase integration-test alone. The same warning applies to the pre- and post- phases; the Failsafe documentation says not to invoke them directly, because they are long to type and may leave a container running. And a POM that lists only the integration-test goal and forgets verify passes with failing tests, again with no error.

Source: Failsafe plugin: Usage

The summary file and the reports

Every integration-test execution writes, in target/failsafe-reports, one TEST-<class>.xml and one <class>.txt per test class, and a failsafe-summary.xml. In the failing run above the directory held:

TEST-io.tidepool.cli.MainIT.xml
TEST-io.tidepool.cli.UnknownStationIT.xml
failsafe-summary.xml
io.tidepool.cli.MainIT.txt
io.tidepool.cli.UnknownStationIT.txt

and the summary, the only thing failsafe:verify reads, was:

<failsafe-summary xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-failsafe-plugin/xsd/failsafe-summary.xsd" result="255" timeout="false">
    <completed>2</completed>
    <errors>0</errors>
    <failures>1</failures>
    <skipped>0</skipped>
    <flakes>0</flakes>
    <failureMessage xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>
</failsafe-summary>

The format is the Surefire one, so a CI server that understands the Surefire XML can pick the TEST-*.xml files up with the glob **/failsafe-reports/TEST-*.xml. A readable HTML report comes from the Surefire Report Plugin, whose failsafe-report-only goal reads that directory. Run on the passing build, mvn org.apache.maven.plugins:maven-surefire-report-plugin:3.6.0:failsafe-report-only -pl tidepool-cli printed

[INFO] --- surefire-report:3.6.0:failsafe-report-only (default-cli) @ tidepool-cli ---
[INFO] Rendering content with org.apache.maven.skins:maven-fluido-skin:jar:2.0.0-M9 skin
[WARNING] Unable to locate Test Source XRef to link to -- DISABLED
[INFO] BUILD SUCCESS

and wrote tidepool-cli/target/reports/failsafe.html. In a project site the Failsafe documentation adds the same goal as a second report set in the <reporting> section (<report>failsafe-report-only</report>, with the report set id integration-tests) next to the unit-test one; see Site & Reporting.

Source: Failsafe plugin: Reporting integration test results

A server around the tests

The documentation’s own example starts Jetty in pre-integration-test, stops it in post-integration-test and lets Failsafe test it. The pattern is general: any plugin goal bound to pre-integration-test can prepare the environment, and the one in post-integration-test always runs, because the failing integration-test goal does not stop the build. The tidepool version has no web server of its own, so it serves the station data with jwebserver, the simple web server shipped with the JDK since JDK 18 (JEP 408), on a port reserved for the build. Three plugins cooperate:

  1. build-helper:reserve-network-port (MojoHaus 3.6.2, third party) picks a free port in initialize and stores it in the project property server.port. A fixed port such as 8080 would break when two builds run on the same machine.

  2. antrun:run (Apache Maven AntRun Plugin 3.2.0) starts jwebserver in pre-integration-test, waits until the port accepts connections, and stops it in post-integration-test.

  3. Failsafe passes server.port to the tests as a system property.

The relevant part of tidepool-cli/pom.xml:

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>build-helper-maven-plugin</artifactId>
  <version>3.6.2</version>
  <executions>
    <execution>
      <id>reserve-server-port</id>
      <phase>initialize</phase>
      <goals><goal>reserve-network-port</goal></goals>
      <configuration>
        <portNames><portName>server.port</portName></portNames>
      </configuration>
    </execution>
  </executions>
</plugin>
<plugin>
  <artifactId>maven-antrun-plugin</artifactId>
  <version>3.2.0</version>
  <executions>
    <execution>
      <id>start-server</id>
      <phase>pre-integration-test</phase>
      <goals><goal>run</goal></goals>
      <configuration>
        <target>
          <exec executable="sh" spawn="true">
            <arg value="-c"/>
            <arg value="echo $$$$ &gt; '${project.build.directory}/server.pid'; exec '${java.home}/bin/jwebserver' -b 127.0.0.1 -p ${server.port} -d '${project.basedir}/../data' &gt; '${project.build.directory}/server.log' 2&gt;&amp;1"/>
          </exec>
          <waitfor maxwait="10" maxwaitunit="second" timeoutproperty="server.timeout">
            <socket server="127.0.0.1" port="${server.port}"/>
          </waitfor>
          <fail if="server.timeout" message="The test server did not start"/>
          <echo message="Test server is listening on port ${server.port}"/>
        </target>
      </configuration>
    </execution>
    <execution>
      <id>stop-server</id>
      <phase>post-integration-test</phase>
      <goals><goal>run</goal></goals>
      <configuration>
        <target>
          <exec executable="sh">
            <arg value="-c"/>
            <arg value="kill $(cat '${project.build.directory}/server.pid')"/>
          </exec>
          <echo message="Test server on port ${server.port} stopped"/>
        </target>
      </configuration>
    </execution>
  </executions>
</plugin>

The test that uses it reads server.port and parses the station file served over HTTP with the production reader:

class StationServerIT {
    @Test
    void servesThreeStations() throws Exception {
        var url = URI.create("http://127.0.0.1:" + System.getProperty("server.port") + "/stations.csv");
        var response = HttpClient.newHttpClient()
                .send(HttpRequest.newBuilder(url).build(), HttpResponse.BodyHandlers.ofString());
        assertEquals(200, response.statusCode());
        assertEquals(3, StationReader.read(new StringReader(response.body())).size());
    }
}

Executions of different plugins bound to one phase run in the order the plugins are declared in the POM, and executions of the same plugin in the order they are listed. A <waitfor> after the start is what keeps the first test from racing the server. jwebserver is bound to the loopback address with -b 127.0.0.1, so the test server is not reachable from the network.

A bug found while writing this example, and a good illustration of why the output must be read: the first version wrote echo $$ > server.pid, and mvn verify printed Test server on port 52018 stopped while a jwebserver process was still running. Ant treats $$ as an escape for a single $, so the shell received echo $ and the pid file contained a lone dollar sign. The stop-server execution printed

[INFO]      [exec] sh: line 0: kill: $: arguments must be process or job IDs
[INFO]      [exec] Result: 1
[WARNING]      [echo] Test server on port 52018 stopped

followed by a build that was still green, because <exec> does not fail on a non-zero result unless failonerror is set and the <echo> after it ran regardless. Writing $$$$ in the POM (the shell sees $$) fixed it, and adding failonerror="true" to the stopping <exec> makes a future mismatch visible. A process left behind this way keeps the port, which is another reason to reserve a new one each time.

To check that the cleanup really happens when tests fail, a throwaway BrokenIT that calls fail(...) was added next to the others. The relevant lines of mvn verify:

[INFO] --- antrun:3.2.0:run (start-server) @ tidepool-cli ---
[WARNING]      [echo] Test server is listening on port 52154
[INFO] --- failsafe:3.6.0:integration-test (default) @ tidepool-cli ---
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.005 s <<< FAILURE! -- in io.tidepool.cli.BrokenIT
[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0
[INFO] --- antrun:3.2.0:run (stop-server) @ tidepool-cli ---
[WARNING]      [echo] Test server on port 52154 stopped
[INFO] --- failsafe:3.6.0:verify (default) @ tidepool-cli ---
[ERROR] BUILD FAILURE

The server was stopped before verify failed the build, and no jwebserver process remained a couple of seconds later (it takes a moment to exit after kill). BrokenIT was removed afterwards.

Other ways to get the same environment, none of which was run here:

  • The Jetty Maven plugin, as in the Failsafe documentation (start with daemon set to true in pre-integration-test, stop in post-integration-test); its version in that page is old, so check the current one.

  • The Fabric8 docker-maven-plugin binds container start and stop goals to the same two phases; see its documentation.

  • Testcontainers starts containers from the test code instead, with no pre- and post- bindings, which removes the leftover-process problem at the cost of needing Docker where the tests run.

Source: Failsafe plugin: Using jetty and maven-failsafe-plugin

Source: build-helper: reserve-network-port

Source: AntRun Plugin: run

Source: JEP 408: Simple Web Server

Selecting and skipping tests

The properties below come from help:describe -Ddetail for failsafe:integration-test 3.6.0 and were run on the tidepool project. The surprise is that -DskipTests, which people use to build quickly, does not skip Failsafe:

Command line What ran Result

-Dit.test=MainIT

only MainIT; the pattern overrides includes and excludes

failsafe:3.6.0:integration-test, Tests run: 1, BUILD SUCCESS

-Dit.test=UnknownStationIT#unknownStationExitsWithOne

one method of one class

Tests run: 1

-DskipITs

Surefire ran, Failsafe printed Tests are skipped. for both goals

BUILD SUCCESS

-DskipTests

Surefire printed Tests are skipped.; Failsafe still ran MainIT (Tests run: 1)

BUILD SUCCESS

-Dmaven.test.skip=true

Surefire and Failsafe both printed Tests are skipped. (the property also stops the test sources from being compiled)

BUILD SUCCESS

-Dmaven.test.skip.exec=true

both skipped, test sources still compiled (deprecated name)

BUILD SUCCESS

So -DskipITs skips only the integration tests, -DskipTests only the unit tests, and -Dmaven.test.skip=true everything. The help:describe text of the Failsafe parameter explains why: "Unlike Surefire’s skipTests, this Failsafe parameter is not bound to the skipTests user property", and skipTests on the Failsafe goal is deprecated in favour of skipITs. A release build that was meant to skip all tests with -DskipTests was therefore still starting a server and running the ITs. Skipping the verify goal needs no separate flag, because skipITs is honoured there too.

Two details from the runs. -Dtest=StationReaderTest (the Surefire selector) in the whole reactor failed the build in tidepool-model with No tests matching pattern "StationReaderTest" were executed! (Set -Dsurefire.failIfNoSpecifiedTests=false to ignore this error.); Failsafe has the matching failsafe.failIfNoSpecifiedTests (and it.failIfNoSpecifiedTests). And it.test takes class names with an optional #method, the same syntax as test, so the method forms in Testing with Surefire apply.

Source: Failsafe: integration-test, skipITs

Source: Failsafe: integration-test, test

Source: Failsafe plugin: running a single test

Flaky tests and reruns

Integration tests depend on timing and on the environment, so they are the usual place for flakiness. Failsafe can rerun a failed test: rerunFailingTestsCount (property failsafe.rerunFailingTestsCount, default 0) sets how many extra attempts a failing test gets. To see the effect a throwaway FlakyIT was written that fails on its first attempt only, using a marker file next to the data file. Without reruns the build failed. With -Dfailsafe.rerunFailingTestsCount=1 it passed and reported the test as a flake:

[INFO] Running io.tidepool.cli.FlakyIT
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.024 s <<< FAILURE! -- in io.tidepool.cli.FlakyIT
[INFO] Running io.tidepool.cli.FlakyIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s -- in io.tidepool.cli.FlakyIT
[WARNING] Flakes:
[WARNING] io.tidepool.cli.FlakyIT.failsOnTheFirstAttemptOnly
[ERROR]   Run 1: FlakyIT.failsOnTheFirstAttemptOnly:18 first attempt fails on purpose ==> expected: <true> but was: <false>
[INFO]   Run 2: PASS
[WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO] BUILD SUCCESS

The summary file for that run had <failures>0</failures> and <flakes>1</flakes>. A passing build with flakes is easy to ignore; the verify goal has failOnFlakeCount (property failsafe.failOnFlakeCount, default 0 meaning unlimited) to turn a number of flakes into a failure. Treat reruns as a way to keep the pipeline moving while the flake is fixed, not as a fix. FlakyIT was deleted after the run.

Source: Failsafe plugin: Rerun failing tests

Source: Failsafe: verify, failOnFlakeCount

Running the tests several times

To run the same tests against several configurations (two databases, two stations), the Failsafe documentation uses several integration-test executions with a different summaryFile each, and a verify execution that lists all the files in summaryFiles. In a copy of tidepool MainIT was changed to read station.id and station.name from system properties (defaulting to Boston), and tidepool-cli got:

<plugin>
  <artifactId>maven-failsafe-plugin</artifactId>
  <executions>
    <execution>
      <id>it-boston</id>
      <goals><goal>integration-test</goal></goals>
      <configuration>
        <systemPropertyVariables>
          <station.id>BOS</station.id>
          <station.name>Boston</station.name>
        </systemPropertyVariables>
        <summaryFile>${project.build.directory}/failsafe-reports/failsafe-summary-boston.xml</summaryFile>
      </configuration>
    </execution>
    <execution>
      <id>it-honolulu</id>
      <goals><goal>integration-test</goal></goals>
      <configuration>
        <systemPropertyVariables>
          <station.id>HNL</station.id>
          <station.name>Honolulu</station.name>
        </systemPropertyVariables>
        <summaryFile>${project.build.directory}/failsafe-reports/failsafe-summary-honolulu.xml</summaryFile>
      </configuration>
    </execution>
    <execution>
      <id>verify-all</id>
      <goals><goal>verify</goal></goals>
      <configuration>
        <summaryFiles>
          <summaryFile>${project.build.directory}/failsafe-reports/failsafe-summary-boston.xml</summaryFile>
          <summaryFile>${project.build.directory}/failsafe-reports/failsafe-summary-honolulu.xml</summaryFile>
        </summaryFiles>
      </configuration>
    </execution>
  </executions>
</plugin>

The log showed failsafe:3.6.0:integration-test (it-boston) and (it-honolulu), each with Tests run: 3, Failures: 0, then the stop-server step, then failsafe:3.6.0:verify (verify-all) and BUILD SUCCESS. The report directory held failsafe-summary-boston.xml and failsafe-summary-honolulu.xml, but only one set of TEST-*.xml and *.txt files: the second execution overwrote the first one’s, because both write to the same reportsDirectory. Give each execution its own reportsDirectory if the per-class reports of every run matter. This copy used no managed integration-test execution in the parent; with one, it would have run a third time.

Source: Failsafe plugin: Running integration tests multiple times

Multi-module projects

The two layouts are equally common:

  • Integration tests in the module they test. This is the tidepool-cli layout above: the module declares the plugin (the parent holds the managed executions with an id, as the Failsafe documentation recommends) and src/test/java holds the *IT classes. Simple, and the right answer for most projects.

  • A separate integration-test module that depends on the artifacts under test, built last. It keeps heavyweight test dependencies out of the production module, and lets one set of tests exercise several modules together.

The second layout was run with a tidepool-it module after tidepool-cli, holding the three ITs and the server plugins, with tidepool-cli as a normal dependency and no tests of its own. mvn verify from the root gave:

[INFO] Reactor Build Order:
[INFO] tidepool-parent                                                    [pom]
[INFO] tidepool-model                                                     [jar]
[INFO] tidepool-engine                                                    [jar]
[INFO] tidepool-cli                                                       [jar]
[INFO] tidepool-it                                                        [jar]
...
[INFO] --- failsafe:3.6.0:integration-test (default) @ tidepool-it ---
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] tidepool-it ........................................ SUCCESS [  2.861 s]
[INFO] BUILD SUCCESS

(... marks omitted lines.) The tests ran against the tidepool-cli JAR built a moment earlier in the same reactor. The catch is building the module alone. mvn verify -pl tidepool-it fails, because the sibling was never built or installed:

[WARNING] The POM for io.tidepool:tidepool-cli:jar:1.0.0-SNAPSHOT is missing, no dependency information available
[ERROR] Failed to execute goal on project tidepool-it: Could not resolve dependencies for project io.tidepool:tidepool-it:jar:1.0.0-SNAPSHOT
[ERROR] dependency: io.tidepool:tidepool-cli:jar:1.0.0-SNAPSHOT (compile)
[ERROR] 	prefixes: Path io/tidepool/tidepool-cli/1.0.0-SNAPSHOT/tidepool-cli-1.0.0-SNAPSHOT.jar NOT allowed from central

Add -am (--also-make) so the modules it needs are built first, as the reactor page explains in Multi-Module Builds & the Reactor. With -pl tidepool-it -am the build passed. (The NOT allowed from central line comes from Maven 3.10’s remote-repository prefix filtering; it only says that Central does not host these coordinates.) Because -pl tidepool-it -am also runs Surefire in the earlier modules, a -Dtest= filter needs -Dsurefire.failIfNoSpecifiedTests=false there.

Source: Failsafe plugin: Usage in multi-module projects

Maven 3.9.16, 3.10.0 and Maven 4

The Failsafe 3.6.0 build above, clean verify with the port reservation, the server steps and three ITs, gave the same result on Maven 3.9.16 and on Maven 4.0.0-rc-7 (release candidate) as on Maven 3.10.0: the same plugin lines in the same order, Tests run: 3, Failures: 0, and BUILD SUCCESS. Failsafe needs no Maven 4 specific version.

What does change is how you name the phases. Maven 4 gives every lifecycle phase a before: and an after: phase, and the What is new in Maven 4 page uses before:integration-test as its example for setting up test data. Binding the server steps to before:integration-test and after:integration-test in a copy of the project gave:

  • Maven 4.0.0-rc-7: the order was the same as with pre- and post-: antrun (start-server), then failsafe:integration-test, then antrun (stop-server), then failsafe:verify; BUILD SUCCESS.

  • Maven 3.10.0: the build did not understand the names and said nothing. There was no start-server and no stop-server line, no warning, and no error about an unknown phase. Failsafe ran against a server that was not there, and StationServerIT failed with Connection refused, so the build failed at verify.

[ERROR]   StationServerIT.servesThreeStations:18 » Connect Connection refused
[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0
[INFO] --- failsafe:3.6.0:verify (default) @ tidepool-cli ---
[ERROR] BUILD FAILURE

So a POM that must build on both Maven 3 and Maven 4 should keep pre-integration-test and post-integration-test: the earlier Maven 4.0.0-rc-7 run, with those names, behaved exactly like Maven 3.10.0. Use the before: and after: forms only for projects that require Maven 4. The meaning of the old post-* phases also changed in Maven 4, as the lifecycle pages describe; see Maven 4 Lifecycle and Migrating to Maven 4.

Source: What is new in Maven 4: before- and after-phases

The same in Ant and Gradle

None of the following was run for this page; the rows are named from the official documentation of each tool.

Need Maven Gradle Ant

Separate integration tests from unit tests

Failsafe plugin; *IT naming; integration-test and verify phases

a second test suite (testing.suites with the JVM Test Suite plugin), wired into check

a second junitlauncher target

Run them after packaging

phases after package

a task dependency on the jar task

a target that depends on the jar target

Start and stop a server around them

plugins bound to pre- and post-integration-test

finalizedBy on the test task, or a build service

<trycatch> with a <finally> block, or <parallel> with a daemon

Do not fail until cleanup has run

two goals, integration-test and verify

finalizedBy, which runs even when the task fails

failureProperty on <junitlauncher> and a later <fail>

Details: Testing JVM Projects (Gradle) and Testing with JUnit & junitlauncher (Ant).

Where the books are outdated

  • Maven 2 era books describe integration tests as extra Surefire executions bound to integration-test with an **/*IT.java include. That works but fails the build before the cleanup runs. The Failsafe plugin, with its split into integration-test and verify, replaced the pattern and is the standard runner.

  • Selenium RC, used for browser tests in those books, was replaced by WebDriver; a browser test today is an ordinary *IT class that starts the browser from its own code or from a container.

  • Earlier Failsafe and Surefire releases chose between a JUnit 3, 4 and 4.7 provider from the test dependencies. Since 3.6.0 all tests run through the JUnit Platform, with the Vintage engine for JUnit 4 and the TestNG engine for TestNG.

  • -DskipTests was the universal switch in those books. With Failsafe 3.6.0 it skips only unit tests; use -DskipITs or -Dmaven.test.skip=true.

Summary of pitfalls

  • Pinning Failsafe in pluginManagement does not run it: bind integration-test and verify.

  • Run mvn verify or later. mvn integration-test exits with status 0 whatever the tests do.

  • -DskipTests leaves the integration tests running; use -DskipITs.

  • A class ending in Test runs in Surefire, in the test phase, before the JAR and the server exist; use the IT naming.

  • Reserve the server port with reserve-network-port instead of hard-coding it.

  • In an AntRun script $$ becomes $; use $$$$ for a shell $$, and check the cleanup step really succeeded.

  • before:integration-test is silently ignored on Maven 3.10.0; keep pre-integration-test and post-integration-test in a POM that must build on Maven 3.

  • Several integration-test executions share one report directory: set reportsDirectory per execution.

  • Building a separate integration-test module with -pl needs -am.

References