Maven Testing with Surefire

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

This page is written against Apache Maven 3.10.0 with the Apache Maven Surefire Plugin 3.6.0 (published on Maven Central on 2026-08-31). Every example was run on JDK 26 on 2026-10-16 in scratch copies of the tidepool project (a parent POM with the tidepool-model and tidepool-engine modules, JUnit Jupiter 6.1.3, every build plugin pinned in pluginManagement). The same builds were also run on Maven 3.9.16 and on Maven 4.0.0-rc-7 (a release candidate, not a final release), and the differences are stated. Paths in the console output were shortened, the Loaded N auto-discovered prefixes lines and elapsed times were left out where they would only add noise, and long output was filtered with grep or sed, which is said each time. Surefire needs Maven 3.6.3 or later and Java 8 or later to run; the tests themselves need whatever the test framework needs (JUnit 6 needs Java 17).

Surefire is the plugin that runs the unit tests of a Maven build. It is bound to the test phase, starts a JVM of its own, hands the test classes to the JUnit Platform and turns the result into console output, text files and XML. This page covers what changed in 3.6.0 (one provider for every framework), how tests are found and selected, how a build is told to skip them, how they fail, how the JVM that runs them is configured, and where the reports go. Integration tests, which run later and need the packaged artifact, are the job of its sibling plugin and have their own page: Integration Testing with Failsafe. The phases themselves are described in Maven Build Lifecycle, and plugin configuration in Plugins & Their Configuration. The Spring Boot side of the same plugin, with JaCoCo and the quality plugins, is on Maven Quality Plugins for Spring Boot.

Where Surefire sits

The jar packaging binds exactly one Surefire goal, surefire:test, to the test phase, after compiler:testCompile. The plugin has no other goal. Its configuration comes from the POM (<configuration> of the plugin, or <properties> through the user properties) or from -D on the command line, which is why most parameters below list both a name and a user property.

A build that does not pin the plugin uses whatever version the Maven in use binds by default. The same small project with no <build> section at all, run on the three Maven versions of this page:

mvn -B -ntp clean test

Apache Maven 3.9.16:
[INFO] --- surefire:3.5.4:test (default-test) @ nopin ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider

Apache Maven 3.10.0:
[INFO] --- surefire:3.6.0:test (default-test) @ nopin ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider

Apache Maven 4.0.0-rc-7:
[WARNING] Version not locked for default bindings plugins [maven-clean-plugin, maven-resources-plugin, maven-compiler-plugin, maven-surefire-plugin], you should define versions in pluginManagement section of your pom.xml or parent
[INFO] --- surefire:3.6.0:test (default-test) @ nopin ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider

(Only the Surefire lines are shown, and the output of the three runs was joined under headings.) Maven 3.9.16 binds Surefire 3.5.4, Maven 3.10.0 and Maven 4.0.0-rc-7 bind 3.6.0. Because the default changes with the Maven release, the project below pins the version in pluginManagement, as the other pages of this section do. Pinning also makes the three Maven versions behave the same, which is why the results in the later sections do not depend on the Maven version.

Source: Apache Maven Surefire Plugin

The example project

The parent POM imports the JUnit BOM (see Dependency Management & BOMs), and pins the compiler, resources, JAR and Surefire plugins. Only the Surefire entry matters for this page:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>io.tidepool</groupId>
  <artifactId>tidepool-parent</artifactId>
  <version>1.0.0-SNAPSHOT</version>
  <packaging>pom</packaging>

  <modules>
    <module>tidepool-model</module>
    <module>tidepool-engine</module>
  </modules>

  <properties>
    <maven.compiler.release>21</maven.compiler.release>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <junit.version>6.1.3</junit.version>
  </properties>

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.junit</groupId>
        <artifactId>junit-bom</artifactId>
        <version>${junit.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
      <dependency>
        <groupId>io.tidepool</groupId>
        <artifactId>tidepool-model</artifactId>
        <version>${project.version}</version>
      </dependency>
      <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-csv</artifactId>
        <version>1.14.1</version>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>3.16.0</version>
        </plugin>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>3.6.0</version>
        </plugin>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-resources-plugin</artifactId>
          <version>3.5.0</version>
        </plugin>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-jar-plugin</artifactId>
          <version>3.5.1</version>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>
</project>

tidepool-engine depends on tidepool-model and Commons CSV, and has one test dependency, the junit-jupiter aggregate, whose version comes from the BOM:

    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter</artifactId>
      <scope>test</scope>
    </dependency>

It has four test classes: TideEngineTest (three tests, one tagged slow), StationReaderTest (one test, tagged fast), StationRegistryTest (a class with a @DisplayName, a @Nested class and a @ParameterizedTest) and StationReaderCheck (one test, in a class whose name does not match the default patterns). One of them, in full:

package io.tidepool.engine;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;

import io.tidepool.model.Constituent;
import io.tidepool.model.Station;
import java.time.Instant;
import java.util.List;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;

class TideEngineTest {
    private static final Station FLAT = new Station("T", "Test", 0, 0, List.of());
    private static final Station SINGLE = new Station("S", "Single", 0, 0, List.of(new Constituent("S2", 1.0, 0.0)));

    @Test
    @DisplayName("a station without constituents is always at datum")
    void flatStation() {
        assertEquals(0.0, TideEngine.predict(FLAT, Instant.EPOCH).metres(), 1e-9);
    }

    @Test
    @DisplayName("one constituent at phase zero starts at its amplitude")
    void singleConstituent() {
        assertEquals(1.0, TideEngine.predict(SINGLE, Instant.EPOCH).metres(), 1e-9);
    }

    @Test
    @Tag("slow")
    @DisplayName("a year of hourly predictions stays within the amplitude")
    void yearOfPredictions() {
        for (int h = 0; h < 24 * 365; h++) {
            double v = TideEngine.predict(SINGLE, Instant.EPOCH.plusSeconds(h * 3600L)).metres();
            assertTrue(Math.abs(v) <= 1.0 + 1e-9);
        }
    }
}

These POMs and this test class are this page’s own lab setup, not examples from the Maven documentation.

Running the build, with the Surefire part of the tidepool-engine module shown:

mvn -B -ntp clean test

[INFO] --- surefire:3.6.0:test (default-test) @ tidepool-engine ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.tidepool.engine.TideEngineTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: N s -- in io.tidepool.engine.TideEngineTest
[INFO] Running Tide predictions for a station
[INFO] Running at the epoch
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: N s -- in at the epoch
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: N s -- in Tide predictions for a station
[INFO] Running io.tidepool.engine.StationReaderTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: N s -- in io.tidepool.engine.StationReaderTest
[INFO] Results:
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0

(Each elapsed time was replaced by N s.) Three things stand out. StationReaderCheck did not run, because of its name (see the include section below). A class with a @DisplayName is reported by that display name on the console, and so is its @Nested class. And in this run the parameterized tests of the outer class were counted on the nested class’s line (4 and 0) although only one test is nested: the per-class lines follow the display-name tree, and only the final total of 8 and the XML report (below) are exact per class.

The module tidepool-model has no tests and prints No tests to run.; that does not fail the build unless failIfNoTests is set (see the section on failures).

Source: Surefire plugin: Usage

One provider for every framework: Surefire 3.6.0

Before 3.6.0 Surefire shipped several providers, small adapters in front of a test framework: surefire-junit3, surefire-junit4, surefire-junit47 (for categories and parallel runs), surefire-testng and surefire-junit-platform. It chose one by looking at the test classpath, and the choice changed which parameters worked: parallel and threadCount were meant for the JUnit 4.7 provider and TestNG, groups forced a provider switch, and so on. A line such as Using auto detected provider org.apache.maven.surefire.junit4.JUnit4Provider told you which one had been picked.

Since 3.6.0 the four older providers are gone. The only one left is surefire-junit-platform, and every framework runs through the JUnit Platform:

Test framework Engine that runs it Minimum version

Test framework	Engine that runs it	Minimum version
JUnit Jupiter (JUnit 5 and 6)	Jupiter engine	5.x / 6.x
JUnit 4	Vintage engine	JUnit 4.12
JUnit 3 style tests	Vintage engine, through the JUnit 4 compatibility	a JUnit 4.12 dependency must be present
TestNG	the TestNG engine for the JUnit Platform (`org.junit.support:testng-engine`)	TestNG 6.14.3

JUnit Jupiter (JUnit 5 and 6)

Jupiter engine

5.x / 6.x

JUnit 4

Vintage engine

JUnit 4.12

JUnit 3 style tests

Vintage engine, through the JUnit 4 compatibility

a JUnit 4.12 dependency must be present

TestNG

the TestNG engine for the JUnit Platform (org.junit.support:testng-engine)

TestNG 6.14.3

The release notes call the change a "major architectural simplification". For a build it means: no provider configuration, one set of filtering and reporting behaviour for every framework, and a few parameters that no longer do anything. The same project with JUnit 4 only, on Surefire 3.5.6 (the last 3.5.x release, published 2026-05-25) and on 3.6.0:

Surefire 3.5.6:
[INFO] --- surefire:3.5.6:test (default-test) @ j4-old ---
[INFO] Using auto detected provider org.apache.maven.surefire.junit4.JUnit4Provider
[INFO] Running io.tidepool.LegacyHeightTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.024 s -- in io.tidepool.LegacyHeightTest

Surefire 3.6.0:
[INFO] --- surefire:3.6.0:test (default-test) @ junit4-only ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] Running io.tidepool.LegacyHeightTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.010 s -- in io.tidepool.LegacyHeightTest

(Two separate one-class projects with only junit:junit:4.13.2 as a test dependency, differing in the plugin version; the headings were added.) Staying on 3.5.x is possible when the tests cannot use the JUnit 4.12 or TestNG 6.14.3 minimums, and the release notes also describe a transitional way to add an old provider as a plugin dependency of 3.6.0, with a warning that it may not be kept in future releases.

Source: Surefire plugin: What is new in 3.6.0

How the engine is chosen

Surefire does not need the engine on the project’s test classpath. It looks at what is there and adds the engine to the provider classpath, the classpath of the Surefire code itself, together with junit-platform-launcher:

graph TD CP["Test classpath of the module"] --> SEL{"Which API is present?"} SEL -->|"Jupiter API or engine"| J["junit-jupiter-engine, same version as the API"] SEL -->|"JUnit 4.12 or later"| V["junit-vintage-engine"] SEL -->|"TestNG 6.14.3 or later"| N["testng-engine"] J --> P["surefire-junit-platform provider with junit-platform-launcher"] V --> P N --> P P --> F["One forked JVM runs the Launcher on the test classes"] F --> R["Console summary and target/surefire-reports"]

Engines are added only when they are missing, so a project that wants a specific engine version can still list the engine as a test dependency (the junit-jupiter aggregate does), or as a dependency of the plugin itself.

The debug output of Maven shows the result. For tidepool-engine, which has the whole junit-jupiter aggregate on its test classpath:

mvn -B -ntp -X clean test | grep -E "^\[DEBUG\] provider\(compact\) classpath"

[DEBUG] provider(compact) classpath:  surefire-junit-platform-3.6.0.jar  surefire-api-3.6.0.jar  surefire-logger-api-3.6.0.jar  surefire-shared-utils-3.6.0.jar  common-java5-3.6.0.jar  junit-platform-launcher-6.1.3.jar

The launcher is resolved at the platform version of the project (6.1.3), not at the 1.14.4 Surefire was built with. Where the project has only the API, Surefire adds the engine too (a project with only junit-jupiter-api 6.1.3):

[DEBUG] test(compact) classpath:  test-classes  classes  junit-jupiter-api-6.1.3.jar  opentest4j-1.3.0.jar  junit-platform-commons-6.1.3.jar  apiguardian-api-1.1.2.jar  jspecify-1.0.0.jar
[DEBUG] provider(compact) classpath:  surefire-junit-platform-3.6.0.jar  surefire-api-3.6.0.jar  surefire-logger-api-3.6.0.jar  surefire-shared-utils-3.6.0.jar  common-java5-3.6.0.jar  junit-platform-engine-6.1.3.jar  junit-platform-launcher-6.1.3.jar  junit-jupiter-engine-6.1.3.jar

With only junit:junit:4.13.2 the Vintage engine appears, at the version Surefire was built with:

[DEBUG] test(compact) classpath:  test-classes  classes  junit-4.13.2.jar  hamcrest-core-1.3.jar
[DEBUG] provider(compact) classpath:  surefire-junit-platform-3.6.0.jar  surefire-api-3.6.0.jar  surefire-logger-api-3.6.0.jar  surefire-shared-utils-3.6.0.jar  common-java5-3.6.0.jar  junit-platform-engine-1.14.4.jar  junit-platform-launcher-1.14.4.jar  opentest4j-1.3.0.jar  junit-platform-commons-1.14.4.jar  apiguardian-api-1.1.2.jar  junit-vintage-engine-5.14.4.jar

With only TestNG 7.12.0 the TestNG engine appears, and the console says so:

[DEBUG] provider(compact) classpath:  surefire-junit-platform-3.6.0.jar  surefire-api-3.6.0.jar  surefire-logger-api-3.6.0.jar  surefire-shared-utils-3.6.0.jar  common-java5-3.6.0.jar  junit-platform-engine-1.14.4.jar  junit-platform-launcher-1.14.4.jar  opentest4j-1.3.0.jar  junit-platform-commons-1.14.4.jar  apiguardian-api-1.1.2.jar  testng-engine-1.1.0.jar  jcommander-1.72.jar  bsh-2.0b6.jar
[INFO] TestNG is present. Adding per default org.junit.support:testng-engine:1.1.0

(Only the interesting lines of three separate one-class projects are shown; the TestNG project has two tests in groups fast and slow.) A project that mixes Jupiter and JUnit 4 tests gets both engines without any extra dependency, and the Vintage engine is chosen at the platform version of the Jupiter dependency:

[DEBUG] provider(compact) classpath:  surefire-junit-platform-3.6.0.jar  surefire-api-3.6.0.jar  surefire-logger-api-3.6.0.jar  surefire-shared-utils-3.6.0.jar  common-java5-3.6.0.jar  junit-platform-engine-6.1.3.jar  junit-platform-launcher-6.1.3.jar  junit-jupiter-engine-6.1.3.jar  junit-vintage-engine-6.1.3.jar

Both tests ran (Tests run: 2). To change the choice, put the engine you want into the plugin’s own <dependencies>, or filter engines by id with includeJUnit5Engines and excludeJUnit5Engines (user properties surefire.includeJUnit5Engines and surefire.excludeJUnit5Engines); the Surefire documentation reserves the system properties includejunit5engines and excludejunit5engines for itself.

Source: Surefire plugin: Using JUnit 5 Platform, provider selection

Source: Surefire plugin: Using JUnit 5 Platform

TestNG and its suite files

TestNG tests need no more than the TestNG dependency, as the debug output above shows, and groups and excludedGroups map to TestNG groups. With the two-group TestNG class:

-Dgroups=fast          -> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
-DexcludedGroups=fast  -> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
-Dtest=TestNgHeightTest#slow*  -> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

(The three summary lines of three runs of the TestNG project, with the command-line option written in front.) What no longer works is suiteXmlFiles, the way earlier books select a TestNG suite file. Setting it on 3.6.0, in the POM or with the user property, runs every test and prints a deprecation warning that is easy to overlook:

[WARNING]  Parameter 'suiteXmlFiles' (user property 'surefire.suiteXmlFiles') is deprecated: not supported after 3.6.0, please use groups or Junit suite support
[INFO] TestNG is present. Adding per default org.junit.support:testng-engine:1.1.0
[INFO] Running io.tidepool.TestNgHeightTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 s -- in io.tidepool.TestNgHeightTest

The suite file in that run selected group fast only, so one test was expected; two ran. Replace the suite file with groups, or use the JUnit Platform suite engine.

Source: Surefire plugin: What is new in 3.6.0, TestNG users

Which classes run: includes and excludes

Surefire scans the compiled test classes (target/test-classes) and runs the classes that match its include patterns and not its exclude patterns. The defaults:

<includes>
  <include>**/Test*.java</include>
  <include>**/*Test.java</include>
  <include>**/*Tests.java</include>
  <include>**/*TestCase.java</include>
</includes>
<excludes>
  <exclude>**/*$*</exclude>
</excludes>

The patterns are written with .java but matched against the compiled class files, relative to the test classes directory. The exclude pattern drops every class whose name contains a dollar sign, so nested and anonymous classes are never started on their own. JUnit still runs a @Nested class (the output above shows at the epoch running), because the JUnit Platform discovers it through the enclosing class. That is why StationReaderCheck, which ends in Check, was not run. Two ways to run it. The parameter includes has the user property surefire.includes (and excludes has surefire.excludes):

mvn -B -ntp test '-Dsurefire.includes=**/*Check.java'

[INFO] Running io.tidepool.engine.StationReaderCheck
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.029 s -- in io.tidepool.engine.StationReaderCheck
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

Setting includes replaces the defaults: only the Check class ran. To keep the defaults, list them again together with the new one:

mvn -B -ntp test '-Dsurefire.includes=**/*Test.java,**/*Check.java'

[INFO] Running io.tidepool.engine.TideEngineTest
[INFO] Running Tide predictions for a station
[INFO] Running at the epoch
[INFO] Running io.tidepool.engine.StationReaderCheck
[INFO] Running io.tidepool.engine.StationReaderTest
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0

(Only the Running lines and the final total are shown.) In a POM the same patterns are <include> and <exclude> elements; a value may hold a comma-separated list. Patterns support more than Ant-style paths:

%regex[.*(Cat|Dog).*Test.*] is a regular expression over the class file path, with forward slashes even on Windows, and with a literal .class suffix that is not part of the expression.
! in front of a pattern excludes it (in includes, test, includesFile; the excludes and excludesFile parameters should not use it).
? replaces one character; extensions are optional; a fully qualified name such as my.package.* also works.
includesFile and excludesFile (user properties surefire.includesFile and surefire.excludesFile) read patterns from a file, one per line, with blank lines and # comments ignored. Since 3.0.0-M6 the exclusions file can also name methods (pkg.SomeTest#testMethod).
dependenciesToScan runs test classes found in dependencies, for example the test-jar of another module, given as groupId[:artifactId[:type[:classifier][:version]]].

Source: Surefire plugin: Inclusions and exclusions, inclusions

Source: Surefire plugin: Inclusions and exclusions, exclusions

Running a subset from the command line: -Dtest

During development one test class, or one method, is run repeatedly. The user property test overrides includes and excludes. The value is a class name (the .java is optional, and an extension is stripped if given), a pattern, or a comma-separated list of them. All of these ran against tidepool-engine:

mvn -B -ntp test -Dtest=TideEngineTest
mvn -B -ntp test '-Dtest=TideEngineTest#flat*'
mvn -B -ntp test '-Dtest=TideEngineTest#flatStation+singleConstituent'
mvn -B -ntp test '-Dtest=StationRegistryTest$AtTheEpoch'
mvn -B -ntp test '-Dtest=StationRegistryTest#amplitudes*'

-Dtest=TideEngineTest                                      Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
-Dtest=TideEngineTest#flat*                                Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
-Dtest=TideEngineTest#flatStation+singleConstituent        Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
-Dtest=StationRegistryTest$AtTheEpoch                      Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
-Dtest=StationRegistryTest#amplitudes*                     Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

(The final summary line of five separate runs, with the option written in front. These ran with -Dsurefire.failIfNoSpecifiedTests=false; see the next paragraph.) Quote the value in a shell: #, *, ! and $ mean something to it. Class#method selects methods, * and ? are wildcards, + joins several methods, and the complex form "-Dtest=???Test, !Unstable*, pkg/**/Ci*leTest.java, *Test#test*One+testTwo?????, #fast*+slowTest" mixes class patterns, exclusions and method patterns.

The Surefire page on single tests says that selecting methods "is supported only for JUnit 4.x and TestNG". That text predates the unified provider: on 3.6.0 the method selection above worked for Jupiter, including the parameterized method, which ran its three invocations.

A name that does not match anything fails the build, which is the most common surprise in a multi-module project. -Dtest=TideEngineTest is applied to every module of the reactor, and tidepool-model has no such class:

mvn -B -ntp test -Dtest=TideEngineTest

[INFO] --- surefire:3.6.0:test (default-test) @ tidepool-model ---
[INFO] No tests to run.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for tidepool-parent 1.0.0-SNAPSHOT:
[INFO]
[INFO] tidepool-engine .................................... SKIPPED
[INFO] tidepool-parent .................................... SUCCESS [  0.000 s]
[ERROR] tidepool-model ..................................... FAILURE [  0.529 s]
...
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.6.0:test (default-test) on project tidepool-model: No tests matching pattern "TideEngineTest" were executed! (Set -Dsurefire.failIfNoSpecifiedTests=false to ignore this error.) -> [Help 1]

(The ... line marks omitted lines.) Two fixes: run only the module that has the test (-pl tidepool-engine, with -am when it needs the modules it depends on), or set the user property surefire.failIfNoSpecifiedTests to false, which is what the examples on this page do; the dependency on -am is described in Multi-Module Builds & the Reactor. The runs of a negated pattern show another catch: -Dtest=!StationRegistryTest excludes the outer class but not its nested class, whose class name is StationRegistryTest$AtTheEpoch, so one of its tests still ran (Tests run: 6 instead of 5). Write -Dtest='!StationRegistryTest*' to exclude both.

mvn surefire:test -Dtest=... runs the goal directly, against whatever is in target/test-classes; use it only after a compile.

Source: Surefire plugin: Running a single test

Source: Surefire plugin: Running a set of methods in a single test class

Tags and groups

groups includes tests by JUnit tag (or TestNG group), excludedGroups excludes them. Both take a tag expression: |, &, ! and parentheses are supported, and a comma-separated list in excludedGroups is a union. Both have a user property of the same name. Against the two @Tag values of the project (slow on one test, fast on two):

mvn -B -ntp test -Dgroups=slow
mvn -B -ntp test -DexcludedGroups=slow
mvn -B -ntp test '-Dgroups=fast | slow'
mvn -B -ntp test '-Dgroups=fast & !slow'

-Dgroups=slow             Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
-DexcludedGroups=slow     Tests run: 7, Failures: 0, Errors: 0, Skipped: 0
-Dgroups=fast | slow      Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
-Dgroups=fast & !slow     Tests run: 4, Failures: 0, Errors: 0, Skipped: 0

(The final summary line of four runs.) The total 8 of the unfiltered run is 7 + 1.

A common build setting is to keep slow tests out of the everyday build but let CI run them. Put the exclusion in a property, wire the property into the plugin, and let a build clear it from the command line:

  <properties>
    <test.excludedGroups>slow</test.excludedGroups>
  </properties>
  ...
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>3.6.0</version>
          <configuration>
            <excludedGroups>${test.excludedGroups}</excludedGroups>
          </configuration>
        </plugin>

mvn -B -ntp clean test                         -> TideEngineTest: Tests run: 2   (slow excluded)
mvn -B -ntp clean test -Dtest.excludedGroups=  -> TideEngineTest: Tests run: 3   (nothing excluded)

(The TideEngineTest line of two runs.) Binding the exclusion to a profile that CI activates works as well; see Profiles.

Source: Surefire plugin: Using JUnit 5 Platform, filtering by tags

Skipping tests

Four switches exist, and they are not equivalent. The same module with clean test and each switch, showing the lines of tidepool-engine that matter:

mvn -B -ntp clean test -DskipTests
mvn -B -ntp clean test -Dmaven.test.skip=true

### mvn clean test -DskipTests
[INFO] --- compiler:3.16.0:compile (default-compile) @ tidepool-engine ---
[INFO] Compiling 2 source files with javac [debug release 21] to target/classes
[INFO] --- compiler:3.16.0:testCompile (default-testCompile) @ tidepool-engine ---
[INFO] Compiling 4 source files with javac [debug release 21] to target/test-classes
[INFO] --- surefire:3.6.0:test (default-test) @ tidepool-engine ---
[INFO] Tests are skipped.

### mvn clean test -Dmaven.test.skip=true
[INFO] --- compiler:3.16.0:compile (default-compile) @ tidepool-engine ---
[INFO] Compiling 2 source files with javac [debug release 21] to target/classes
[INFO] --- compiler:3.16.0:testCompile (default-testCompile) @ tidepool-engine ---
[INFO] Not compiling test sources
[INFO] --- surefire:3.6.0:test (default-test) @ tidepool-engine ---
[INFO] Tests are skipped.

(The two runs were joined under ### headings, and the resources and clean lines were removed.)

Switch User property Effect

Switch	User property	Effect
`skipTests`	`skipTests`	Test classes are compiled, the tests are not run. The test classes stay available to a `test-jar`.
`skip`	`maven.test.skip`	Skips running and compiling the tests (and copying test resources). Honoured by Surefire, Failsafe and the compiler plugin. The documentation calls its use not recommended.
`skipExec`	`maven.test.skip.exec`	Deprecated alias of `skipTests`.
`skipITs`	`skipITs`	Failsafe only, see the Failsafe page.

skipTests

Test classes are compiled, the tests are not run. The test classes stay available to a test-jar.

skip

maven.test.skip

Skips running and compiling the tests (and copying test resources). Honoured by Surefire, Failsafe and the compiler plugin. The documentation calls its use not recommended.

skipExec

maven.test.skip.exec

Deprecated alias of skipTests.

skipITs

Failsafe only, see the Failsafe page.

The difference matters when another module needs the compiled tests. With the test-jar goal of the JAR plugin in tidepool-engine, package -DskipTests still built tidepool-engine-1.0.0-SNAPSHOT-tests.jar, with 15 entries. With -Dmaven.test.skip=true the test-jar goal printed no Building jar line and no -tests.jar file existed, so a module that depends on that JAR cannot be resolved in the reactor. Use -DskipTests unless the test sources really cannot be compiled.

Since Failsafe 3.6.0 the skipTests property no longer skips integration tests: -DskipTests=true stops only Surefire, and -DskipITs or -Dmaven.test.skip is needed to stop Failsafe. Earlier books treat -DskipTests as the universal switch; see Integration Testing with Failsafe.

A POM can skip tests by default and let the command line turn them back on, by going through a property:

  <properties>
    <skipTests>true</skipTests>
  </properties>
  ...
          <configuration>
            <skipTests>${skipTests}</skipTests>
          </configuration>

mvn install -DskipTests=false then runs them. The same works for the other boolean parameters.

Source: Surefire plugin: Skipping tests

When tests fail

A failing test fails the module and, unless the reactor is told otherwise, the build. A copy of the project with a deliberately wrong assertion (the M2 constituent at the epoch is cos(-120 degrees) * 0.55, which is -0.275, not 0.55):

[INFO] Running io.tidepool.engine.PredictionRegressionTest
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.057 s <<< FAILURE! -- in io.tidepool.engine.PredictionRegressionTest
[ERROR] io.tidepool.engine.PredictionRegressionTest.m2AtTheEpoch -- Time elapsed: 0.031 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <0.55> but was: <-0.2749999999999999>
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1063)
	at io.tidepool.engine.PredictionRegressionTest.m2AtTheEpoch(PredictionRegressionTest.java:17)

...
[INFO] Results:
[ERROR] Failures:
[ERROR]   PredictionRegressionTest.m2AtTheEpoch:17 expected: <0.55> but was: <-0.2749999999999999>
[ERROR] Tests run: 11, Failures: 1, Errors: 0, Skipped: 0
...
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.6.0:test (default-test) on project tidepool-engine: There are test failures.
[ERROR]
[ERROR] See .../tidepool-engine/target/surefire-reports for the individual test results.
[ERROR] See dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.

(The ... lines mark omitted lines of the run, which also contained a second failing test class for the sections below.) Stack trace handling was reimplemented in 3.6.0, and two parameters are new there: stackTraceFilterPrefixes (the package prefixes filtered from captured stack traces; the default is java., javax., sun. and jdk.) and stackTraceMaxFrames (how many frames are kept when console output is attributed to a test class, default 15). A failure is an assertion that did not hold; an error is an exception the test did not expect. Both fail the build.

Options for working with failures:

-Dmaven.test.failure.ignore=true (testFailureIgnore) keeps going and ends with BUILD SUCCESS although the failures are printed (the run above ended with [ERROR] There are test failures. followed by [INFO] BUILD SUCCESS). It hides problems on a CI system; prefer --fail-at-end, which continues with independent modules but still ends in failure (see Running Maven (CLI)).
-Dsurefire.skipAfterFailureCount=1 stops the run after that many failures and marks the rest as skipped. With the failing class first, it reported Tests run: 10, Failures: 1, Errors: 0, Skipped: 9, and every later class printed Skipped: N. The documentation warns that forked or parallel runs make the count approximate.
-DfailIfNoTests=true fails a module that has no tests: tidepool-model then fails with No tests to run!.
-Dsurefire.failIfNoSpecifiedTests=false is the other side, discussed above.

Source: Surefire plugin: surefire:test, testFailureIgnore

Source: Surefire plugin: Skip after failure

Flaky tests: rerunning failures

A test that passes on a second try is a flake. rerunFailingTestsCount (user property surefire.rerunFailingTestsCount) reruns each failing test up to that many times and, if a rerun passes, reports it as a flake instead of a failure. For a test that fails when target/flaky.marker does not exist yet and passes afterwards (so it always fails the first time in a clean build):

mvn -B -ntp clean test -Dsurefire.rerunFailingTestsCount=2 -Dtest=FlakyTest

[INFO] Running io.tidepool.engine.FlakyTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.037 s <<< FAILURE! -- in io.tidepool.engine.FlakyTest
[ERROR] io.tidepool.engine.FlakyTest.failsOnTheFirstAttemptOnly -- Time elapsed: 0.026 s <<< FAILURE!
org.opentest4j.AssertionFailedError: first attempt always fails ==> expected: <true> but was: <false>
	at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:232)
	at io.tidepool.engine.FlakyTest.failsOnTheFirstAttemptOnly(FlakyTest.java:15)

[INFO] Running io.tidepool.engine.FlakyTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in io.tidepool.engine.FlakyTest
[INFO] Results:
[WARNING] Flakes:
[WARNING] io.tidepool.engine.FlakyTest.failsOnTheFirstAttemptOnly
[ERROR]   Run 1: FlakyTest.failsOnTheFirstAttemptOnly:15 first attempt always fails ==> expected: <true> but was: <false>
[INFO]   Run 2: PASS
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1

The build succeeded, with the flake listed. A test that fails in every rerun stays a failure and prints one Run N line per attempt (the regression test above printed Run 1 to Run 3 with -Dsurefire.rerunFailingTestsCount=2). The XML report records the flake as a flakyFailure child of the test case:

  <testcase name="failsOnTheFirstAttemptOnly" classname="io.tidepool.engine.FlakyTest" time="0.001">
    <flakyFailure message="first attempt always fails ==&gt; expected: &lt;true&gt; but was: &lt;false&gt;" type="org.opentest4j.AssertionFailedError">
      <stackTrace><![CDATA[org.opentest4j.AssertionFailedError: first attempt always fails ==> expected: <true> but was: <false>
	at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:232)
	at io.tidepool.engine.FlakyTest.failsOnTheFirstAttemptOnly(FlakyTest.java:15)
]]></stackTrace>
    </flakyFailure>
  </testcase>

Reruns hide flakiness unless something counts it. failOnFlakeCount (user property surefire.failOnFlakeCount) fails the build when the number of flakes exceeds the value. With one flake and -Dsurefire.failOnFlakeCount=1 the build still succeeded; with two flaky test classes it failed:

[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Flakes: 2
[ERROR] BUILD FAILURE
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.6.0:test (default-test) on project tidepool-engine: There are 2 flakes and failOnFlakeCount is set to 1.

Since 3.6.0 the reruns happen after the initial execution completes, for every framework.

Source: Surefire plugin: Rerun failing tests

The forked JVM

By default Surefire starts one JVM per module and runs every test class in it (forkCount=1, reuseForks=true). The tests do not run in Maven’s own JVM, so MAVEN_OPTS does not apply to them; JVM options go in argLine, which is also where the JVM to use is selected (jvm, or a toolchain, see Compiling & Toolchains).

To see which JVM ran what, three probe test classes with two methods each printed their process id and thread name. Default run (the order of the classes is the file system order, A, C, B):

[INFO] Running io.tidepool.engine.ProbeATest
probe A.one pid=5916 thread=main fork=null
probe A.two pid=5916 thread=main fork=null
[INFO] Running io.tidepool.engine.ProbeCTest
probe C.one pid=5916 thread=main fork=null
probe C.two pid=5916 thread=main fork=null
[INFO] Running io.tidepool.engine.ProbeBTest
probe B.one pid=5916 thread=main fork=null
probe B.two pid=5916 thread=main fork=null

(Result lines removed.) One process, one thread. Now with -DforkCount=2, then with -DreuseForks=false, then with -DforkCount=0; only the probe lines are shown:

-DforkCount=2
probe A.one pid=6352 thread=main fork=null
probe C.one pid=6353 thread=main fork=null
probe A.two pid=6352 thread=main fork=null
probe C.two pid=6353 thread=main fork=null
probe B.one pid=6352 thread=main fork=null
probe B.two pid=6352 thread=main fork=null

-DreuseForks=false
probe A.one pid=6725 thread=main fork=null
probe A.two pid=6725 thread=main fork=null
probe C.one pid=6759 thread=main fork=null
probe C.two pid=6759 thread=main fork=null
probe B.one pid=6917 thread=main fork=null
probe B.two pid=6917 thread=main fork=null

-DforkCount=0
[WARNING] useSystemClassLoader setting has no effect when not forking
[WARNING] The parameter forkCount should likely not be 0. Forking a JVM for tests improves test accuracy. Ensure to have a <forkCount> >= 1.
probe A.one pid=7012 thread=main fork=null
...

forkCount (user property forkCount) is the maximum number of JVMs at the same time; a value ending in C is multiplied by the number of cores (1.5C). Classes A and C ran in different processes at the same time, and B reused the first.
reuseForks=false starts a fresh JVM for each test class, so static state cannot leak between classes, at the price of the JVM start for every class. Each of the three classes ran in its own process.
forkCount=0 runs the tests inside the Maven JVM. Surefire warns twice. It is only worth it for debugging, and it cannot be combined with -T parallel builds (the system properties would be shared).
argLine passes JVM options. systemPropertyVariables, environmentVariables (since 2.1.3) and workingDirectory are the other levers; user properties given with -D on the Maven command line are promoted to system properties of the tests (promoteUserPropertiesToSystemProperties, since 3.4.0, true by default).
enableAssertions (default true) turns -ea on, so assert statements run in tests.

Source: Surefire plugin: Fork options, forked test execution

One fork, one schema: the forkNumber placeholder

Each concurrent fork gets a number, counted from 1, that can be used in argLine, environmentVariables (since 3.2.0), the working directory and the system properties, so concurrent forks do not share a database schema, a port or a directory. A profile that configures system properties, an environment variable and a working directory, with two forks:

<configuration>
  <forkCount>2</forkCount>
  <reuseForks>true</reuseForks>
  <argLine>-Xmx256m</argLine>
  <systemPropertyVariables>
    <fork.number>fork-${surefire.forkNumber}</fork.number>
  </systemPropertyVariables>
  <workingDirectory>${project.build.directory}/fork-${surefire.forkNumber}</workingDirectory>
</configuration>

probe C.one pid=11193 thread=main fork=fork-1
probe A.one pid=11194 thread=main fork=fork-2
probe C.two pid=11193 thread=main fork=fork-1
probe A.two pid=11194 thread=main fork=fork-2
probe B.one pid=11193 thread=main fork=fork-1
probe B.two pid=11193 thread=main fork=fork-1

Surefire created target/fork-1 and target/fork-2 by itself. There is one trap. Written as the whole value, the placeholder is lost. This configuration gave a test that read fork.number the value null:

<fork.number>${surefire.forkNumber}</fork.number>

Maven’s expression evaluator sees a single unresolved expression and the debug output shows what the plugin got:

[DEBUG]   (s) systemPropertyVariables = {fork.number=null}
[DEBUG]   (s) workingDirectory = .../tidepool-engine/target/fork-${surefire.forkNumber}

The same placeholder inside a longer value is kept for the plugin to replace later: the second line shows the working directory with the placeholder intact. So put some text around the expression (fork-${surefire.forkNumber}, as in the working example) or build the value in argLine.

Source: Surefire plugin: Fork options, forked test execution

Other configuration of the forked process, from a profile that sets a system property with the path of the CSV file, an environment variable and the working directory, and a test that prints them:

<configuration>
  <systemPropertyVariables>
    <stations.file>${project.basedir}/../data/stations.csv</stations.file>
  </systemPropertyVariables>
  <environmentVariables>
    <TIDEPOOL_ZONE>UTC</TIDEPOOL_ZONE>
  </environmentVariables>
  <workingDirectory>${project.build.directory}</workingDirectory>
</configuration>

stations.file=.../tidepool-engine/../data/stations.csv
TIDEPOOL_ZONE=UTC
user.dir ends with target

The working directory changes where relative paths in tests resolve, so a test that opens data/stations.csv should use an absolute path from a property like this.

Source: Surefire plugin: surefire:test, systemPropertyVariables

Source: Surefire plugin: surefire:test, environmentVariables

Source: Surefire plugin: surefire:test, workingDirectory

argLine and JaCoCo

The JaCoCo agent is attached to the test JVM by an argLine that the jacoco:prepare-agent goal sets as a Maven property in an earlier phase. A Surefire <argLine> in the POM replaces that property, silently dropping the agent. In a copy of the project with prepare-agent bound and a plain <argLine>-Xmx256m</argLine>, the build succeeded and no coverage file was produced:

[INFO] --- jacoco:0.8.15:prepare-agent (default) @ tidepool-engine ---
[INFO] argLine set to -javaagent:.../org.jacoco.agent-0.8.15-runtime.jar=destfile=.../tidepool-engine/target/jacoco.exec
...
[DEBUG]   (s) argLine = -Xmx256m

(ls tidepool-engine/target/*.exec found nothing.) The late-replacement placeholder @{argLine} is evaluated by Surefire when it runs, so it picks up the property as changed by other plugins. The same build with <argLine>@{argLine} -Xmx256m</argLine>:

[DEBUG]   (s) argLine = @{argLine} -Xmx256m

and tidepool-engine/target/jacoco.exec existed. If the property is not set at all (no agent), @{argLine} is replaced by an empty string. The reporting side of JaCoCo is on Site & Reporting.

Source: Surefire plugin: surefire:test, argLine

Timeouts

forkedProcessTimeoutInSeconds (user property surefire.timeout) kills the forked JVM when a test run takes too long. With a test that sleeps for ten minutes and -Dsurefire.timeout=3:

[ERROR] Surefire is going to kill self fork JVM. Received SHUTDOWN {KILL} command from Maven shutdown hook.
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[ERROR] tidepool-engine .................................... FAILURE [  4.017 s]
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.6.0:test (default-test) on project tidepool-engine: There was a timeout in the fork -> [Help 1]

(Selected lines.) The run ended after four seconds and left .dump and .dumpstream files in target/surefire-reports; they are where the details of a crashed or killed fork go, and the "See dump files" line of the failure message points at them. For a single test, JUnit’s @Timeout is more precise; this parameter is a safety net for a hung build.

Run order and parallel execution

The default runOrder is filesystem, the order the operating system lists the class files (the probes ran A, C, B). Other values: alphabetical, reversealphabetical, random, failedfirst and balanced (the last is for the parallel option and keeps a statistics file). With the probe classes:

-Dsurefire.runOrder=alphabetical         Running ProbeATest, ProbeBTest, ProbeCTest
-Dsurefire.runOrder=reversealphabetical  Running ProbeCTest, ProbeBTest, ProbeATest
-Dsurefire.runOrder=random               [INFO] Tests will run in random order. To reproduce ordering use flag -Dsurefire.runOrder.random.seed=213353504304416

(Summarised from three runs.) random prints its seed, and -Dsurefire.runOrder.random.seed=<seed> (runOrderRandomSeed, since 3.0.0-M6) repeats an ordering. Use random in CI to find tests that depend on each other. Only the order of classes changes; the order of methods inside a class is JUnit’s own (@TestMethodOrder).

The Surefire parameters for parallel execution are parallel (classes, methods, both, suites and so on), threadCount, perCoreThreadCount, useUnlimitedThreads and the threadCountClasses, threadCountMethods and threadCountSuites variants. Their documentation says they were written for the JUnit 4.7 provider and for TestNG. On 3.6.0 with the JUnit Platform they do nothing, and nothing says so. Running the three probe classes with -Dparallel=classes -DthreadCount=3 -DperCoreThreadCount=false:

[INFO] Running io.tidepool.engine.ProbeATest
probe A.one pid=8412 thread=main fork=null
probe A.two pid=8412 thread=main fork=null
[INFO] Running io.tidepool.engine.ProbeCTest
probe C.one pid=8412 thread=main fork=null
...

All six probes ran one after the other on the main thread. Parallelism for the Jupiter engine is JUnit’s own and is configured with configuration parameters, which Surefire passes to the launcher through the configurationParameters property (a junit-platform.properties file in src/test/resources does the same). A profile:

<configuration>
  <properties>
    <configurationParameters>
      junit.jupiter.execution.parallel.enabled = true
      junit.jupiter.execution.parallel.mode.default = concurrent
      junit.jupiter.execution.parallel.config.strategy = fixed
      junit.jupiter.execution.parallel.config.fixed.parallelism = 4
    </configurationParameters>
  </properties>
</configuration>

[INFO] Running io.tidepool.engine.ProbeATest
[INFO] Running io.tidepool.engine.ProbeCTest
[INFO] Running io.tidepool.engine.ProbeBTest
probe A.two pid=11972 thread=ForkJoinPool-1-worker-2 fork=null
probe B.two pid=11972 thread=ForkJoinPool-1-worker-1 fork=null
probe C.one pid=11972 thread=ForkJoinPool-1-worker-4 fork=null
probe C.two pid=11972 thread=ForkJoinPool-1-worker-3 fork=null
probe A.one pid=11972 thread=ForkJoinPool-1-worker-4 fork=null
probe B.one pid=11972 thread=ForkJoinPool-1-worker-1 fork=null

One process, four worker threads of JUnit’s fork-join pool, methods of different classes interleaved. The same parameters work as -D system properties on the command line (for example -Djunit.jupiter.execution.parallel.enabled=true), because user properties are promoted to system properties. Tests that run in parallel must not share mutable state. Forking (forkCount) and JUnit parallelism combine: processes run classes, each process runs methods on its own threads. Maven’s own parallel build, -T, multiplies both; see Build Performance: Cache, Parallel Builds & mvnd. JUnit’s parallel execution and its configuration parameters are documented in the JUnit user guide.

Source: Surefire plugin: Using JUnit 5 Platform, configuration parameters

Source: Surefire plugin: Fork options, parallel test execution

Reports and output

Surefire writes into target/surefire-reports (parameter reportsDirectory). After the first run of this page:

TEST-io.tidepool.engine.StationReaderTest.xml
TEST-io.tidepool.engine.StationRegistryTest.xml
TEST-io.tidepool.engine.TideEngineTest.xml
io.tidepool.engine.StationReaderTest.txt
io.tidepool.engine.StationRegistryTest.txt
io.tidepool.engine.TideEngineTest.txt

The .txt file is the per-class summary (reportFormat is brief or plain). The TEST-*.xml file is the machine-readable report that CI servers read; its schema is published by the plugin. The XML of StationRegistryTest, with the properties element left out, reports exact per-class counts and names tests by method:

<testsuite ... name="io.tidepool.engine.StationRegistryTest" time="0.058" tests="4" errors="0" skipped="0" failures="0" flakes="0">
  <testcase name="amplitudes(double)[1]" classname="io.tidepool.engine.StationRegistryTest" time="0.007"/>
  <testcase name="amplitudes(double)[2]" classname="io.tidepool.engine.StationRegistryTest" time="0.001"/>
  <testcase name="amplitudes(double)[3]" classname="io.tidepool.engine.StationRegistryTest" time="0.005"/>
  <testcase name="equalsTheAmplitude" classname="io.tidepool.engine.StationRegistryTest$AtTheEpoch" time="0.001"/>
</testsuite>

The report names the methods, not the @DisplayName text. To get display names in the XML, select the JUnit 5 variant of the report extension (available since 3.0.0-M4):

<configuration>
  <statelessTestsetReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5Xml30StatelessReporter">
    <usePhrasedFileName>false</usePhrasedFileName>
    <usePhrasedTestSuiteClassName>true</usePhrasedTestSuiteClassName>
    <usePhrasedTestCaseClassName>true</usePhrasedTestCaseClassName>
    <usePhrasedTestCaseMethodName>true</usePhrasedTestCaseMethodName>
  </statelessTestsetReporter>
</configuration>

  <testcase name="a station without constituents is always at datum" classname="io.tidepool.engine.TideEngineTest" time="0.011"/>
  <testcase name="one constituent at phase zero starts at its amplitude" classname="io.tidepool.engine.TideEngineTest" time="0.002"/>
  <testcase name="a year of hourly predictions stays within the amplitude" classname="io.tidepool.engine.TideEngineTest" time="0.003"/>

(The testcase lines of TEST-io.tidepool.engine.TideEngineTest.xml after the change.) Similar consoleOutputReporter and statelessTestsetInfoReporter variants exist; the page of the JUnit Platform examples lists all their options, and the third-party tree reporter it mentions is not part of Apache Maven.

What the tests print to standard output goes to the console by default, which is why the probe lines appeared above. With -Dmaven.test.redirectTestOutputToFile=true (redirectTestOutputToFile) the console stays quiet and the output goes to a file beside the report:

$ ls tidepool-engine/target/surefire-reports
TEST-io.tidepool.engine.ProbeATest.xml
io.tidepool.engine.ProbeATest-output.txt
io.tidepool.engine.ProbeATest.txt
$ cat tidepool-engine/target/surefire-reports/io.tidepool.engine.ProbeATest-output.txt
probe A.one pid=22446 thread=main fork=null
probe A.two pid=22446 thread=main fork=null

An HTML report is a separate plugin and a site task: mvn surefire-report:report-only after the tests wrote tidepool-engine/target/reports/surefire.html (3.6.0, with a harmless Unable to locate Test Source XRef to link to — DISABLED warning when the JXR report is not part of the build). See Site & Reporting.

Source: Surefire plugin: Using JUnit 5 Platform, extensions and reports for @DisplayName

Source: Apache Maven Surefire Report Plugin

Parameters at a glance

The parameters used on this page, from the surefire:test goal documentation of 3.6.0. A blank default means none.

Parameter User property Default Purpose

Parameter	User property	Default	Purpose
`test`	`test`		Class and method patterns that override `includes` and `excludes`
`includes`, `excludes`	`surefire.includes`, `surefire.excludes`	see above	Class patterns; `includesFile`, `excludesFile` read them from a file
`failIfNoSpecifiedTests`	`surefire.failIfNoSpecifiedTests`	`true`	Fail when `-Dtest` matches nothing
`failIfNoTests`	`failIfNoTests`	`false`	Fail when a module has no tests
`groups`, `excludedGroups`	`groups`, `excludedGroups`		Tag expressions or TestNG groups
`skipTests`	`skipTests`	`false`	Compile the tests, do not run them
`skip`	`maven.test.skip`	`false`	Neither compile nor run tests
`testFailureIgnore`	`maven.test.failure.ignore`	`false`	Report failures but do not fail the build
`skipAfterFailureCount`	`surefire.skipAfterFailureCount`	`0`	Skip the remaining tests after this many failures
`rerunFailingTestsCount`	`surefire.rerunFailingTestsCount`	`0`	Rerun failing tests; a pass on rerun is a flake
`failOnFlakeCount`	`surefire.failOnFlakeCount`	`0`	Fail when more than this many flakes occurred (`0` means unlimited)
`forkCount`	`forkCount`	`1`	Number of forked JVMs; `0` runs in the Maven JVM
`reuseForks`	`reuseForks`	`true`	`false` starts a JVM per test class
`argLine`	`argLine`		JVM options of the fork; `@{...}` placeholders are replaced late
`systemPropertyVariables`, `environmentVariables`, `workingDirectory`	`basedir` for the directory		System properties, environment and working directory of the fork
`forkedProcessTimeoutInSeconds`	`surefire.timeout`		Kill the fork after this many seconds (`0` waits forever)
`runOrder`, `runOrderRandomSeed`	`surefire.runOrder`, `surefire.runOrder.random.seed`	`filesystem`	Order of the test classes
`properties` (`configurationParameters`)			JUnit Platform configuration parameters
`includeJUnit5Engines`, `excludeJUnit5Engines`	`surefire.includeJUnit5Engines`, `surefire.excludeJUnit5Engines`		Engine ids to include or exclude
`redirectTestOutputToFile`	`maven.test.redirectTestOutputToFile`	`false`	Write the test output to a file in the reports directory
`reportsDirectory`		`${project.build.directory}/surefire-reports`	Where reports are written
`trimStackTrace`	`trimStackTrace`	`false`	Show only the lines in the test in report stack traces
`useModulePath`	`surefire.useModulePath`	`true`	Use the module path when a `module-info.java` is present (JDK 9 and later)
`additionalClasspathElements`, `classpathDependencyExcludes`	`maven.test.additionalClasspath`, `maven.test.dependency.excludes`		Add to, or remove from, the test classpath
`dependenciesToScan`	`dependenciesToScan`		Dependencies to search for test classes
`enableProcessChecker`	`surefire.enableProcessChecker`		Let forks end early when the Maven process is killed (`ping`, `native` or `all`)
`parallel`, `threadCount` and relatives	`parallel`, `threadCount`		No effect with the JUnit Platform (see the parallel section)

test

Class and method patterns that override includes and excludes

includes, excludes

surefire.includes, surefire.excludes

see above

Class patterns; includesFile, excludesFile read them from a file

failIfNoSpecifiedTests

surefire.failIfNoSpecifiedTests

true

Fail when -Dtest matches nothing

failIfNoTests

false

Fail when a module has no tests

groups, excludedGroups

Tag expressions or TestNG groups

skipTests

false

Compile the tests, do not run them

skip

maven.test.skip

false

Neither compile nor run tests

testFailureIgnore

maven.test.failure.ignore

false

Report failures but do not fail the build

skipAfterFailureCount

surefire.skipAfterFailureCount

0

Skip the remaining tests after this many failures

rerunFailingTestsCount

surefire.rerunFailingTestsCount

0

Rerun failing tests; a pass on rerun is a flake

failOnFlakeCount

surefire.failOnFlakeCount

0

Fail when more than this many flakes occurred (0 means unlimited)

forkCount

1

Number of forked JVMs; 0 runs in the Maven JVM

reuseForks

true

false starts a JVM per test class

argLine

JVM options of the fork; @{...} placeholders are replaced late

systemPropertyVariables, environmentVariables, workingDirectory

basedir for the directory

System properties, environment and working directory of the fork

forkedProcessTimeoutInSeconds

surefire.timeout

Kill the fork after this many seconds (0 waits forever)

runOrder, runOrderRandomSeed

surefire.runOrder, surefire.runOrder.random.seed

filesystem

Order of the test classes

properties (configurationParameters)

JUnit Platform configuration parameters

includeJUnit5Engines, excludeJUnit5Engines

surefire.includeJUnit5Engines, surefire.excludeJUnit5Engines

Engine ids to include or exclude

redirectTestOutputToFile

maven.test.redirectTestOutputToFile

false

Write the test output to a file in the reports directory

reportsDirectory

${project.build.directory}/surefire-reports

Where reports are written

trimStackTrace

false

Show only the lines in the test in report stack traces

useModulePath

surefire.useModulePath

true

Use the module path when a module-info.java is present (JDK 9 and later)

additionalClasspathElements, classpathDependencyExcludes

maven.test.additionalClasspath, maven.test.dependency.excludes

Add to, or remove from, the test classpath

dependenciesToScan

Dependencies to search for test classes

enableProcessChecker

surefire.enableProcessChecker

Let forks end early when the Maven process is killed (ping, native or all)

parallel, threadCount and relatives

parallel, threadCount

No effect with the JUnit Platform (see the parallel section)

Source: Surefire plugin: surefire:test

Modular projects (a module-info.java in the main sources) run their tests on the module path by default, with test classes patched into the module; the Surefire page on Java modularity in tests describes the layout and useModulePath=false returns to the classpath. That mode was not run for this page.

Source: Surefire plugin: Java modularity (JPMS) in tests

Maven 3.9.16, 3.10.0 and Maven 4

The project pins Surefire 3.6.0, so the plugin behaved the same on all three Maven versions. A set of six runs on each of Maven 3.9.16, 3.10.0 and 4.0.0-rc-7 (release candidate) gave identical results:

Run Result on all three

Run	Result on all three
`clean test` of `tidepool-engine`	`Tests run: 8, Failures: 0, Errors: 0, Skipped: 0`
`-Dtest='TideEngineTest#flat*'`	`Tests run: 1`
`-Dgroups=slow`	`Tests run: 1`
`-Dtest=TideEngineTest` (without the failIfNoSpecifiedTests switch)	`No tests matching pattern "TideEngineTest" were executed!` in `tidepool-model`
`-DskipTests`	`Tests are skipped.` printed once per module (twice)
reruns of the flaky test, `-Dsurefire.rerunFailingTestsCount=2`	`Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1`
the failing-regression project, `clean test`	`There are test failures.`

clean test of tidepool-engine

Tests run: 8, Failures: 0, Errors: 0, Skipped: 0

-Dtest='TideEngineTest#flat*'

Tests run: 1

-Dgroups=slow

Tests run: 1

-Dtest=TideEngineTest (without the failIfNoSpecifiedTests switch)

No tests matching pattern "TideEngineTest" were executed! in tidepool-model

-DskipTests

Tests are skipped. printed once per module (twice)

reruns of the flaky test, -Dsurefire.rerunFailingTestsCount=2

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1

the failing-regression project, clean test

There are test failures.

What does depend on the Maven version is the plugin version used when the POM does not pin one: 3.5.4 on Maven 3.9.16, 3.6.0 on Maven 3.10.0 and on Maven 4.0.0-rc-7, which also warns with Version not locked for default bindings plugins [...]. A project with only junit-jupiter-api 6.1.3 on Maven 3.9.16, which used Surefire 3.5.4, still ran: the JUnit Platform provider of 3.5.4 ran the Jupiter 6.1.3 test (Tests run: 1). Maven 4 adds before: and after: phases and changes the meaning of the post- phases, which matters for Failsafe more than for Surefire; see Maven 4: Lifecycle.

Surefire has been moving with the Maven releases: the JUnit Platform provider was introduced with 2.22.0 (2018-06-11), 3.0.0 followed on 2023-03-11, the 3.5.x line ran from 2024-08-22 to 2026-05-25, and 3.6.0 appeared on 2026-08-31. Dates are from the Maven Central directory listing.

Source: Surefire plugin: History

The same in Ant and Gradle

None of the following was run for this page; the rows are named from the official documentation of each tool.

Need Maven Gradle Ant

Need	Maven	Gradle	Ant
Run unit tests	`surefire:test` in the `test` phase	the `test` task of the Java plugin, which uses the JUnit Platform when `useJUnitPlatform()` is configured	a `junitlauncher` task in a target of your own
Select one test	`-Dtest=Class#method`	`--tests Class.method` on the test task	`test` elements inside `junitlauncher`, or a property in the build file
Tags	`groups`, `excludedGroups`	`includeTags`, `excludeTags` in `useJUnitPlatform { }`	a `tags` element of the launcher
Parallel classes and forks	JUnit parallel parameters, `forkCount`	`maxParallelForks`, `forkEvery`	`fork` attributes of the task
Reports	`target/surefire-reports`, `surefire-report`	`build/reports/tests` and `build/test-results`	`junitlauncher` listeners such as legacy XML

Run unit tests

surefire:test in the test phase

the test task of the Java plugin, which uses the JUnit Platform when useJUnitPlatform() is configured

a junitlauncher task in a target of your own

Select one test

-Dtest=Class#method

--tests Class.method on the test task

test elements inside junitlauncher, or a property in the build file

Where the books are outdated

The Maven books consulted for this section were written for Maven 2 and Maven 3; material of that era predates Surefire 3.6.0.

Surefire used to pick a JUnit 3, JUnit 4 or TestNG provider from the classpath. Since 3.6.0 a single provider runs every framework through the JUnit Platform; JUnit 4 needs 4.12 or later, TestNG 6.14.3 or later, and JUnit 5 and 6 are first class.
TestNG suites selected with suiteXmlFiles are no longer supported (the parameter is ignored with a warning); the way to select tests is groups and excludedGroups.
parallel and threadCount, documented for the JUnit 4.7 provider and TestNG, have no effect with the JUnit Platform; parallel execution of Jupiter tests is configured through JUnit.
-DskipTests and -Dmaven.test.skip=true are often described as the same switch. They differ (the second also skips compiling the tests and the test JAR), and -DskipTests no longer stops Failsafe.
Selecting methods with -Dtest=Class#method is described for JUnit 4 and TestNG only, which is also what the plugin page still says; it works for Jupiter on 3.6.0.
The coverage tool in the books, EMMA, was replaced by JaCoCo in practice; it is attached to the test JVM through argLine, see Site & Reporting.

Summary of pitfalls

Pin the Surefire version in pluginManagement: Maven 3.9.16 binds 3.5.4, Maven 3.10.0 and Maven 4.0.0-rc-7 bind 3.6.0, and 3.6.0 changes providers and parameters.
Setting includes replaces the default patterns; list the defaults again to keep them.
A class that does not end or start with the default patterns (Check, IT for Failsafe) is not run, and nothing warns you.
-Dtest=SomeTest fails in every module that does not have SomeTest; use -pl, or -Dsurefire.failIfNoSpecifiedTests=false.
-Dtest=!Outer does not exclude Outer$Nested; use !Outer*.
Quote -Dtest and -Dgroups values in the shell.
-Dmaven.test.skip=true also skips building a test-jar; use -DskipTests if another module needs the tests.
A plain <argLine> removes the JaCoCo agent; write @{argLine} in front of your own options.
${surefire.forkNumber} alone as a value is lost and becomes null; put text around it.
parallel and threadCount do nothing for JUnit Platform tests; use JUnit parallel parameters.
suiteXmlFiles is ignored on 3.6.0, so a TestNG build silently runs every test.
-DfailIfNoTests, failOnFlakeCount and rerunFailingTestsCount need to be set deliberately: the defaults accept a module without tests and hide a flaky one.
Reports name test methods, not display names, unless the JUnit 5 reporter extension is configured.

Maven Testing with Surefire

Where Surefire sits

The example project

One provider for every framework: Surefire 3.6.0

How the engine is chosen

TestNG and its suite files

Which classes run: includes and excludes

Running a subset from the command line: -Dtest

Tags and groups

Skipping tests

When tests fail

Flaky tests: rerunning failures

The forked JVM

One fork, one schema: the forkNumber placeholder

argLine and JaCoCo

Timeouts

Run order and parallel execution

Reports and output

Parameters at a glance

Maven 3.9.16, 3.10.0 and Maven 4

The same in Ant and Gradle

Where the books are outdated

Summary of pitfalls

References