Gradle Testing JVM Projects

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

This page is written against Gradle 9.8.1 (the current release on 2026-10-16). Every example was run through the Gradle wrapper on macOS with JDK 26 (Oracle java version "26"), which is the JVM that runs the daemon and that compiles the code, apart from the one example that moves the tests to another JDK. The Kotlin DSL is shown first and the Groovy DSL second wherever the syntax differs, and the tidepool build was run in both DSLs with the same commands; the console output is the same apart from timings and from the order in which independent tasks are reported. Every example ran with the configuration cache on (org.gradle.configuration-cache=true in gradle.properties) and reported no configuration cache problems, so all of them are compatible with it (a run with --warning-mode all printed no deprecation warning for the build of this page). The scratch project pointed its wrapper at a local copy of the distribution archive; the properties file that a real project commits is

distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionSha256Sum=dce76f55f8e251a3a1f130eb120f30b3d271de2b76c9b0729d316b5a1b6dc01f
distributionUrl=https\://services.gradle.org/distributions/gradle-9.8.1-bin.zip
networkTimeout=10000
retries=0
retryBackOffMs=500
validateDistributionUrl=true
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists

In the listings the lines about the configuration cache (Calculating task graph ..., Reusing configuration cache., Configuration cache entry stored. and the like) are left out, and the project directory of the scratch copy is replaced by /work/tidepool. A line consisting of ... marks output that was cut. The JVM default locale on the machine was Spanish; the Checkstyle examples set org.gradle.jvmargs=-Duser.language=en -Duser.country=US in gradle.properties, because otherwise Checkstyle prints its messages in Spanish, and no other output on this page depends on the locale. Features newer than Gradle 8.14 name their release. Gradle 8.14 was not installed here, so what differs in it comes from the official documentation and release notes and is labelled "not run here".

Testing is where the three build tools differ least in what they do and most in how they are told to do it. All of them compile the test sources against the production classes, start a JVM, hand the test classes to a framework and turn the result into a report that CI servers can read. Ant has no idea what a test is until you call the <junit> or <junitlauncher> task (see Testing with JUnit & junitlauncher); Maven binds the Surefire plugin to the test phase and the Failsafe plugin to integration-test (see Maven Testing with Surefire and Maven Integration Testing with Failsafe). Gradle has a task type, Test, that every JVM plugin instantiates for you, plus a declarative layer on top of it, the JVM Test Suite plugin, that is the recommended way to describe more than one kind of test. This page is about both layers, about the test frameworks Gradle can drive, about the reports it writes and about the plugins that measure and police tests: JaCoCo for coverage, Checkstyle and PMD for static checks. The Java plugins whose source sets the tests live in are on Gradle Java Projects; choosing the JDK that runs the tests is on Gradle Toolchains; testing a Gradle plugin with TestKit is on Testing & Publishing Plugins. For the language-level view of testing see Java Testing, Kotlin Testing and Groovy Testing with Spock & Geb.

The pieces: source set, Test task, test suite

Three objects take part in every test run, and most of the confusion about Gradle testing comes from not keeping them apart.

Object What it is Where it comes from

Source set test

A directory tree (src/test/java, src/test/resources) and the compile task, classes directory and configurations (testImplementation, testRuntimeOnly, testCompileClasspath, testRuntimeClasspath) derived from its name.

The java plugin creates it. Its classpath contains the production classes, which is why tests can call main code without declaring a dependency.

Test task test

A task that starts a separate JVM (the test worker), loads the test classes in it, runs them with a test framework and writes the results. Its inputs are the test classes, the runtime classpath and its own settings; its outputs are the XML results and the HTML report.

Registered for each suite, so test for the default suite, integrationTest for a suite of that name, and so on.

Test suite test (type JvmTestSuite)

A named bundle of sources, dependencies, testing framework and one or more targets, each backed by a Test task.

The JVM Test Suite plugin, applied by the java plugin. It is the declarative front end of the two objects above.

Before 7.3 (2021) only the first two existed, and a second kind of tests meant a hand-made source set, configurations and a Test task, as shown on Gradle Java Projects. The JVM Test Suite plugin was introduced in Gradle 7.3 as an incubating feature and does the same in a few declarative lines. The java-test-fixtures plugin (Gradle 5.6) shares test helpers between projects, JUnit 5 support arrived in Gradle 4.6, and the sections below add what came later: richer HTML reports (9.3), non-class-based tests (9.4) and the changes of Gradle 9.0 that made the conventions of Test tasks explicit.

Source: Gradle User Manual: Testing in Java & JVM projects, JVM Test Suite plugin: terminology, Gradle 7.3 release notes, Gradle 5.6 release notes, Gradle 4.6 release notes

The example build

The running example is tidepool, the small tide prediction toolkit of the other pages of this section. For testing it has four subprojects: tidepool-model (records plus test fixtures), tidepool-engine (the TideEngine and a CSV StationReader, with three kinds of tests), tide-test-engine (a tiny JUnit Platform engine used in the section on non-class-based tests) and tidepool-reports (a project that only aggregates reports). The tree of the files that matter for testing, build output left out (the build.gradle.kts files are build.gradle in the Groovy DSL):

tidepool
|-- build.gradle.kts
|-- settings.gradle.kts
|-- gradle.properties
|-- gradle/libs.versions.toml
|-- data/stations.csv
|-- tidepool-model
|   |-- build.gradle.kts
|   `-- src
|       |-- main/java/io/tidepool/model/...
|       |-- test/java/io/tidepool/model/HeightTest.java
|       `-- testFixtures/java/io/tidepool/model/Stations.java
|-- tidepool-engine
|   |-- build.gradle.kts
|   `-- src
|       |-- main/java/io/tidepool/engine/{StationReader,TideEngine}.java
|       |-- test/java/io/tidepool/engine/{StationReaderTest,TideEngineTest}.java
|       |-- integrationTest/java/io/tidepool/engine/StationReaderIT.java
|       `-- goldenTest/definitions/{diurnal/K1.tide,semidiurnal/M2.tide}
|-- tide-test-engine
|   |-- build.gradle.kts
|   `-- src/main/{java/io/tidepool/testengine/TideTestEngine.java,resources/META-INF/services/...}
`-- tidepool-reports
    `-- build.gradle.kts

The settings file, the version catalog and gradle.properties are the same as on the other pages (repositories declared once, versions in the catalog), reduced to what this page needs:

Kotlin DSL: settings.gradle.kts
dependencyResolutionManagement {
    repositoriesMode = RepositoriesMode.FAIL_ON_PROJECT_REPOS
    repositories {
        mavenCentral()
    }
}

rootProject.name = "tidepool"

include("tidepool-model", "tidepool-engine", "tide-test-engine", "tidepool-reports")
Groovy DSL: settings.gradle
dependencyResolutionManagement {
    repositoriesMode = RepositoriesMode.FAIL_ON_PROJECT_REPOS
    repositories {
        mavenCentral()
    }
}

rootProject.name = 'tidepool'

include 'tidepool-model', 'tidepool-engine', 'tide-test-engine', 'tidepool-reports'
gradle.properties
org.gradle.configuration-cache=true
group=io.tidepool
version=1.0.0
gradle/libs.versions.toml (the same for both DSLs; highlight.js has no TOML grammar, so it is shown as plain text)
[versions]
commons-csv = "1.14.1"
junit = "6.1.3"

[libraries]
commons-csv = { module = "org.apache.commons:commons-csv", version.ref = "commons-csv" }
junit-platform-engine = { module = "org.junit.platform:junit-platform-engine" }
junit-bom = { module = "org.junit:junit-bom", version.ref = "junit" }

The tests themselves are plain JUnit Jupiter. The engine’s unit tests, in tidepool-engine/src/test/java/io/tidepool/engine/TideEngineTest.java, use a parameterized test and the Stations fixture of the model project:

package io.tidepool.engine;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;

import io.tidepool.model.Stations;
import java.time.Instant;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;

class TideEngineTest {
    @Test
    void flatWaterStaysFlat() {
        assertEquals(0.0, TideEngine.predict(Stations.flatWater(), Instant.EPOCH).metres());
    }

    @Test
    void pureM2StartsAtItsAmplitude() {
        assertEquals(1.0, TideEngine.predict(Stations.pureM2(), Instant.EPOCH).metres(), 1e-9);
    }

    @ParameterizedTest(name = "t = {0} s")
    @ValueSource(longs = {0, 3_600, 21_600, 43_200, 86_400})
    void heightNeverExceedsTheAmplitude(long epochSecond) {
        double metres = TideEngine.predict(Stations.pureM2(), Instant.ofEpochSecond(epochSecond)).metres();
        assertTrue(Math.abs(metres) <= 1.0 + 1e-9, "height " + metres);
    }
}

StationReaderTest parses a text block of CSV and has one test tagged @Tag("slow") that parses 20,000 stations; HeightTest in the model project checks a record and the fixture. The whole project was built with ./gradlew check (after clean in the Kotlin DSL copy) and ended with BUILD SUCCESSFUL in both DSLs.

The Test task

A Test task does five things, and each is a place where a build can be configured or can go wrong.

  1. It resolves the test runtime classpath and the directories of compiled test classes (testClassesDirs). Both are properties of the task; for the test task of the java plugin they are wired to the test source set.

  2. It starts a JVM, the test worker (the console calls it Gradle Test Executor N), with the settings of the task: maxHeapSize, jvmArgs, systemProperty, environment, workingDir (the project directory by default), the javaLauncher (the JDK) and so on.

  3. In that JVM, the tests are detected by inspecting the compiled test classes, with criteria that depend on the framework: for JUnit 3 and 4 a class that inherits from TestCase, carries @RunWith or has an @Test method; for TestNG the methods annotated with @Test. scanForTestClasses = false switches the scan off so that only the include and exclude patterns select classes. With the JUnit Platform the scan setting has no effect: only include and exclude filter the classes, and the engines found on the classpath do the discovery. Abstract classes are not executed.

  4. It executes the tests and streams the events to the build, which prints them according to testLogging, writes the XML result files and the HTML report, and fails the task when a test failed (unless ignoreFailures is true).

  5. It records inputs and outputs, so that a second run with nothing changed is UP-TO-DATE.

The properties used most often:

Property or option Effect

useJUnitPlatform(), useJUnit(), useTestNG()

Selects the framework of a hand-made Test task. With the test suite DSL you use useJUnitJupiter(), useJUnit(), useTestNG() or useSpock() on the suite instead, which also adds the framework’s dependencies.

maxHeapSize, minHeapSize, jvmArgs, jvmArgumentProviders, enableAssertions

JVM settings of the worker. Assertions (-ea) are on by default.

systemProperty(name, value), systemProperties, environment(name, value), workingDir

Properties, environment variables and working directory of the worker. System properties of the Gradle process are not passed on, so only the ones set here reach the worker; environment variables are: the worker starts with the environment of the build process, and environment adds to it or overrides entries.

include, exclude, filter { ... }, --tests

Select which classes or methods run (see "Selecting tests" below).

maxParallelForks, forkEvery

How many worker JVMs run at the same time and after how many test classes a worker is replaced.

failFast, --fail-fast, ignoreFailures

Stop at the first failed test; let the build go on after failures.

failOnNoDiscoveredTests

Since Gradle 9.0, fail when there are test sources but no test was discovered (default true).

testLogging { ... }

What is printed to the console while the tests run.

reports { html, junitXml }, binaryResultsDirectory

Where and whether the HTML report and the XML result files are written.

javaLauncher

The JDK that runs the worker (a toolchain launcher).

testDefinitionDirs

Directories with non-class-based test definition files (incubating, Gradle 9.4).

dryRun, --test-dry-run

Incubating: report the tests without executing them.

The command line options of the task are listed by help --task:

./gradlew :tidepool-engine:help --task test
> Task :tidepool-engine:help
Detailed task information for test

Paths
     :tide-test-engine:test (registered by plugin 'org.gradle.jvm-test-suite')
     :tidepool-engine:test (registered by plugin 'org.gradle.jvm-test-suite')
     :tidepool-model:test (registered by plugin 'org.gradle.jvm-test-suite')

Type
     Test (org.gradle.api.tasks.testing.Test)

Options
     --debug-jvm     Enable debugging for the test process. The process is started suspended and listening on port 5005.
     --no-debug-jvm     Disables option --debug-jvm.
     --fail-fast     Stops test execution after the first failed test.
     --no-fail-fast     Disables option --fail-fast.
     --test-dry-run     Simulate test execution.
     --no-test-dry-run     Disables option --test-dry-run.
     --tests     Sets test class or method name to be included (in addition to the test task filters), '*' is supported.
     --rerun     Causes the task to be re-run even if up-to-date.

Description
     Runs the test suite.

Group
     verification

BUILD SUCCESSFUL in 847ms

--debug-jvm starts the worker suspended and waits for a debugger on port 5005, so it was not run here; the IDE integrations use the same mechanism. Running test the first time on the example prints the events of the default testLogging of the build (configured below) and the report task of JaCoCo:

./gradlew --configuration-cache :tidepool-engine:test
> Task :tidepool-engine:processTestResources NO-SOURCE
> Task :tidepool-engine:processResources NO-SOURCE
> Task :tidepool-model:processTestFixturesResources NO-SOURCE
> Task :tidepool-model:processResources NO-SOURCE
> Task :tidepool-model:compileJava
> Task :tidepool-model:classes
> Task :tidepool-model:jar
> Task :tidepool-model:compileTestFixturesJava
> Task :tidepool-model:testFixturesClasses
> Task :tidepool-model:testFixturesJar
> Task :tidepool-engine:compileJava
> Task :tidepool-engine:classes
> Task :tidepool-engine:compileTestJava
> Task :tidepool-engine:testClasses
> Task :tidepool-engine:test
StationReaderTest > parsesManyStations() PASSED
StationReaderTest > parsesOneStation() PASSED
TideEngineTest > flatWaterStaysFlat() PASSED
TideEngineTest > heightNeverExceedsTheAmplitude(long) > t = 0 s PASSED
TideEngineTest > heightNeverExceedsTheAmplitude(long) > t = 3600 s PASSED
TideEngineTest > heightNeverExceedsTheAmplitude(long) > t = 21600 s PASSED
TideEngineTest > heightNeverExceedsTheAmplitude(long) > t = 43200 s PASSED
TideEngineTest > heightNeverExceedsTheAmplitude(long) > t = 86400 s PASSED
TideEngineTest > pureM2StartsAtItsAmplitude() PASSED
> Task :tidepool-engine:jacocoTestReport

BUILD SUCCESSFUL in 2s
8 actionable tasks: 8 executed

Two things in that listing are worth noticing. The test fixtures of the model project (compileTestFixturesJava, testFixturesJar) were built because the engine’s tests depend on them, and the engine’s test task ran after its inputs were ready without the build script ordering anything: task dependencies come from the dependencies of the classpath. A second run with nothing changed does not run the tests at all, which is the incremental build at work (see Incremental Build). The options --rerun (rerun this one task) and the task cleanTest (delete its outputs) force a run:

./gradlew --configuration-cache :tidepool-engine:test
./gradlew --configuration-cache :tidepool-engine:test --rerun
./gradlew --configuration-cache :tidepool-engine:cleanTest :tidepool-engine:test
> Task :tidepool-engine:test UP-TO-DATE
BUILD SUCCESSFUL in 730ms

> Task :tidepool-engine:test
BUILD SUCCESSFUL in 3s

> Task :tidepool-engine:cleanTest
> Task :tidepool-engine:test
BUILD SUCCESSFUL in 3s

(three separate runs; only the lines about the test task and the result are shown). --rerun applies to the one task it follows, unlike --rerun-tasks, which reruns every task of the build, compilation included. Do not make a Test task always run by switching up-to-date checking off (outputs.upToDateWhen { false }); use --rerun when you need it, because the build cache and the caching of the other tasks depend on the task being well behaved.

Source: Gradle User Manual: Test execution, Test detection, DSL reference: Test, Forcing tests to run, Debugging when running tests

The JUnit Platform and its launcher

Gradle runs JUnit 5 and 6 (JUnit Jupiter) on the JUnit Platform. Since Gradle 4.6 a Test task switches to the platform with useJUnitPlatform(). What the task needs on the test runtime classpath is more than the Jupiter API: it needs a Jupiter engine, and it needs the JUnit Platform launcher, which is the entry point that Gradle calls inside the worker. The launcher and the engines must have versions that belong together, which is why the JUnit team publishes a bill of materials (org.junit:junit-bom).

Older builds got the launcher for free: Gradle loaded some test framework classes from its own distribution. That behavior was deprecated in Gradle 8.x ("Relying on automatic test framework implementation dependencies", announced for removal in 9.0.0) because it caused version conflicts with the test classpath, and in Gradle 9 it is gone. A hand-written Test task therefore has to declare the launcher. The following engine build, which wires the framework by hand, shows what happens when it does not (the build was otherwise the same as the one above; this is a scratch variation):

Kotlin DSL: tidepool-engine/build.gradle.kts (scratch variation, wired by hand)
plugins {
    `java-library`
}

dependencies {
    api(project(":tidepool-model"))
    implementation(libs.commons.csv)
    testImplementation(testFixtures(project(":tidepool-model")))
    testImplementation(platform("org.junit:junit-bom:6.1.3"))
    testImplementation("org.junit.jupiter:junit-jupiter")
}

tasks.withType<JavaCompile>().configureEach {
    options.release = 21
}

tasks.test {
    useJUnitPlatform()
}
Groovy DSL: tidepool-engine/build.gradle (scratch variation, wired by hand)
plugins {
    id 'java-library'
}

dependencies {
    api project(':tidepool-model')
    implementation libs.commons.csv
    testImplementation testFixtures(project(':tidepool-model'))
    testImplementation platform('org.junit:junit-bom:6.1.3')
    testImplementation 'org.junit.jupiter:junit-jupiter'
}

tasks.withType(JavaCompile).configureEach {
    options.release = 21
}

tasks.test {
    useJUnitPlatform()
}
> Task :tidepool-engine:test FAILED

Gradle Test Executor 27 FAILED
    org.gradle.api.internal.tasks.testing.TestSuiteExecutionException at SuiteTestDefinitionProcessor.java:45
        Caused by: org.gradle.api.internal.tasks.testing.RequiresTestFrameworkTestDefinitionProcessor$TestFrameworkNotAvailableException at JUnitPlatformTestDefinitionProcessor.java:101

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':tidepool-engine:test' (registered by plugin 'org.gradle.jvm-test-suite').
> Test process encountered an unexpected problem.
   > Could not start Gradle Test Executor 27.
      > Failed to load JUnit Platform.  Please ensure that all JUnit Platform dependencies are available on the test's runtime classpath, including the JUnit Platform launcher.

* Try:
> Check common problems https://docs.gradle.org/9.8.1/userguide/java_testing.html#sec:java_testing_troubleshooting.
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights from a Build Scan (powered by Develocity).
> Get more help at https://help.gradle.org.

BUILD FAILED in 1s

The message names the fix: add testRuntimeOnly("org.junit.platform:junit-platform-launcher") (Groovy: testRuntimeOnly 'org.junit.platform:junit-platform-launcher') below the Jupiter dependency; the bill of materials supplies the version. With that line the same build ended with BUILD SUCCESSFUL in both DSLs. The error is also the reason the next section prefers test suites: a suite adds the framework, the launcher and the version together, so this class of mistake cannot occur.

Source: Gradle User Manual: Using JUnit 5, Upgrading within 8.x: test framework implementation dependencies, Troubleshooting: test dependencies are missing, Gradle 4.6 release notes (JUnit 5 support)

The JVM Test Suite plugin

A test suite is declared under the testing extension. The java plugin has already created the suite test, which is the only one that sees the production code automatically (its source set output and the implementation dependencies of the project); every other suite has to ask for it. In tidepool-engine/build.gradle.kts the suite section is

Kotlin DSL: tidepool-engine/build.gradle.kts (excerpt)
testing {
    suites {
        named<JvmTestSuite>("test") {
            useJUnitJupiter(libs.versions.junit)
        }

        register<JvmTestSuite>("integrationTest") {
            useJUnitJupiter(libs.versions.junit)
            dependencies {
                implementation(project())
            }
            targets.all {
                testTask.configure {
                    shouldRunAfter(tasks.test)
                }
            }
        }
        // the goldenTest suite is shown in the section on non-class-based tests
    }
}

tasks.check {
    dependsOn(testing.suites.named("integrationTest"), testing.suites.named("goldenTest"))
}
Groovy DSL: tidepool-engine/build.gradle (excerpt)
testing {
    suites {
        test {
            useJUnitJupiter(libs.versions.junit)
        }

        integrationTest(JvmTestSuite) {
            useJUnitJupiter(libs.versions.junit)
            dependencies {
                implementation project()
            }
            targets.all {
                testTask.configure {
                    shouldRunAfter tasks.test
                }
            }
        }
        // the goldenTest suite is shown in the section on non-class-based tests
    }
}

tasks.check {
    dependsOn testing.suites.integrationTest, testing.suites.goldenTest
}

The pieces, in the order of the listing:

  • useJUnitJupiter(version) selects the framework and adds junit-jupiter and the launcher to the suite’s configurations, at the version given. Without an argument the plugin uses its built-in default, which is not necessarily the newest JUnit; in Gradle 9.8.1 the default is JUnit Jupiter 5.12.2 (a dependencies report of a project that called useJUnitJupiter() without argument resolved junit-jupiter:5.12.2, platform 1.12.2). The Gradle 9.0 upgrade notes name the defaults of that release (JUnit Jupiter 5.12.2, TestNG 7.11.0, Spock 2.3). Pass the version, or a provider such as libs.versions.junit, so that the framework is pinned in the version catalog.

  • dependencies { implementation(project()) } adds the project’s own production classes to the suite. project() with no argument means "this project"; in the test suite this is implicit. The dependencies block inside a suite is scoped to the suite, so implementation, compileOnly, runtimeOnly and annotationProcessor there mean the integrationTestImplementation and friends, and it is equivalent to using the long names in the top-level block.

  • targets.all { testTask.configure { ... } } configures the Test task of every target of the suite. Today a suite has exactly one target and its task has the suite’s name; the structure leaves room for several targets, for example one per JDK.

  • shouldRunAfter(tasks.test) is only an ordering hint: the slower integration tests start after the unit tests when both are in the task graph, but nothing makes one a prerequisite of the other. It is a soft rule that Gradle may break to keep workers busy: in the clean check run of this page integrationTest started before test. mustRunAfter is the strict form.

  • The suite does not attach itself to check. A suite other than test is not part of the verification lifecycle until the build says so, which is the last block above. Forgetting it is the classic bug: the task exists, the tests pass when it is run by name, and no CI build that runs build ever executes it. The dry run of check shows the wiring after the line was added (only the engine tasks are shown):

./gradlew --configuration-cache :tidepool-engine:check --dry-run
:tidepool-engine:goldenTestClasses SKIPPED
:tidepool-engine:goldenTest SKIPPED
:tidepool-engine:testClasses SKIPPED
:tidepool-engine:test SKIPPED
:tidepool-engine:integrationTestClasses SKIPPED
:tidepool-engine:integrationTest SKIPPED
:tidepool-engine:jacocoTestCoverageVerification SKIPPED
:tidepool-engine:jacocoTestReport SKIPPED
:tidepool-engine:check SKIPPED

Before the dependsOn line was added, the dry run of check listed test but neither integrationTest nor goldenTest. The verification group lists the tasks the suites created:

Verification tasks
------------------
check - Runs all checks.
goldenTest - Runs the golden test suite.
integrationTest - Runs the integration test suite.
jacocoTestCoverageVerification - Verifies code coverage metrics based on specified rules for the test task.
jacocoTestReport - Generates code coverage report for the test task.
test - Runs the test suite.

The integration test itself reads the shipped data file through the system property set in the common Test configuration (see "Logging, failures and fail-fast" below, which shows the systemProperty line). The integration test source lives in the default location for a suite, src/integrationTest/java, with no further configuration:

./gradlew --configuration-cache :tidepool-engine:integrationTest
> Task :tidepool-engine:integrationTest
StationReaderIT > readsTheThreeShippedStations() PASSED

BUILD SUCCESSFUL in 6s

(only the task and its result lines are shown). A suite also owns its own results and report: the integration test run above wrote build/test-results/integrationTest/ and build/reports/tests/integrationTest/, next to the test directories, so reports of different kinds never overwrite each other.

More of what the suite DSL offers, from the manual:

  • Sources: sources { java { setSrcDirs(listOf("src/it/java")) } } (Groovy: srcDirs = ['src/it/java']) moves a suite’s source directory; the default is src/<suite>/java. A suite’s Test task can be configured in targets.all { testTask.configure { ... } }, as the manual does for maxHeapSize.

  • Dependencies reaching other suites or projects: implementation(project()) for the own project, implementation(project(":other")) or the notation testFixtures(project(...)) for others.

  • Sharing configuration between suites: a block applied to every suite (withType<JvmTestSuite> { ... } in the Kotlin DSL, configureEach { ... } in Groovy), a filtered matching { ... }.configureEach { ... }, or a configuration block extracted to a variable (the three methods of the manual).

  • The frameworks: useJUnitJupiter(), useJUnit() (JUnit 4), useTestNG() and useSpock(); any suite other than the built-in test works as if useJUnitJupiter() had been called, and test must always name its framework.

  • Outgoing variants: each suite publishes its results as a variant that the aggregation plugins consume (see below).

The manual’s older text mentions a testType property of JvmTestSuite; it was removed in Gradle 8.13 (see "Gradle 8.14 and Gradle 9"), so builds should not set it.

Source: JVM Test Suite plugin: declare an additional test suite, Configure dependencies of a test suite, Reference project outputs, Configure source directories, Sharing configuration, Upgrading to Gradle 9: default versions of testing frameworks, Upgrading within 8.x: changes to JvmTestSuite

A hand-made Test task, and the Gradle 9 change

The suite is a convenience over the Test task; a builder who registers a task by hand still can. Until Gradle 8, a Test task registered with no configuration inherited the test classes directories and classpath of the test source set by convention, and it ran the same tests again, which hid the missing configuration. The convention was deprecated in 8.1 and removed in Gradle 9.0.0, and the notes warn that builds relying on it "will silently stop executing tests". This is what the scratch project did on 9.8.1:

Kotlin DSL
// Created without any configuration: since Gradle 9.0 it has no test classes and no classpath.
tasks.register<Test>("otherTest")
Groovy DSL
// Created without any configuration: since Gradle 9.0 it has no test classes and no classpath.
tasks.register('otherTest', Test)
> Task :otherTest NO-SOURCE
BUILD SUCCESSFUL in 438ms

The task is green, nothing ran, and nothing says so unless the NO-SOURCE label is noticed. To run extra tests in a Test task, give it testClassesDirs and classpath (taken from the suite: testClassesDirs = files(test.map { it.sources.output.classesDirs }) and classpath = files(test.map { it.sources.runtimeClasspath }) in the notes' example), or better, add a target to the suite through the testing extension.

Source: Upgrading to Gradle 9: Test tasks may no longer execute expected tests, Upgrading within 8.x: conventions for custom Test tasks

Test frameworks: Jupiter, JUnit 4, TestNG and Spock

The suite DSL names the framework; the engine that runs the tests is whatever the JUnit Platform finds on the runtime classpath. This decides which tests Gradle sees. A scratch project with its own small build.gradle.kts (and build.gradle) tested all four ways at once:

Kotlin DSL: legacy/build.gradle.kts
plugins {
    `java-library`
}

tasks.withType<JavaCompile>().configureEach {
    options.release = 21
}

testing {
    suites {
        // Jupiter tests plus old JUnit 4 tests in the same task, through the Vintage engine.
        named<JvmTestSuite>("test") {
            useJUnitJupiter("6.1.3")
            dependencies {
                implementation("junit:junit:4.13.2")
                runtimeOnly("org.junit.vintage:junit-vintage-engine")
            }
        }

        // A suite that is plain JUnit 4 and runs through the old runner.
        register<JvmTestSuite>("junit4Test") {
            useJUnit("4.13.2")
        }

        register<JvmTestSuite>("testNgTest") {
            useTestNG("7.12.0")
        }
    }
}

tasks.withType<Test>().configureEach {
    testLogging {
        events("passed", "skipped", "failed")
    }
}

tasks.check {
    dependsOn(testing.suites.named("junit4Test"), testing.suites.named("testNgTest"))
}
Groovy DSL: legacy/build.gradle
plugins {
    id 'java-library'
}

tasks.withType(JavaCompile).configureEach {
    options.release = 21
}

testing {
    suites {
        // Jupiter tests plus old JUnit 4 tests in the same task, through the Vintage engine.
        test {
            useJUnitJupiter('6.1.3')
            dependencies {
                implementation 'junit:junit:4.13.2'
                runtimeOnly 'org.junit.vintage:junit-vintage-engine'
            }
        }

        // A suite that is plain JUnit 4 and runs through the old runner.
        junit4Test(JvmTestSuite) {
            useJUnit('4.13.2')
        }

        testNgTest(JvmTestSuite) {
            useTestNG('7.12.0')
        }
    }
}

tasks.withType(Test).configureEach {
    testLogging {
        events 'passed', 'skipped', 'failed'
    }
}

tasks.check {
    dependsOn testing.suites.junit4Test, testing.suites.testNgTest
}

The sources are one JUnit 4 class (org.junit.Test, org.junit.Assert) and one Jupiter class under src/test/java, one JUnit 4 class under src/junit4Test/java and one TestNG class with a @DataProvider under src/testNgTest/java. check ran all three tasks:

> Task :junit4Test
PlainJUnit4Test > runsWithTheJUnit4Runner PASSED
> Task :testNgTest
Gradle suite > Gradle test > legacy.TestNgTest > adds[0](1, 1, 2) PASSED
Gradle suite > Gradle test > legacy.TestNgTest > adds[1](2, 2, 4) PASSED
> Task :test
NewStyleTest > jupiterTest() PASSED
OldStyleTest > junit4TestRunsOnTheJUnitPlatform PASSED
> Task :check
BUILD SUCCESSFUL in 14s

What each line of the build does:

  • JUnit 4 with the Vintage engine. The junit-vintage-engine runs JUnit 3 and 4 tests inside the JUnit Platform, so one Test task runs a codebase that is migrating (the book’s era and most existing code bases are JUnit 4). It needs the JUnit 4 library on the compile classpath of the tests (implementation("junit:junit:4.13.2")) and the Vintage engine at runtime. The bill of materials supplies the Vintage version. The sibling test class, NewStyleTest, is Jupiter. Both appear in the test task output above.

  • Without the Vintage engine, JUnit 4 tests silently disappear. With the runtimeOnly line removed, the same test task printed NewStyleTest > jupiterTest() PASSED only and still ended with BUILD SUCCESSFUL: the platform has no engine for @org.junit.Test, and one class with tests discovered is enough for the check described next to pass.

  • useJUnit("4.13.2") is the original JUnit 4 runner (no JUnit Platform). It adds junit:junit itself, runs @Test classes of JUnit 4 and supports categories (useJUnit { includeCategories("...") }).

  • useTestNG("7.12.0") runs TestNG and adds it to the suite. In the console, TestNG tests are named after the suite and test of TestNG (Gradle suite > Gradle test > legacy.TestNgTest), and data-provider invocations are listed with their index and parameters (adds[0](1, 1, 2)). TestNG-specific options (useTestNG { suites("..."); includeGroups("fast") }) are in the manual; Gradle 9.7.0 supports the new way of controlling thread pools in TestNG 7.10 and later.

  • Spock: useSpock() adds the Spock engine, whose default version in Gradle 9 is 2.3, and runs specifications on the JUnit Platform; the Groovy side is covered on Groovy Testing with Spock & Geb and needs the groovy plugin so that the specification classes compile. It was not run here.

The silent drop above has a counterpart that is not silent. When there are test sources and the engine finds no test at all, Gradle 9 fails the task. A scratch project with a single JUnit 4 test class and a Jupiter-only test suite printed:

...
> Task :test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There are test sources present and no filters are applied, but the test task did not discover any tests to execute. This is likely due to a misconfiguration. Please check your test configuration. If this is not a misconfiguration, this error can be disabled by setting the 'failOnNoDiscoveredTests' property to false.

* Try:
> Run with --scan to get full insights from a Build Scan (powered by Develocity).

BUILD FAILED in 627ms
2 actionable tasks: 2 executed
Configuration cache entry stored.

That is the Gradle 9.0 behavior ("test task fails when no tests are discovered"); in Gradle 8 a green build was printed (not run here), and Gradle 8.x announced the change as a deprecation ("Deprecated running test task successfully when no test executed"). The switch is failOnNoDiscoveredTests = false on the task, for a module whose test sources are intentionally empty. The failure appears only when every class is invisible, so mixed projects still need the Vintage line.

Source: Executing legacy tests with JUnit Vintage, Troubleshooting: tests are not detected or executed, Upgrading to Gradle 9: test task fails when no tests are discovered, Upgrading within 8.x: running test task successfully when no test executed, Gradle 9.7.0 release notes: TestNG thread pool factories

Selecting tests

A build with thousands of tests needs ways to run a subset, from the command line while developing and from the build script for the CI stages. Gradle has four mechanisms; use the first one most often.

--tests on the command line
./gradlew --configuration-cache :tidepool-engine:test --tests 'io.tidepool.engine.TideEngineTest.pureM2*'
./gradlew --configuration-cache :tidepool-engine:test --tests '*StationReader*'
./gradlew --configuration-cache :tidepool-engine:test --tests '*flatWater*' --tests '*parsesOne*'
> Task :tidepool-engine:test
TideEngineTest > pureM2StartsAtItsAmplitude() PASSED

> Task :tidepool-engine:test
StationReaderTest > parsesManyStations() PASSED
StationReaderTest > parsesOneStation() PASSED

> Task :tidepool-engine:test
StationReaderTest > parsesOneStation() PASSED
TideEngineTest > flatWaterStaysFlat() PASSED

(three separate runs; only the lines of the test task are shown). The rules are those of the manual: a pattern is a fully qualified class name, optionally followed by a dot and a method name; since Gradle 4.7 a pattern that starts with an upper-case letter is taken as a simple class name (--tests TideEngineTest), and the wildcard * is purely textual and matches dots too, so *.TideEngineTest matches any package. The option can be repeated and all patterns take effect, filters declared in the build script still apply on top of --tests, and a parameterized method is selected by its method name (the manual shows *ParameterizedTest.*[2] for one invocation). Combined with --continuous it reruns a subset after every change. A pattern that selects nothing is an error and not an empty success:

> Task :tidepool-engine:test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':tidepool-engine:test'.
> No tests found for given includes: [NoSuchTest](--tests filter)

* Try:
> Run with --scan to get full insights from a Build Scan (powered by Develocity).

BUILD FAILED in 550ms

The build script has the same filter as an API, filter { includeTestsMatching("...") }, plus excludeTestsMatching, includeTest(className, methodName) and isFailOnNoMatchingTests. The older include "**/IT*.class" and exclude patterns work on class file paths, are still useful with scanForTestClasses = false and are what separates the Ant and Maven (**/*Test.java) worlds from this one. The JUnit Platform has its own, framework-level selection through tags:

Kotlin DSL: tidepool-engine/build.gradle.kts (excerpt)
tasks.test {
    useJUnitPlatform {
        if (providers.gradleProperty("fast").isPresent) {
            excludeTags("slow")
        }
    }
}
Groovy DSL: tidepool-engine/build.gradle (excerpt)
tasks.test {
    useJUnitPlatform {
        if (providers.gradleProperty('fast').present) {
            excludeTags 'slow'
        }
    }
}
./gradlew --configuration-cache :tidepool-engine:test -Pfast
> Task :tidepool-engine:test
StationReaderTest > parsesOneStation() PASSED
TideEngineTest > flatWaterStaysFlat() PASSED
TideEngineTest > heightNeverExceedsTheAmplitude(long) > t = 0 s PASSED
TideEngineTest > heightNeverExceedsTheAmplitude(long) > t = 3600 s PASSED
TideEngineTest > heightNeverExceedsTheAmplitude(long) > t = 21600 s PASSED
TideEngineTest > heightNeverExceedsTheAmplitude(long) > t = 43200 s PASSED
TideEngineTest > heightNeverExceedsTheAmplitude(long) > t = 86400 s PASSED
TideEngineTest > pureM2StartsAtItsAmplitude() PASSED

The -Pfast run executed eight tests; the unfiltered run executes nine, the extra one being parsesManyStations tagged slow. The property is read through the provider API (providers.gradleProperty(...)), which is what keeps the example compatible with the configuration cache: a plain project.hasProperty("fast") would also work, but the provider records the property as an input of the cached configuration. includeTags and excludeTags accept tag expressions of the JUnit Platform, and a task can also pick tests by engine (includeEngines("junit-jupiter")). Task-level selection does not replace separate suites: if the "fast" and "slow" tests need different dependencies, a different working directory or a different JVM, give them different suites.

The fourth mechanism is --test-dry-run (the property dryRun, incubating): it discovers the tests and reports each as skipped without executing it.

./gradlew --configuration-cache :tidepool-engine:test --test-dry-run
> Task :tidepool-engine:test
StationReaderTest > parsesManyStations() SKIPPED
StationReaderTest > parsesOneStation() SKIPPED
TideEngineTest > flatWaterStaysFlat() SKIPPED
TideEngineTest > pureM2StartsAtItsAmplitude() SKIPPED
> Task :tidepool-engine:jacocoTestReport

BUILD SUCCESSFUL in 2s

The parameterized test heightNeverExceedsTheAmplitude is not listed; the dry run reports four of the nine tests. Note also that the JaCoCo report task still ran, because it is finalizedBy of test in this build, and found no coverage data of its own.

Source: Gradle User Manual: Test filtering, Simple name pattern, Fully qualified name pattern, Test grouping, Filtering test engines, DSL reference: Test

Logging, failures and fail-fast

At the default log level (lifecycle) Gradle prints only failed tests, with a short exception format, plus a summary line and the report location; skipped tests and the output of tests appear at --info. The testLogging block of the common Test configuration in tidepool-engine turns events on; it is also the place for the system property used by the integration test and for the heap size of the worker:

Kotlin DSL: tidepool-engine/build.gradle.kts (excerpt)
import org.gradle.api.tasks.testing.logging.TestExceptionFormat

tasks.withType<Test>().configureEach {
    maxHeapSize = "512m"
    systemProperty("tidepool.data", layout.projectDirectory.dir("../data").asFile.path)
    testLogging {
        events("passed", "skipped", "failed")
        exceptionFormat = TestExceptionFormat.FULL
    }
}
Groovy DSL: tidepool-engine/build.gradle (excerpt)
tasks.withType(Test).configureEach {
    maxHeapSize = '512m'
    systemProperty 'tidepool.data', layout.projectDirectory.dir('../data').asFile.path
    testLogging {
        events 'passed', 'skipped', 'failed'
        exceptionFormat = 'full'
    }
}

The logging options are events (started, passed, skipped, failed, standardOut, standardError), exceptionFormat (short, full), showExceptions, showCauses, showStackTraces, showStandardStreams (a shortcut for the two stream events), stackTraceFilters and minGranularity / maxGranularity (which levels of the test hierarchy are shown). Since Gradle 9.7.0 framework start-up failures of TestNG, JUnit 4 and the JUnit Platform (for example an exception thrown from a TestNG test class constructor) are always printed, even when the granularity settings would hide them. To see failure output in a controlled test, a scratch class RoundingTest was added to the engine, with a deliberately wrong assertion, a test that prints to standard output and a @Disabled test:

class RoundingTest {
    @Test
    void halfwayThroughAQuarterCycle() {
        // M2 has a period of 12.42 h, so a quarter cycle is not 3 h: this assertion is wrong on purpose.
        double metres = TideEngine.predict(Stations.pureM2(), Instant.ofEpochSecond(3 * 3600)).metres();
        assertEquals(0.0, metres, 1e-6, "height after three hours");
    }

    @Test
    void writesToStandardOutput() {
        System.out.println("predicting for the pure M2 station");
        assertEquals(1.0, TideEngine.predict(Stations.pureM2(), Instant.EPOCH).metres(), 1e-9);
    }

    @Disabled("waiting for the datum conversion")
    @Test
    void convertsToChartDatum() {
    }
}
./gradlew --configuration-cache :tidepool-engine:test
> Task :tidepool-engine:test FAILED

RoundingTest > convertsToChartDatum() SKIPPED

RoundingTest > halfwayThroughAQuarterCycle() FAILED
    org.opentest4j.AssertionFailedError: height after three hours ==> expected: <0.0> but was: <0.05316710997744344>
        at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1073)
        at app//io.tidepool.engine.RoundingTest.halfwayThroughAQuarterCycle(RoundingTest.java:15)

RoundingTest > writesToStandardOutput() PASSED

StationReaderTest > parsesManyStations() PASSED
...
TideEngineTest > pureM2StartsAtItsAmplitude() PASSED

12 tests completed, 1 failed, 1 skipped

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':tidepool-engine:test'.
> There were failing tests. See the report at: file:///work/tidepool/tidepool-engine/build/reports/tests/test/index.html

* Try:
> Run with --scan to get full insights from a Build Scan (powered by Develocity).

BUILD FAILED in 2s

The failure message is the one of the assertion library, the stack trace is filtered to the first frames of the test, and the task failure points to the HTML report. The other switches, each run in the same scratch project:

  • --fail-fast stops after the first failed test. The remaining tests are reported as skipped: the run printed 4 tests completed, 1 failed, 2 skipped (instead of the 12 of the full run), and StationReaderTest > parsesManyStations() SKIPPED. The build script property is failFast = true.

  • showStandardStreams = true prints what the test wrote while it ran, attached to the test that wrote it:

RoundingTest > writesToStandardOutput() STANDARD_OUT
    predicting for the pure M2 station

RoundingTest > writesToStandardOutput() PASSED

  • ignoreFailures = true lets the build continue after failures; the output shows the failure and BUILD SUCCESSFUL. This is meant for builds that still want their reports aggregated and published; the usual alternative is to keep the default and add --continue on the command line, so that other tasks still run but the build fails. A subtle result of the combination with the up-to-date check: the run with ignoreFailures printed the failure the first time and a second invocation printed > Task :tidepool-engine:test UP-TO-DATE and BUILD SUCCESSFUL, so a red test stays hidden until --rerun.

The XML result files carry the full standard output and error of every test class as well (<system-out>), and since Gradle 9.3 the output stays attached to the test that produced it instead of being merged into the class.

Source: Gradle User Manual: Test logging, DSL reference: TestLoggingContainer, Gradle 9.7.0 release notes: framework initialization failures, Gradle 9.3.0 release notes: test standard output and error changes

Reports and results

Every Test task writes two representations of the same run, plus a binary one that Gradle uses to build them. For the test task of the engine:

tidepool-engine/build/test-results/test/TEST-io.tidepool.engine.StationReaderTest.xml
tidepool-engine/build/test-results/test/TEST-io.tidepool.engine.TideEngineTest.xml
tidepool-engine/build/test-results/test/binary/output-events.bin
tidepool-engine/build/test-results/test/binary/results-generic.bin
tidepool-engine/build/reports/tests/test/index.html
tidepool-engine/build/reports/tests/test/io.tidepool.engine.StationReaderTest/index.html
tidepool-engine/build/reports/tests/test/io.tidepool.engine.TideEngineTest/index.html
tidepool-engine/build/reports/tests/test/io.tidepool.engine.TideEngineTest/heightNeverExceedsTheAmplitude(long)/index.html
...

(the CSS and JavaScript files of the report are left out). The directories are named after the task, so integrationTest and goldenTest write next to test, never over it. The XML is the JUnit XML format that CI servers (Jenkins, GitHub Actions test summaries, GitLab, Azure DevOps) read; one file per test class, with one testcase element per test:

<testsuite name="io.tidepool.engine.TideEngineTest" tests="7" skipped="0" failures="0" errors="0" ...>
  <properties/>
  <testcase name="flatWaterStaysFlat()" classname="io.tidepool.engine.TideEngineTest" time="0.002"/>
  <testcase name="t = 0 s" classname="io.tidepool.engine.TideEngineTest" time="0.006"/>
  ...
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

(the timestamp and hostname attributes are cut). The reports block of the task switches each report on or off and moves it: reports { html.required = false; junitXml.outputLocation = layout.buildDirectory.dir("ci-results") }, and the outputPerTestCase and mergeReruns properties of the junitXml report control how output and reruns of flaky tests are recorded (not run here). The binary results are not meant to be read by hand; TestReport tasks and the Tooling API use them.

What changed in the HTML report over the Gradle 9 line:

  • 9.3.0: a more detailed HTML report. It now shows nested JUnit classes nested under their enclosing class, a parameterized test as a suite of its invocations (paramTest1 containing paramTest1[0], paramTest1[1]), suites of JUnit 4, Jupiter and TestNG as containers of their classes, and no synthetic package level (a class is listed with its fully qualified name). The standard output and error stay with the test that wrote them. The aggregate report distinguishes its sources as separate tabs. The XML keeps its format. The hierarchy in the report corresponds to the one printed by testLogging with minGranularity and maxGranularity, which is the reason a parameterized test appears as heightNeverExceedsTheAmplitude(long) > t = 0 s in the console listing above.

  • 9.4.0: JUnit Platform tests can publish additional data with the TestReporter API (publishEntry("key", "value"), publishFile(...)), which Gradle captures into the HTML report (two extra tabs) and the XML, and a TestMetadataListener can observe it through addTestMetadataListener. Screenshots of failed UI tests are the typical use. This was not run here.

  • 9.6.0: sortable columns in the HTML test report (numeric columns sort descending on the first click, the success rate ascending), which helps to find the failing classes of a large project.

The aggregate and coverage reports are described in the sections below. Build Scans (--scan) show the same results online with timelines; publishing a scan means accepting terms of use and was not done here.

Source: Gradle User Manual: Test reporting, Communicating test results to CI servers and other tools via XML files, Gradle 9.3.0 release notes: test reporting improvements, Gradle 9.4.0 release notes: additional test data capture, Gradle 9.6.0 release notes: sortable columns

Forks, parallelism and the JDK of the tests

A test worker is a JVM, and there are three levers over how many of them run and what they run on.

Kotlin DSL
tasks.withType<Test>().configureEach {
    maxParallelForks = 2   // at most two worker JVMs at the same time
    forkEvery = 1          // replace the worker after every test class
}
Groovy DSL
tasks.withType(Test).configureEach {
    maxParallelForks = 2   // at most two worker JVMs at the same time
    forkEvery = 1          // replace the worker after every test class
}

maxParallelForks (default 1) divides the test classes between that many workers, so several classes run at the same time and each one in its own JVM; a common value is half the number of cores, Runtime.getRuntime().availableProcessors().div(2).coerceAtLeast(1) (not run here). forkEvery (default 0, no limit) restarts the worker after that many classes, which isolates tests that leak state at a high startup cost. The effect was observed with four scratch classes that sleep for 400 ms and print the worker that ran them (org.gradle.test.worker), run with --tests selecting only those four:

(default: one worker)
    AlphaTest ran in worker 39
    BravoTest ran in worker 39
    CharlieTest ran in worker 39
    DeltaTest ran in worker 39

(maxParallelForks = 2)
    AlphaTest ran in worker 2
    BravoTest ran in worker 1
    CharlieTest ran in worker 2
    DeltaTest ran in worker 1

(maxParallelForks = 2 and forkEvery = 1)
    BravoTest ran in worker 40
    AlphaTest ran in worker 41
    DeltaTest ran in worker 42
    CharlieTest ran in worker 43

The number is the value of the org.gradle.test.worker system property, which the manual describes as unique for each test process (and useful for naming files and other resources so that parallel tests do not collide). What matters here is how many different values appear: one, two, and four (a new JVM for every class). The Groovy DSL run printed four distinct workers for the last setting as well. Forks are process level parallelism, between test classes. Inside a worker, JUnit Jupiter can also run classes and methods on several threads: set junit.jupiter.execution.parallel.enabled=true (and junit.jupiter.execution.parallel.mode.default=concurrent) in junit-platform.properties on the test classpath, or as systemProperty of the task; tests that share mutable state or files must then be written for it. Parallel forks need tests that do not share ports, files or databases.

The JDK that runs the workers is independent of the JDK of the daemon and of the one that compiles the code. A test task takes a javaLauncher from the toolchain service (see Gradle Toolchains):

Kotlin DSL: tidepool-engine/build.gradle.kts (scratch variation)
// Run the tests on another JDK than the one that runs Gradle.
tasks.withType<Test>().configureEach {
    javaLauncher = javaToolchains.launcherFor {
        languageVersion = JavaLanguageVersion.of(21)
    }
}
Groovy DSL: tidepool-engine/build.gradle (scratch variation)
// Run the tests on another JDK than the one that runs Gradle.
tasks.withType(Test).configureEach {
    javaLauncher = javaToolchains.launcherFor {
        languageVersion = JavaLanguageVersion.of(21)
    }
}

A scratch test printed Runtime.version().feature() and java.vendor with showStandardStreams on. With org.gradle.java.installations.auto-download=false in gradle.properties (so no JDK could be downloaded) the daemon ran on JDK 26 and the tests ran on the installed JetBrains Runtime 21:

JvmTest > reportsTheJvmThatRunsTheTests() STANDARD_OUT
    tests run on Java 21 (JetBrains s.r.o.)

JvmTest > reportsTheJvmThatRunsTheTests() PASSED

JaCoCo’s agent worked on that JDK too (the report task ran after the tests and the build ended BUILD SUCCESSFUL). The same mechanism tests a library on the oldest and the newest JDK it supports: one suite target per JDK is the use case the suite structure was designed for. A toolchain whose JDK is missing makes the task fail, or downloads the JDK when a resolver is configured (which was disabled here).

Source: DSL reference: Test, maxParallelForks, forkEvery, javaLauncher, Gradle User Manual: Consuming toolchains, Test execution

Test fixtures

Test helpers (builders, sample data, fake servers, custom assertions) are needed by the tests of several projects. Two classic solutions are bad: copying them, and putting them in main, where they ship to users. The Maven answer is a test-jar of the test classes, which does not carry the test-scoped dependencies along (a consumer has to declare them again); Gradle’s answer is the java-test-fixtures plugin, available since Gradle 5.6, which adds a source set testFixtures with its own dependencies, a JAR with the classifier test-fixtures and a feature that other projects can depend on.

Kotlin DSL: tidepool-model/build.gradle.kts
plugins {
    `java-library`
    `java-test-fixtures`
    jacoco
}

tasks.withType<JavaCompile>().configureEach {
    options.release = 21
}

testing {
    suites {
        named<JvmTestSuite>("test") {
            useJUnitJupiter(libs.versions.junit)
        }
    }
}
Groovy DSL: tidepool-model/build.gradle
plugins {
    id 'java-library'
    id 'java-test-fixtures'
    id 'jacoco'
}

tasks.withType(JavaCompile).configureEach {
    options.release = 21
}

testing {
    suites {
        test {
            useJUnitJupiter(libs.versions.junit)
        }
    }
}

(the JaCoCo task wiring that both scripts also contain is shown in the JaCoCo section). The fixtures are ordinary classes under src/testFixtures/java:

package io.tidepool.model;

import java.util.List;

/** Ready-made stations for tests in this and in other projects. */
public final class Stations {
    private Stations() {
    }

    /** A station with a single M2 constituent of amplitude 1 m and phase 0. */
    public static Station pureM2() {
        return new Station("M2", "Pure M2", 0, 0, List.of(new Constituent("M2", 1.0, 0)));
    }

    /** A station without constituents: the sea never moves. */
    public static Station flatWater() {
        return new Station("FLAT", "Flat water", 0, 0, List.of());
    }
}

The plugin gives the project the configurations testFixturesApi, testFixturesImplementation, testFixturesCompileOnly and testFixturesRuntimeOnly; the fixtures automatically see the production classes (an api-like dependency of testFixtures on main), and the project’s own test suite sees the fixtures (the HeightTest of the model uses Stations). Another project consumes the fixtures with the testFixtures(...) notation, as the engine does (testImplementation(testFixtures(project(":tidepool-model"))) in the engine’s dependencies; Groovy: testImplementation testFixtures(project(':tidepool-model'))). The build produced the extra JAR next to the normal one:

./gradlew -q :tidepool-model:build
ls tidepool-model/build/libs
tidepool-model-1.0.0-test-fixtures.jar
tidepool-model-1.0.0.jar

The fixtures are a feature of the same module, with their own capability, which is how dependency resolution and publishing know them (see Variant-aware Resolution & Module Metadata). outgoingVariants prints it:

./gradlew -q :tidepool-model:outgoingVariants --variant testFixturesApiElements
--------------------------------------------------
Variant testFixturesApiElements
--------------------------------------------------
API elements for the 'testFixtures' feature.

Capabilities
    - io.tidepool:tidepool-model-test-fixtures:1.0.0
Attributes
    - org.gradle.category            = library
    - org.gradle.dependency.bundling = external
    - org.gradle.jvm.version         = 21
    - org.gradle.libraryelements     = jar
    - org.gradle.usage               = java-api
Artifacts
    - build/libs/tidepool-model-1.0.0-test-fixtures.jar (artifactType = jar, classifier = test-fixtures)

Secondary Variants (*)
...

When the library is published, the fixtures go along: by convention an artifact with the classifier test-fixtures is published next to the regular ones, and with maven-publish or ivy-publish they are also published as additional variants in Gradle Module Metadata, so that another Gradle project can write testFixtures("group:module:version"). A consumer of a module that publishes no Gradle Module Metadata gets a resolution error that no variant with the capability test-fixtures exists. A project that should not publish its fixtures can skip the two variants (components.java.withVariantsFromConfiguration(configurations["testFixturesApiElements"]) { skip() } and the same for testFixturesRuntimeElements, not run here); publishing is on Publishing. The fixtures' own dependencies are declared with testFixturesApi when the fixture API exposes them (for example a JUnit or AssertJ type in a signature), so that consumers get them at compile time too.

Source: Gradle User Manual: Java test fixtures plugin, Layout, Configurations, Consuming test fixtures, Testing in Java: test fixtures

Non-class-based tests (Gradle 9.4)

Some test frameworks do not describe tests as classes: Cucumber with its .feature files, tools that read XML, YAML or CSV scripts, golden-file tests. They run on the JUnit Platform as test engines that discover tests in files. Before Gradle 9.4 such a build needed a workaround (an empty test class to satisfy Gradle’s class-based discovery, or a JUnit extension). Gradle 9.4.0 added direct support: the Test task takes a property testDefinitionDirs with the directories to scan, gives the engine a DirectorySelector for each of them and reports the tests it finds. The feature is incubating, as are the filtering rules for non-class-based tests.

The example is a tiny engine, tide-test-engine, that reads *.tide files. Each non-comment line name amplitude phase epochSecond expected is one test: it builds a one-constituent station, asks the TideEngine for the height at the instant and compares it with the expected value within 1 mm. The golden files live in tidepool-engine/src/goldenTest/definitions:

# semidiurnal/M2.tide
M2 1.0 0 0 1.0
M2 1.0 90 0 0.0
M2 0.5 0 21600 -0.4972

# diurnal/K1.tide
K1 0.38 210 0 -0.3291
K1 1.0 0 86400 0.9999

The engine is a normal JUnit Platform TestEngine, registered with ServiceLoader through META-INF/services/org.junit.platform.engine.TestEngine (a file whose only line is io.tidepool.testengine.TideTestEngine). It depends on the platform’s engine API, taken from the bill of materials:

Kotlin DSL: tide-test-engine/build.gradle.kts
plugins {
    `java-library`
}

dependencies {
    api(platform(libs.junit.bom))
    api(libs.junit.platform.engine)
    implementation(project(":tidepool-engine"))
}

tasks.withType<JavaCompile>().configureEach {
    options.release = 21
}
Groovy DSL: tide-test-engine/build.gradle
plugins {
    id 'java-library'
}

dependencies {
    api platform(libs.junit.bom)
    api libs.junit.platform.engine
    implementation project(':tidepool-engine')
}

tasks.withType(JavaCompile).configureEach {
    options.release = 21
}
package io.tidepool.testengine;

import io.tidepool.engine.TideEngine;
import io.tidepool.model.Constituent;
import io.tidepool.model.Station;
import java.io.IOException;
import java.io.UncheckedIOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import java.util.List;
import java.util.stream.Stream;
import org.junit.platform.engine.EngineDiscoveryRequest;
import org.junit.platform.engine.ExecutionRequest;
import org.junit.platform.engine.TestDescriptor;
import org.junit.platform.engine.TestExecutionResult;
import org.junit.platform.engine.TestEngine;
import org.junit.platform.engine.UniqueId;
import org.junit.platform.engine.discovery.DirectorySelector;
import org.junit.platform.engine.support.descriptor.AbstractTestDescriptor;
import org.junit.platform.engine.support.descriptor.EngineDescriptor;
import org.junit.platform.engine.support.descriptor.FileSource;

/**
 * A tiny JUnit Platform engine for non-class-based tests: every line of a *.tide file
 * "name amplitude phase epochSecond expected" is one test that compares the prediction of the
 * tide engine for a one-constituent station with the expected height in metres.
 */
public final class TideTestEngine implements TestEngine {
    @Override
    public String getId() {
        return "tide-engine";
    }

    @Override
    public TestDescriptor discover(EngineDiscoveryRequest request, UniqueId uniqueId) {
        EngineDescriptor root = new EngineDescriptor(uniqueId, "Tide golden files");
        for (DirectorySelector selector : request.getSelectorsByType(DirectorySelector.class)) {
            Path dir = selector.getPath();
            try (Stream<Path> files = Files.walk(dir)) {
                files.filter(f -> f.toString().endsWith(".tide")).sorted().forEach(f -> addFile(root, dir, f));
            } catch (IOException e) {
                throw new UncheckedIOException(e);
            }
        }
        return root;
    }

    private static void addFile(EngineDescriptor root, Path dir, Path file) {
        String name = dir.relativize(file).toString().replace('/', '.').replaceAll("\\.tide$", "");
        FileDescriptor container = new FileDescriptor(root.getUniqueId().append("file", name), name, file);
        try {
            List<String> lines = Files.readAllLines(file);
            for (int i = 0; i < lines.size(); i++) {
                String line = lines.get(i).strip();
                if (!line.isEmpty() && !line.startsWith("#")) {
                    container.addChild(new CaseDescriptor(container.getUniqueId().append("line", String.valueOf(i + 1)),
                            file, i + 1, line));
                }
            }
        } catch (IOException e) {
            throw new UncheckedIOException(e);
        }
        root.addChild(container);
    }

    @Override
    public void execute(ExecutionRequest request) {
        var listener = request.getEngineExecutionListener();
        TestDescriptor root = request.getRootTestDescriptor();
        listener.executionStarted(root);
        for (TestDescriptor container : root.getChildren()) {
            listener.executionStarted(container);
            for (TestDescriptor test : container.getChildren()) {
                listener.executionStarted(test);
                try {
                    ((CaseDescriptor) test).run();
                    listener.executionFinished(test, TestExecutionResult.successful());
                } catch (Throwable t) {
                    listener.executionFinished(test, TestExecutionResult.failed(t));
                }
            }
            listener.executionFinished(container, TestExecutionResult.successful());
        }
        listener.executionFinished(root, TestExecutionResult.successful());
    }

    private static final class FileDescriptor extends AbstractTestDescriptor {
        FileDescriptor(UniqueId id, String name, Path file) {
            super(id, name, FileSource.from(file.toFile()));
        }

        @Override
        public Type getType() {
            return Type.CONTAINER;
        }
    }

    private static final class CaseDescriptor extends AbstractTestDescriptor {
        private final String line;

        CaseDescriptor(UniqueId id, Path file, int number, String line) {
            super(id, line, FileSource.from(file.toFile(), org.junit.platform.engine.support.descriptor.FilePosition.from(number)));
            this.line = line;
        }

        @Override
        public Type getType() {
            return Type.TEST;
        }

        void run() {
            String[] p = line.split("\\s+");
            Station station = new Station(p[0], p[0], 0, 0, List.of(new Constituent(p[0], Double.parseDouble(p[1]), Double.parseDouble(p[2]))));
            double actual = TideEngine.predict(station, Instant.ofEpochSecond(Long.parseLong(p[3]))).metres();
            double expected = Double.parseDouble(p[4]);
            if (Math.abs(actual - expected) > 0.001) {
                throw new AssertionError("expected " + expected + " m but the engine predicted " + actual + " m");
            }
        }
    }
}

The golden test suite needs three things, which are the three steps of the manual: the Jupiter platform (so that the launcher is present), the engine on the runtime classpath, and the definition directory. The suite has no Java sources at all, so the task for compiling its sources reports NO-SOURCE:

Kotlin DSL: tidepool-engine/build.gradle.kts (excerpt)
register<JvmTestSuite>("goldenTest") {
    useJUnitJupiter(libs.versions.junit)
    dependencies {
        runtimeOnly(project(":tide-test-engine"))
    }
    targets.all {
        testTask.configure {
            testDefinitionDirs.from("src/goldenTest/definitions")
        }
    }
}
Groovy DSL: tidepool-engine/build.gradle (excerpt)
goldenTest(JvmTestSuite) {
    useJUnitJupiter(libs.versions.junit)
    dependencies {
        runtimeOnly project(':tide-test-engine')
    }
    targets.all {
        testTask.configure {
            testDefinitionDirs.from('src/goldenTest/definitions')
        }
    }
}
./gradlew --configuration-cache :tidepool-engine:goldenTest
> Task :tidepool-engine:goldenTest
diurnal.K1 > K1 0.38 210 0 -0.3291 PASSED
diurnal.K1 > K1 1.0 0 86400 0.9999 PASSED
semidiurnal.M2 > M2 1.0 0 0 1.0 PASSED
semidiurnal.M2 > M2 1.0 90 0 0.0 PASSED
semidiurnal.M2 > M2 0.5 0 21600 -0.4972 PASSED

BUILD SUCCESSFUL in 21s

(the first run, which also built the engine project, took 21 s). The rules the engine and the task agree on show in the output. The name of a non-class-based test container is built from the relative path of the file: semidiurnal/M2.tide becomes semidiurnal.M2, and that is the name filters match, so the --tests option works at file granularity:

./gradlew --configuration-cache :tidepool-engine:goldenTest --tests 'semidiurnal.*'
semidiurnal.M2 > M2 1.0 0 0 1.0 PASSED
semidiurnal.M2 > M2 1.0 90 0 0.0 PASSED
semidiurnal.M2 > M2 0.5 0 21600 -0.4972 PASSED

The manual warns that the conversion from a path to a name is ambiguous when directories or files contain dots or spaces (sub.ExampleTests.xml, sub/ExampleTests.xml and sub/ExampleTests all become sub.ExampleTests), so non-class-based definitions should follow Java package naming for directories and use simple file names. The result files are named after the container as well, in the XML (TEST-diurnal-K1.tide.xml, with classname="diurnal/K1.tide" for its test cases) and in the HTML report. A failing definition behaves like any failing test, with the engine’s own message as the failure. Changing the expected value of the last M2 line to 0.4972 (the sign is wrong) printed:

> Task :tidepool-engine:goldenTest FAILED
diurnal.K1 > K1 0.38 210 0 -0.3291 PASSED
diurnal.K1 > K1 1.0 0 86400 0.9999 PASSED
semidiurnal.M2 > M2 1.0 0 0 1.0 PASSED
semidiurnal.M2 > M2 1.0 90 0 0.0 PASSED
semidiurnal.M2 > M2 0.5 0 21600 0.4972 FAILED
    java.lang.AssertionError: expected 0.4972 m but the engine predicted -0.49717325841664645 m
        at io.tidepool.testengine.TideTestEngine$CaseDescriptor.run(TideTestEngine.java:118)
        at io.tidepool.testengine.TideTestEngine.execute(TideTestEngine.java:77)

5 tests completed, 1 failed

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':tidepool-engine:goldenTest'.
> There were failing tests. See the report at: file:///work/tidepool/tidepool-engine/build/reports/tests/goldenTest/index.html

Use real engines for real work: a Cucumber project registers cucumber-junit-platform-engine as runtimeOnly and points testDefinitionDirs at src/test/resources or wherever the .feature files are, as the 9.4.0 notes show (not run here, since the build of this page has no Cucumber). The manual recommends not mixing class-based and non-class-based tests in one task: separate suites let each have its own engines, dependencies and report. The three-step structure is also how a custom engine for any text-driven test format (SQL scripts, HTTP request collections) is hooked into the build.

Source: Gradle User Manual: Non-class-based testing, Basic example, Filtering, Gradle 9.4.0 release notes: non-class-based testing

Code coverage with JaCoCo

The jacoco plugin attaches the JaCoCo agent to every Test task of the project, records coverage in a .exec file per task, and adds two tasks for the test task: jacocoTestReport (reports) and jacocoTestCoverageVerification (rules that can fail the build). It replaces the EMMA and Cobertura of the era of the book: EMMA is dead, the Cobertura plugin was always third party and unmaintained, and JaCoCo is the core Gradle plugin.

Kotlin DSL: tidepool-engine/build.gradle.kts (excerpt)
plugins {
    `java-library`
    jacoco
}

tasks.test {
    finalizedBy(tasks.jacocoTestReport)
}

tasks.jacocoTestReport {
    dependsOn(tasks.test)
    reports {
        xml.required = true
        csv.required = true
    }
}

tasks.jacocoTestCoverageVerification {
    violationRules {
        rule {
            limit {
                minimum = "0.80".toBigDecimal()
            }
        }
    }
}

tasks.check {
    dependsOn(tasks.jacocoTestCoverageVerification)
}
Groovy DSL: tidepool-engine/build.gradle (excerpt)
plugins {
    id 'java-library'
    id 'jacoco'
}

tasks.test {
    finalizedBy tasks.jacocoTestReport
}

tasks.jacocoTestReport {
    dependsOn tasks.test
    reports {
        xml.required = true
        csv.required = true
    }
}

tasks.jacocoTestCoverageVerification {
    violationRules {
        rule {
            limit {
                minimum = 0.80
            }
        }
    }
}

tasks.check {
    dependsOn tasks.jacocoTestCoverageVerification
}

The pieces:

  • finalizedBy(tasks.jacocoTestReport) makes the report run after every test, even when tests fail (a failed run is when the report is most useful). dependsOn(tasks.test) on the report does the reverse, so that asking for the report runs the tests first. The plugin does neither for you, and a report with no execution data is skipped.

  • Report formats: HTML (default, in build/reports/jacoco/test/html), XML (xml.required, the input of SonarQube, Codecov and most CI integrations) and CSV. Each has an outputLocation.

  • Coverage counters are INSTRUCTION, BRANCH, LINE, COMPLEXITY, METHOD and CLASS; their values are COVEREDRATIO, MISSEDRATIO, COVEREDCOUNT, MISSEDCOUNT and TOTALCOUNT. A rule applies to an element (BUNDLE, the default, or PACKAGE, CLASS, SOURCEFILE, METHOD) and has limit blocks; includes and excludes narrow it to classes. In the Kotlin DSL a limit’s minimum is a BigDecimal, which is why the value is written "0.80".toBigDecimal().

  • jacoco { toolVersion = "..." } selects the JaCoCo version. The default of Gradle 9.8.1, taken from the resolved jacocoAgent configuration of the build, is 0.8.15 (8.14 used 0.8.13). The JaCoCo version limits the class file versions that can be instrumented, so a build on a new JDK needs a new enough JaCoCo, which is one more reason to keep Gradle current.

  • The agent attached to every Test task. After a full check, the engine’s build/jacoco directory contains test.exec, integrationTest.exec and goldenTest.exec; jacocoTestReport reads only the first one, because it is configured for the test task. To report the combined coverage of several suites, a JacocoReport task of your own has to be given the execution data of all of them (not run here). The extension that the plugin adds to every Test task, jacoco { ... } in Groovy and extensions.configure<JacocoTaskExtension> { ... } in Kotlin, moves the .exec file (destinationFile) and controls the agent’s options.

The CSV report of the engine after the unit tests (INSTRUCTION and BRANCH are the interesting columns: every instruction of StationReader is covered, and 9 instructions of TideEngine are not, namely the branches of the switch for constituents other than the ones the tests use):

GROUP,PACKAGE,CLASS,INSTRUCTION_MISSED,INSTRUCTION_COVERED,BRANCH_MISSED,BRANCH_COVERED,LINE_MISSED,LINE_COVERED,COMPLEXITY_MISSED,COMPLEXITY_COVERED,METHOD_MISSED,METHOD_COVERED
tidepool-engine,io.tidepool.engine,StationReader,0,92,0,4,0,10,0,3,0,1
tidepool-engine,io.tidepool.engine,TideEngine,9,51,5,3,5,8,5,2,0,1

The overall rule passed, because 143 of 152 instructions are covered, more than 80 percent. A stricter rule shows what a violation looks like. A second rule was added in a scratch copy, per class and for branches:

Kotlin DSL (scratch variation)
rule {
    element = "CLASS"
    limit {
        counter = "BRANCH"
        minimum = "0.90".toBigDecimal()
    }
}
Groovy DSL (scratch variation)
rule {
    element = 'CLASS'
    limit {
        counter = 'BRANCH'
        minimum = 0.90
    }
}
> Task :tidepool-engine:jacocoTestCoverageVerification FAILED
[ant:jacocoReport] Rule violated for class io.tidepool.engine.TideEngine: branches covered ratio is 0.37, but expected minimum is 0.90

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':tidepool-engine:jacocoTestCoverageVerification' (registered by plugin 'org.gradle.jacoco').
> A failure occurred while executing org.gradle.internal.jacoco.AntJacocoCheck
   > Rule violated for class io.tidepool.engine.TideEngine: branches covered ratio is 0.37, but expected minimum is 0.90

(the Groovy DSL produced the same message). JaCoCo reports only the first violated rule, so a build with several broken rules shows them one at a time. The verification task attaches to check by the explicit dependsOn in the script: the manual explains that the task is not a dependency of check by default because it is not incremental (it declares no outputs), so every check would rerun it. Thresholds are a team decision: a rule on a new-code coverage, as SonarQube computes it, is more useful than a hard overall percentage that people satisfy with meaningless tests.

Source: Gradle User Manual: The JaCoCo plugin, Configuring the JaCoCo plugin, Report configuration, Coverage verification and violation rules, Task configuration, Upgrading within 8.x: JaCoCo 0.8.13

Aggregating test and coverage reports

In a multi-project build every project writes its own report, which is what you do not want on a CI summary page. Two plugins collect them: test-report-aggregation for test results and jacoco-report-aggregation for coverage. Both consume the variants that the test suites publish (their outgoing variants), so aggregation uses ordinary project dependencies, not file paths. The manual shows two arrangements: apply the plugin in a project that depends on the others anyway (an application or WAR project), or use a standalone project. The example is the second:

Kotlin DSL: tidepool-reports/build.gradle.kts
plugins {
    base
    `test-report-aggregation`
    `jacoco-report-aggregation`
}

dependencies {
    testReportAggregation(project(":tidepool-model"))
    testReportAggregation(project(":tidepool-engine"))
    jacocoAggregation(project(":tidepool-model"))
    jacocoAggregation(project(":tidepool-engine"))
}

reporting {
    reports {
        create<AggregateTestReport>("testAggregateTestReport") {
            testSuiteName = "test"
        }
        create<JacocoCoverageReport>("testCodeCoverageReport") {
            testSuiteName = "test"
        }
    }
}
Groovy DSL: tidepool-reports/build.gradle
plugins {
    id 'base'
    id 'test-report-aggregation'
    id 'jacoco-report-aggregation'
}

dependencies {
    testReportAggregation project(':tidepool-model')
    testReportAggregation project(':tidepool-engine')
    jacocoAggregation project(':tidepool-model')
    jacocoAggregation project(':tidepool-engine')
}

reporting {
    reports {
        testAggregateTestReport(AggregateTestReport) {
            testSuiteName = 'test'
        }
        testCodeCoverageReport(JacocoCoverageReport) {
            testSuiteName = 'test'
        }
    }
}

The two plugins add the configurations testReportAggregation and jacocoAggregation, on which the projects to aggregate are declared. The reporting block creates the reports and, with them, their tasks. Because this project does not apply the java plugin (it has no code), it applies no test suite of its own, and the reports have to be created manually; the plugins create them automatically (testAggregateTestReport and testCodeCoverageReport, named after the suite) in a project that applies java. The testSuiteName selects which suite of the dependencies is collected, so the integrationTest results would be a second report. Running the two report tasks:

./gradlew --configuration-cache :tidepool-reports:testAggregateTestReport :tidepool-reports:testCodeCoverageReport
> Task :tidepool-model:test
> Task :tidepool-engine:test
StationReaderTest > parsesManyStations() PASSED
...
TideEngineTest > pureM2StartsAtItsAmplitude() PASSED
> Task :tidepool-reports:testAggregateTestReport
> Task :tidepool-engine:jacocoTestReport
> Task :tidepool-model:jacocoTestReport
> Task :tidepool-reports:testCodeCoverageReport

BUILD SUCCESSFUL in 1s
13 actionable tasks: 13 executed

(compilation tasks are left out). The reports are in the aggregating project: tidepool-reports/build/reports/tests/test/aggregated-results/index.html lists the test classes of both projects (io.tidepool.engine.StationReaderTest, io.tidepool.engine.TideEngineTest and io.tidepool.model.HeightTest; eleven tests, nine of the engine and two of the model, none failed), and tidepool-reports/build/reports/jacoco/testCodeCoverageReport/ contains an HTML tree with the packages of both projects and an XML file. The aggregation ran the tests of the dependencies when needed, since the results are task outputs that the variants point to. To get a report even when tests fail, run the report tasks with --continue. The test report aggregation of 9.3 shows overlapping test sources (the same class tested in two projects) as separate tabs. The aggregation plugins were described as incubating in the 8.13 upgrade notes, which also changed how suites with several targets are aggregated.

Source: Gradle User Manual: Test Report Aggregation plugin, Manual report creation, JaCoCo Report Aggregation plugin, Manual report creation (JaCoCo), Upgrading within 8.x: changes to the aggregation plugins

Static checks: Checkstyle and PMD

The book’s chapter on code quality covers JaCoCo, Checkstyle, PMD and Sonar. Of the static analysis tools of that era, Gradle still ships checkstyle and pmd as core plugins (FindBugs and JDepend were deprecated in 5.0 and removed in 6.0; SpotBugs is the third-party successor, with its own plugin id). Each plugin adds one task per source set (checkstyleMain, checkstyleTest, pmdMain, …​) and attaches them to check. A scratch project (legacy, extended by one production class) showed both:

Kotlin DSL: build.gradle.kts (excerpt)
plugins {
    `java-library`
    checkstyle
    pmd
}

pmd {
    ruleSets = listOf("category/java/bestpractices.xml", "category/java/errorprone.xml")
    isConsoleOutput = true
}
Groovy DSL: build.gradle (excerpt)
plugins {
    id 'java-library'
    id 'checkstyle'
    id 'pmd'
}

pmd {
    ruleSets = ['category/java/bestpractices.xml', 'category/java/errorprone.xml']
    consoleOutput = true
}

Checkstyle reads config/checkstyle/checkstyle.xml by default; this file was used:

<?xml version="1.0"?>
<!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN" "https://checkstyle.org/dtds/configuration_1_3.dtd">
<module name="Checker">
    <module name="NewlineAtEndOfFile"/>
    <module name="TreeWalker">
        <module name="UnusedImports"/>
        <module name="MissingJavadocType"/>
    </module>
</module>

The class under check has an unused import, no class Javadoc, an unused private field and an empty catch:

package legacy;

import java.util.List;

public class Tides {
    private int never;

    public static double twice(double metres) {
        try {
            return metres * 2;
        } catch (RuntimeException e) {
        }
        return 0;
    }
}
./gradlew checkstyleMain
> Task :checkstyleMain FAILED
[ant:checkstyle] [ERROR] /work/tidepool/src/main/java/legacy/Tides.java:3:8: Unused import - java.util.List. [UnusedImports]
[ant:checkstyle] [ERROR] /work/tidepool/src/main/java/legacy/Tides.java:5:1: Missing a Javadoc comment. [MissingJavadocType]

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':checkstyleMain' (registered by plugin 'org.gradle.checkstyle').
> A failure occurred while executing org.gradle.api.plugins.quality.internal.CheckstyleInvoker
   > Checkstyle rule violations were found. See the report at: file:///work/tidepool/build/reports/checkstyle/main.html
     Checkstyle files with violations: 1
     Checkstyle violations by severity: [error:2]
./gradlew pmdMain
> Task :classes UP-TO-DATE
/work/tidepool/src/main/java/legacy/Tides.java:6:	UnusedPrivateField:	Avoid unused private fields such as 'never'.
/work/tidepool/src/main/java/legacy/Tides.java:11:	EmptyCatchBlock:	Avoid empty catch blocks
/work/tidepool/src/main/java/legacy/Tides.java:11:	AvoidCatchingGenericException:	Avoid catching RuntimeException in try-catch block
> Task :pmdMain FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':pmdMain' (registered by plugin 'org.gradle.pmd').
> A failure occurred while executing org.gradle.api.plugins.quality.internal.PmdInvoker
   > 3 PMD rule violations were found. See the report at: file:///work/tidepool/build/reports/pmd/main.html

(only the lines that matter are shown; in both runs the violations are printed to the console, Checkstyle by its Ant task and PMD because of consoleOutput). Both ran in both DSLs with the same result. Notes: Checkstyle’s version in Gradle 9.8.1 resolved to 10.24.0 and PMD’s to 7.24.0 (see the checkstyle and pmd configurations of the dependencies report; toolVersion changes them); the PMD ruleset names used are the PMD 7 categories (category/java/bestpractices.xml); the Checkstyle and PMD reports are HTML and XML, and since Gradle 9.4 PMD also writes CSV, Code Climate and SARIF reports, configured on the task (tasks.pmdMain { reports { sarif.required = true } }) and not on the pmd extension. Violations fail the build by default; ignoreFailures (both plugins) and maxWarnings and maxErrors (Checkstyle) relax that (not run here). SonarQube is a service and is out of scope here; its Gradle integration is the org.sonarqube plugin, which reads the JaCoCo XML reports above, and the legacy sonar plugin of the book was removed in Gradle 3.0.

Source: Gradle User Manual: The Checkstyle plugin, The PMD plugin, Gradle 9.4.0 release notes: PMD reports, Gradle 3.0 release notes: Sonar plugin removed, SonarScanner for Gradle

Classic mistakes

  • The integration suite is not part of check. Fix: tasks.check { dependsOn(testing.suites.named("integrationTest")) }. The dry run above proves it.

  • JUnit 4 tests vanish after a move to Jupiter. The junit-vintage-engine is missing; only when no test is discovered does Gradle 9 fail the task.

  • Failed to load JUnit Platform on a hand-wired Test task: the launcher is not on the runtime classpath. Use test suites, which add it, or add junit-platform-launcher.

  • A Test task registered without configuration runs nothing since 9.0 and says only NO-SOURCE.

  • A red test stays hidden with ignoreFailures = true plus the up-to-date check, until --rerun.

  • Tests that depend on the working directory. The worker’s working directory is the project directory, not the root; the example passes the data location as a system property computed from the project layout instead of hard-coding ../.

  • Sharing state between parallel forks: files in a fixed location, fixed ports, a shared database.

  • Shipping test code in main instead of using test fixtures.

  • Coverage thresholds as the goal: a violation rule with an overall percentage rewards trivial tests; per-class rules on branches, as above, or new-code rules from a quality server catch more.

  • Reading project properties inside a test task action (project.property) breaks the configuration cache; use providers (providers.gradleProperty) at configuration time, as in the -Pfast example.

  • Timing claims from a first run. The 21 s of the first goldenTest run include building the projects and the daemon start; compare runs only after a warm-up.

Configuration cache

Every build script of this page was run with --configuration-cache and stored the entry without problems, and a second invocation reused it. What makes test builds compatible: only task properties and providers are read at configuration time (the system property of Test uses layout.projectDirectory, not a File captured from project); the JVM Test Suite plugin, JaCoCo, Checkstyle, PMD and the aggregation plugins are compatible, which is also true of the Test task. With the cache on, tasks of one project can run in parallel, which the page’s clean check output shows through the interleaved order of independent tasks. Incompatible things to avoid in test configuration: project in doFirst/doLast of a test task, a TestListener that holds on to Project, and build scripts that read environment variables at configuration time without declaring them. The configuration cache is on page Configuration Cache & Isolated Projects; a test task is cacheable by the build cache (see Build Cache) only if the tests are deterministic and do not depend on undeclared inputs: pass the inputs through inputs.file(...), inputs.property(...) or systemProperty with a provider, not by reading them in the test.

Gradle 8.14 and Gradle 9

The column for 8.14 comes from the official documentation and release notes (not run here); the column for 9.8.1 is what was run above, or the 9.x notes.

Topic Gradle 8.14 Gradle 9.8.1

JUnit Platform launcher

Loaded from the Gradle distribution when not declared, a behavior deprecated in 8.x ("automatic test framework implementation dependencies"); test suites declare it automatically.

Must be on the test runtime classpath of a hand-wired Test task; the failure is the Failed to load JUnit Platform error above. Test suites are unaffected.

Test task conventions

A Test task with no configuration inherits the classes and classpath of the test source set, with a deprecation warning since 8.1.

The convention is removed; such a task ends NO-SOURCE without a message.

No test discovered

A deprecation warning in 8.x for the successful run without executed tests.

The task fails (failOnNoDiscoveredTests, default true).

Default testing framework versions of test suites

Older than the 9.0 defaults.

JUnit Jupiter 5.12.2, TestNG 7.11.0, Spock 2.3 (9.0); pass a version to use a newer one, as the page does with JUnit 6.1.3.

JvmTestSuite.testType

Removed in 8.13: the attribute is no longer used; aggregation selects suites by name.

Not available.

Tests on JVM 6 or 7

Deprecated in 8.x.

An error since 9.0.0. The daemon needs Java 17 or later; test workers can run on an older JDK through javaLauncher.

JaCoCo

0.8.13 (8.14).

0.8.15, the default resolved here.

HTML test report

The classic report: tests grouped by package and class, output merged per class, aggregate report as one list.

Detailed report with nested classes, parameterized-test suites and per-test output (9.3), sortable columns (9.6) and test metadata tabs (9.4).

Non-class-based tests, testDefinitionDirs

Not available; workarounds with empty classes.

Gradle 9.4.0, incubating.

Test and JaCoCo report aggregation

Incubating, with the changes of 8.13.

Run above; overlapping sources as tabs (9.3).

--test-dry-run

The dryRun property is in the 8.14 DSL reference as an incubating feature (not run here).

Incubating option, run above.

failOnNoDiscoveredTests, testDefinitionDirs

Neither property is in the 8.14 DSL reference.

Both are in the 9.8.1 reference: the first since 9.0, the second since 9.4.

Framework start-up failures

Could be hidden by the logging granularity.

Always printed (9.7.0).

PMD report formats

XML and HTML.

CSV, Code Climate and SARIF as well (9.4.0).

Source: Gradle User Manual: Upgrading to Gradle 9, Upgrading within 8.x, Gradle 9.3.0 release notes, Gradle 9.4.0 release notes, Gradle 9.7.0 release notes

The same in Ant and Maven

Concern Ant Maven Gradle

Run unit tests

<junitlauncher> (JUnit Platform) or <junit> (JUnit 3 and 4), called from a target you write

Surefire, bound to the test phase

The Test task of each suite, wired by the JVM plugins

The test classpath and JVM

A <classpath> and a forked <junitlauncher> or <junit> with fork and jvm

The test scope, the Surefire argLine, jdkToolchain and forkCount

testImplementation, testRuntimeOnly; jvmArgs, javaLauncher, maxParallelForks

Select tests

<test> and <batchtest> with includes; a property for one test

-Dtest=..., includes and excludes, tags and groups

--tests, filter, include/exclude, includeTags

Integration tests

A second compile and run target

Failsafe, bound to integration-test and verify

A test suite attached to check

Reports

<junitreport> turns the XML into HTML

target/surefire-reports and the Surefire report in the site

HTML and XML in build/reports/tests and build/test-results; aggregation plugins

Coverage

JaCoCo ant tasks (jacoco:coverage, jacoco:report)

jacoco-maven-plugin (prepare-agent, report, check)

The jacoco plugin: jacocoTestReport, jacocoTestCoverageVerification

Test helpers shared between modules

A shared JAR or directory by hand

A test-jar of the maven-jar-plugin

java-test-fixtures

Static checks

Checkstyle and PMD ant tasks

maven-checkstyle-plugin, maven-pmd-plugin

The checkstyle and pmd plugins

The deeper comparison lives on the sibling pages: Testing with JUnit & junitlauncher, Maven Testing with Surefire, Maven Integration Testing with Failsafe and Maven Site & Reporting for the quality reports. The worked Maven setup of the Surefire, Failsafe, JaCoCo, Checkstyle and PMD plugins in a Spring Boot project is in Spring Boot Maven Quality Plugins, and the overall comparison is in Comparing Ant, Maven and Gradle. What carries over when a Maven build moves to Gradle is on Migrating from Maven.

Where the books are outdated

Gradle in Action (Muschko, Manning 2014, Gradle 1.7) devotes its testing chapter to the test pyramid, to JUnit, TestNG and Spock, to forking and parallel execution, to logging, filtering and listeners, to reports, and to integration and functional tests with hand-made source sets and tasks. The concepts survive and the mechanics changed:

  • Integration tests with a hand-made source set, configurations and Test task are replaced by the JVM Test Suite plugin (7.3); the hand-made form still works and is shown on Gradle Java Projects.

  • compile, testCompile, runtime and testRuntime are gone since 7.0; use implementation, testImplementation, runtimeOnly and testRuntimeOnly.

  • The book’s JUnit is JUnit 4 and the old runner. JUnit 5 support arrived in 4.6, JUnit 6 is the current line here, and the Vintage engine bridges old tests.

  • The book’s quality chapter covers JaCoCo, Checkstyle, PMD and Sonar. JaCoCo, Checkstyle and PMD remain core plugins; FindBugs and JDepend, core plugins of the era, were removed in 6.0; the legacy sonar plugin was removed in 3.0 and replaced by SonarQube’s own org.sonarqube plugin; the Cobertura plugin was third party and unmaintained.

  • The book’s ProjectBuilder-only tests of plugins are replaced by TestKit’s GradleRunner (see Testing & Publishing Plugins).

  • The book’s test listeners are one way to observe results; for results consumed by CI, the XML files and the aggregation plugins are the supported path, and in a build with the configuration cache on, listeners that capture the project are a problem (see above).

  • The book predates the configuration cache, version catalogs, toolchains and javaLauncher; the page numbers of the book were not checked, and the statements follow the research notes behind this section.

Where to go next

References