Repositories & Content Filtering

Gradle has built-in shorthands for the repositories that nearly every build uses. They are methods of RepositoryHandler, and each resolves from a fixed URL.

Every repository has a name. The default name of mavenCentral() is MavenRepo (as in the messages below) and the Plugin Portal’s is Gradle Central Plugin Repository; a repository declared with a block (maven { name = "internal" }) can set its own name, which appears in error messages and is what credentials properties and Maven mirror settings are matched by.

Source: Gradle User Manual: Maven Central repository, Google Maven repository and Declaring a custom repository by URL

Repositories are searched in declaration order. When Gradle finds a module’s metadata in a repository, it downloads all of that module’s artifacts from the same repository; it does not mix the POM from one repository with the JAR from another. Searching continues down the list only while a repository answers "not found". That has three consequences that the rest of the page builds on:

A related rule is not about order but about scope. A Maven POM may name additional repositories, and Maven honours them; Gradle ignores them and uses only the repositories the build declares. The documentation describes this as both a reproducibility safeguard and a security measure: otherwise an updated dependency could pull artifacts from unverified sources into your build.

Source: Gradle User Manual: Declaring multiple repositories and Strict limitation for Maven repositories

repositoriesMode says what happens when a project also declares its own repositories. There are three modes.

The three modes with the same project, to which a stray repositories { mavenCentral() } was added at the end of the build script (the settings file is the one above, in the stated mode). The script also applies the io.github.ben-manes.versions plugin from the Gradle Plugin Portal; it plays no part in the modes and is used later, in "Plugin repositories":

With FAIL_ON_PROJECT_REPOS (./gradlew help --no-configuration-cache) the build stops during configuration:

In Groovy the same failure is wrapped in an evaluation error:

With PREFER_SETTINGS the project repository is ignored and the build only warns (./gradlew compileJava, which succeeded):

With the default PREFER_PROJECT the project’s mavenCentral() silently replaces the settings repositories, so the internal module can no longer be found. The error message names the cause, which is worth remembering because the stack of Could not find lines alone points at the module, not at the repository list:

(The failing runs used --no-configuration-cache: with the configuration cache on, a resolution failure that happens while the cache entry is being stored is reported wrapped in Configuration cache state could not be cached.)

Source: Gradle User Manual: Repositories mode, Available modes and Best practice: set up your dependency repositories in the settings file

maven { url = ... } declares a repository in the Maven layout (group/as/path/artifact/version/artifact-version.ext), which is what Maven Central, Nexus, Artifactory and Reposilite serve and what maven-publish writes. Gradle reads the POM and, when it is published next to it, the Gradle Module Metadata file (.module); with both present, the .module file wins, because it carries variants and attributes that a POM cannot express (see Variant-Aware Resolution & Module Metadata). In the lab, the internal module is resolved from its .module file, which is why dependencyInsight lists a variant called apiElements for it and compile for the POM-only commons-csv.

Source: Gradle User Manual: Maven repositories and Supported transport protocols

A URL that starts with http: is refused unless the repository opts in. Without the opt-in, Gradle 9.8.1 fails the build with this message (./gradlew compileJava --no-configuration-cache, a repository at http://localhost:18081):

The opt-in is a repository property, named isAllowInsecureProtocol in the Kotlin DSL (a Boolean property whose getter starts with is) and allowInsecureProtocol in Groovy. It is intended for servers on localhost or inside a closed network, never for the public Internet; Central itself is HTTPS only.

Source: Gradle 6.0 release notes and UrlArtifactRepository

An ivy {} repository serves modules described by ivy.xml files. Its default layout (the "gradle" layout) is [organisation]/[module]/[revision]/[artifact]-[revision].[ext], which is what ivy-publish writes, so the producer’s output can be consumed directly:

With the build script of the lab (./gradlew compileJava dependencies --configuration compileClasspath --configuration-cache; the configuration cache entry was stored, the example is compatible):

The named layouts are gradle (the default), maven and ivy (layout("maven")). A repository with any other directory structure uses a patternLayout {}, where the tokens in square brackets are Ivy’s pattern variables; ivy(...) adds a pattern for descriptor files and setM2compatible(true) (Groovy: m2compatible = true) turns dots in the organisation into slashes. If a repository holds only JARs and no descriptors, metadataSources { artifact() } tells Gradle to use the existence of the artifact as the only metadata. The following repository is a directory with the layout tidepool-model/1.0.0/jar/tidepool-model.jar:

This resolves io.tidepool:tidepool-model:1.0.0 and compiles (the dependency tree is identical to the one above). Without the metadataSources block, Gradle looks for an ivy.xml that is not there. Note that the message also lists Maven Central: the content filter above restricts the Ivy repository, not Central (that is the next section’s subject).

Ivy repositories also have a "dynamic resolve mode" that makes Gradle prefer the revConstraint attribute of an ivy.xml over rev; it is available only for Gradle’s own Ivy repositories (not run here).

Source: Gradle User Manual: Ivy repositories, Custom pattern layout Ivy repository and Enabling Ivy dynamic resolve mode

flatDir { dirs("libs") } makes Gradle look for JAR files in a plain directory. There is no metadata: Gradle synthesizes metadata from the file name, so the dependency has no transitive dependencies and no group. The coordinates are written with an empty group (":commons-csv:1.14.1"). The documentation discourages flat directories, because without POMs or Ivy files nothing says what the JAR needs; if the JAR must live in the source tree, a Maven or Ivy repository on a file: URL is preferred. Gradle also prefers modules with real metadata over ones synthesized by a flat directory, so a flat directory cannot override an artifact that another declared repository describes properly.

commons-csv is on the compile class path, but its own dependencies commons-io and commons-codec, which the previous tree showed, are not: they would be missing at run time. That is the practical cost of a flat directory.

Source: Gradle User Manual: Flat directory repository

credentials(PasswordCredentials::class) (Groovy: credentials(PasswordCredentials)) asks for two Gradle properties derived from the repository’s name: <name>Username and <name>Password. They can come from gradle.properties (project or Gradle user home), from -P on the command line, or from environment variables named ORG_GRADLE_PROJECT_<name>Username and ORG_GRADLE_PROJECT_<name>Password. Gradle fails the build when they are missing, and says which ones:

With -PinternalUsername=build-user -PinternalPassword=placeholder-secret, with the two environment variables, or with the two lines in $GRADLE_USER_HOME/gradle.properties, the build succeeds; all three ways were run. The server log of the -P run shows how a username and password are used by default: Gradle first sends the request without credentials, waits for the 401 challenge, and then repeats it with them.

By default Gradle tries every scheme the Apache HttpClient library supports. Declaring an authentication {} block restricts the repository to the schemes it names. The supported schemes are BasicAuthentication (credentials sent preemptively), DigestAuthentication and HttpHeaderAuthentication (a custom header such as a private or OAuth token, used with HttpHeaderCredentials). Some servers answer an unauthenticated request with something other than 401 (GitHub returns 404), so the challenge never comes; for them, declare BasicAuthentication explicitly to send the credentials preemptively:

The server log of this run begins with an authenticated request; there is no 401 round trip:

Token authentication with HttpHeaderCredentials and HttpHeaderAuthentication, the digest scheme, sftp:, s3: (including IAM authentication) and gcs: repositories and their system properties (org.gradle.s3.endpoint, org.gradle.gcs.endpoint) are documented but were not run here, because they need servers and accounts that do not belong in an example. The examples of this page use values read from properties; the documentation’s snippets with literal keys are for illustration only.

With the configuration cache enabled, credentials passed to a repository can be read at configuration time and become part of the cached build configuration. The cache entry is encrypted on disk; the documentation points to the configuration cache page on credentials and secrets for the implications. The runs above, which passed the credentials through -P or environment variables, stored and reused cache entries without a problem.

Source: Gradle User Manual: Basic authentication, Using preemptive authentication and Using HTTP header authentication

content {} inside a repository restricts what that repository is asked for. By default a repository includes everything and excludes nothing; once you declare an include, it excludes everything else; once you declare an exclude, it includes everything else; with both, it contains what is included and not excluded. A filter works on a group, a module or a version, strictly or with a regular expression.

The lab compares four variants with the access logs of the two servers. "Internal" is the repository on port 18081, "public" the proxy to Central on port 18082, and the build is the lab build. After each run, two commands count the requests that should never have happened: requests to the internal server for a module outside io.tidepool, and requests to the public server for io.tidepool:

The results, with the repositories in the order and with the filters stated (all runs were ./gradlew compileJava --refresh-dependencies, configuration cache on, and all succeeded; the Kotlin and Groovy runs gave identical numbers):

The reading of the table is the heart of this section. Without filters, the order decides who is bothered: with the internal repository first, it receives six HEAD requests that end in 404, for the POMs of commons-csv, of its dependencies and of its parent POMs; with the public repository first, Central receives a request for io/tidepool/tidepool-model/1.0.0/tidepool-model-1.0.0.pom, which is the name of a company-internal module, and answers 404. The third row is the subtle one: a filter on the internal repository does not stop an earlier repository from being asked. A repository-level filter declares what this repository contains; it says nothing about whether other repositories may also have it.

exclusiveContent states the stronger fact: these artifacts are found in this repository and in no other. Gradle then asks no other repository for them, whatever the declaration order, and the rule is declared in one place. The documentation calls exclusive content filters a best practice. The settings file shown at the start of this page is the form used throughout the section, and the last row of the table shows the exclusiveContent repository declared after the public one without that being a problem:

Three details from the documentation. First, an exclusive filter must be written in extension: it says what the repository contains, and that is all it says, so the transitive dependencies of the exclusive modules must also be found in some declared repository. Second, exclusiveContent takes an inclusive descriptor, so only the include... methods exist inside its filter {}. Third, using exclusive content in the pluginManagement section of the settings file makes it illegal to add more repositories through a project’s buildscript.repositories; the build then fails, and the options are to declare all repositories in settings or to use non-exclusive filters.

A last caution about the pattern at the end of the best-practices page: the last repository is often declared without any filter so that it serves as the default for everything the others did not claim. That is convenient, and the documentation warns that it can be a security risk, since anything not claimed by an earlier filter falls through to a repository that may serve it from anywhere.

Source: Gradle User Manual: Declaring a repository filter, Declaring content exclusively found in one repository, Best practice: use content filtering with multiple repositories and RepositoryContentDescriptor

For Maven repositories, mavenContent {} declares whether a repository holds releases or snapshots, which is how company repositories are usually split. A repository declared snapshotsOnly() is not asked for a release at all; the following internal repository (which holds the release 1.0.0) is skipped, and the error lists only Central:

releasesOnly() is the mirror image. Together with the -SNAPSHOT version suffix and the change-detection rules for dynamic and changing versions (on Platforms, Constraints & Rich Versions), this is what keeps snapshot repositories from being consulted for release builds.

Source: Gradle User Manual: Maven repository filtering