Paths, Filesets & Patterns

Ant has no single "file set" class. It has a handful of types that all answer the question "which files?" in a different way, and a handful of tasks that consume the answer. A path-like structure is an ordered list of file system locations (a PATH, a CLASSPATH); a fileset is a group of files found by scanning a directory tree and matching patterns; a dirset is the same for directories; a filelist names files explicitly, whether they exist or not.

Two ideas run through the whole page. First, a set of files is described once, with an id, and used many times with a refid; the description is evaluated again each time it is used, so the result always reflects the files that exist at that moment. Second, patterns are relative to a base directory and are matched one directory level at a time, which explains almost every surprise a newcomer meets.

Source: Apache Ant manual: Path-like Structures

Source: Apache Ant manual: Directory-based Tasks

A path-like structure is Ant’s value for "a list of locations separated by the platform’s path separator": the CLASSPATH of javac and java, the module path, the PATH an exec task needs, the list of source directories. The manual’s phrase is that wherever a path-like value is needed, a nested element can be used. Three elements take the same content:

<pathconvert> turns a path, or any resource collection, into a string, and either stores it in a property, writes it to a resource or logs it. Its two jobs are producing a platform-specific path for another program (a script, a manifest, a shell command) and producing a list with a separator of your choice.

The nested elements are <map from="..." to="..."/> (replace a leading prefix; the first matching map wins, so list the longest prefix first; the match is case-insensitive on Windows), a nested <mapper> (since Ant 1.6.2) for any other file-name transformation, and the resource collections themselves when refid is not used. The Windows result in the output above came from targetos="windows" and one map; the second pathconvert used a flatten mapper to keep the file names only and a comma as separator.

An empty result is where the defaults bite. With setonempty="true" (the default) the property exists and is empty, so a later unless="..." test sees it as set; with setonempty="false" the property is not defined:

The unexpanded ${p} in the first line is how Ant shows a property that does not exist. Calling pathconvert with nothing to convert is an error:

Source: Apache Ant manual: PathConvert

A runnable JAR that needs libraries has them listed in its manifest’s Class-Path attribute, as file names relative to the JAR. The <manifestclasspath> task (since Ant 1.7) computes that string from an Ant path, so that the manifest never contains an absolute name of the machine that built it. It is also the standard workaround for the Windows limit on the length of a command line, because a launcher JAR with a long Class-Path replaces a class path of hundreds of entries.

The build.xml below compiles all three modules with one javac (whose <src> is a path built from a dirset), copies the libraries next to the JAR, computes the Class-Path, builds tidepool-cli.jar and runs it:

(the decimal comma is the locale of the machine that ran the example) and the manifest the jar task wrote, read with unzip -p build/dist/tidepool-cli.jar META-INF/MANIFEST.MF, shows the property re-wrapped at 72 characters as the JAR specification requires:

Two failure modes are worth knowing. The first is a library that is on the compile path but was not copied next to the JAR: the Class-Path is built from the files that exist, so the missing JAR is simply not listed, and the program fails at run time. The first run of this example had commons-io missing from lib (it is a dependency of commons-csv), and Class-Path listed only two libraries:

The JVM ignores a Class-Path entry whose file is missing without any message, so this failure is the only sign. The second failure mode is an entry that is more than maxParentLevels directories away from the JAR. With a limit of 1, the library directory two levels up is rejected:

The relative layout that manifestclasspath assumes must be the layout of the deployed files, which is why the example copies the libraries into build/dist/lib first and computes the value from that directory. Maven does the same job with the addClasspath manifest setting of the maven-jar-plugin and Gradle with a Class-Path attribute computed from runtimeClasspath; both are on the Maven packaging page and the Gradle Java projects page. Packing the libraries inside one JAR is the subject of Packaging Archives.

Source: Apache Ant manual: ManifestClassPath

Every class path Ant hands to a compiler or a forked JVM can be combined with the system class path, the one in effect when Ant itself runs. The build.sysclasspath property chooses how:

The default differs from task to task, and the property also affects the bootclasspath settings since Ant 1.7 (when the property is unset it defaults to ignore there). In day-to-day builds the property is rarely set. What matters more is its consequence for javac: the includeantruntime attribute says whether the Ant libraries themselves are put on the compile class path, and it defaults to "yes, unless build.sysclasspath is set". That default is why a javac without the attribute prints a warning on every build, and why a build that works on one machine can fail on another where Ant is installed differently:

Setting includeantruntime="false" removes the warning and makes the compile class path exactly what the build file says. A project whose sources really use Ant classes (a custom task, see Writing Custom Tasks) must either set it to true or name the Ant jars. The probe below is a class that imports org.apache.tools.ant.Main; it fails to compile with includeantruntime="false", compiles with true, and compiles with an explicit class path only when both ant.jar and ant-launcher.jar are named:

With only ${ant.core.lib} (the property that points at ant.jar) on the class path, javac stopped with error: cannot access AntMain and class file for org.apache.tools.ant.launch.AntMain not found, because Main is declared in terms of a class that lives in ant-launcher.jar; adding ${ant.library.dir}/ant-launcher.jar fixed it. ant.core.lib, ant.library.dir and ant.home are among the built-in properties listed on Properties. The full treatment of javac is on Compiling Java.

Source: Apache Ant manual: build.sysclasspath

Source: Apache Ant manual: Javac

Many tasks work on a directory tree: javac compiles the .java files under srcdir, delete removes files below a directory, copy copies a subset. For those tasks, the task element is a fileset. The manual’s list of tasks that can act as an implicit fileset includes checksum, delete, dependset, fixcrlf, javac, replace, rmic, xslt, tar and zip, and the deprecated copydir, jlink and jspc. On such a task the attributes includes, excludes, includesfile and excludesfile and the nested elements include, exclude, includesfile and excludesfile describe the subset, exactly as on a fileset.

javac compiled three of the four model sources, because TideEvent.java was excluded; the explicit <fileset> in copy and delete is the other way to write the same thing, and the only way when a task needs more than one set (the explicit form is the only one copy accepts, because it is not in the manual’s list of implicit-fileset tasks). When a task is an implicit fileset, the selectors of Selectors can be nested directly inside it.

Source: Apache Ant manual: Directory-based Tasks

A pattern selects files by their path relative to the base directory of the set, and only files below that directory are ever scanned. A pattern such as ../foo.java is syntactically legal and never matches anything, because the parent of the base directory is not scanned.

The matching rules are few:

The manual’s own table of examples is reproduced in spirit by the next listing, run for real. The list macro prints the files of the set it is given, with their paths relative to the project directory (the macrodef is explained on Macros & Presets). The file is patterns.xml:

What the listing shows, pattern by pattern:

A * inside a name never matches a /, so *.java selects only the files directly in the base directory, and a frequent beginner error is to expect it to find the sources of a package tree. The mirror error is a fileset whose base directory is already the package directory but whose pattern still starts with src/.

Source: Apache Ant manual: Patterns

Directory-based tasks skip a fixed list of files that are rarely meant to be part of a build: the temporary files of editors and the metadata of version control systems. The list in Ant 1.10.18 is:

Both runs of data in the previous listing show the effect: .DS_Store, .gitignore and stations.csv~ are in the directory, but only stations.csv is part of the default set. A single set can switch the list off with defaultexcludes="false". The list for a whole build can be changed with the <defaultexcludes> task, which alters the default excludes for all subsequent processing:

The backup file stations.csv~ came back into the set, and x.bak stayed out. The task is global and sticky for the rest of the build (and of the JVM, if Ant is embedded), so a build file that changes the list restores it with default="true". With echo="true" the task prints the current list (in the order of a hash set, not the order of the table above); in 1.10.18 its first line carries a literal %n where a newline was meant:

A frequent reason to touch the list is the Subversion _svn convention: remove **/.svn and **/.svn/** and add **/_svn and **/_svn/**, as the manual shows. In practice, the commonest default-excludes surprise has the opposite cause: a file that should be packaged, such as a .gitignore shipped inside a project template, is left out because it is in the list. The fix is defaultexcludes="false" on that one fileset, not a global change.

Gradle uses the same Ant DirectoryScanner underneath its file trees and therefore the same default excludes; it lets a settings script change them with DirectoryScanner.removeDefaultExclude(...) and does not allow changing them during the execution phase.

Source: Apache Ant manual: Default Excludes

Source: Apache Ant manual: DefaultExcludes

Source: Gradle User Manual: Changing the default excludes

A <fileset> is a group of files found in a directory tree below a base directory and matched by patterns and selectors. It holds an implicit patternset, so it takes the patternset’s attributes and nested elements directly. It also acts as an <and> selector container, so any nested selector must select a file for it to belong to the set.

erroronmissingdir defaults to true "for backward compatibility", so the first thing a clean checkout does with a javac that points at a not-yet-created generated-sources directory is fail. Setting it to false on the sets that describe optional directories, for example a src/generated directory that only exists after a code-generation step, keeps the build file valid before and after:

Symbolic links deserve a sentence of their own. followsymlinks="false" excludes symbolic links and junctions completely, so a link that points at a directory of sources makes the whole subtree vanish from the set; true lets selectors with their own followsymlinks attribute decide whether to look at the link or its target. Before Ant 1.10.2 "symbolic link" meant any file whose canonical path differs from its path, which gave false results on some platforms; Ant 1.10.2 and later use Files#isSymbolicLink. With a link data/model-link to ../tidepool-model the difference is the whole module:

Many errors come from the way the element is written, and Ant’s messages are precise about them:

Source: Apache Ant manual: FileSet

A <patternset> groups patterns so that they can be named and shared. It may appear nested in a fileset or in a directory-based task, and also stand-alone at the same level as <target> (a child of <project> or of <target>).

The comma-or-space convenience of the attributes does not apply to the nested element: a nested include takes one pattern. The if and unless attributes of include and exclude test whether a named property is set (not its value), which is the cheap way to make a source list depend on the command line. In the file sets.xml:

A patternset also reads patterns from a file. The file extra.patterns below has a comment line, and the example shows that the file has no comment syntax: every line is a pattern, so the comment became a pattern that matches nothing (harmless here, but a source of confusion). The pattern *.csv matched nothing under . either, because it is relative to the base directory and * does not cross directories:

The <invert> element is only accepted inside a patternset, not directly in a fileset (a direct nesting fails with fileset doesn't support the nested "invert" element):

Gradle’s counterpart is PatternFilterable (the include, exclude, setIncludes family used by fileTree, copy specs and the source sets), with the same Ant-style patterns; Maven’s counterpart is the <includes> and <excludes> list of a resource directory or a plugin configuration.

Source: Apache Ant manual: PatternSet

A <dirset> is the directory counterpart of a fileset: a group of directories matched by patterns and selectors below a base directory. It takes the same attributes and nested elements as a fileset except file (its dir is required), and it is an <and> selector container as well. A dirset contains the directories only; the files inside them are not part of the set, which is why it is most often seen nested in a path (a source path, a list of class directories):

The build file in the manifestclasspath section used a dirset for the opposite purpose: its <path id="main.sources"> wraps the dirset of */src/main/java, which javac took as a <src refid>. A bare <dirset id> cannot be passed as a refid where a path is expected; the type error is

and the fix is to wrap it in a <path>, as the example does. The manual’s last dirset example, a present selector with a glob mapper, selects every directory that contains a marker file, which is also the way to find every module directory in a multi-module tree; selectors are on Selectors.

Source: Apache Ant manual: DirSet

A <filelist> is an explicit list of files relative to a directory. Where a fileset filters what exists, a filelist lists names whether or not the files exist, and does not expand wildcards (a * in a name is kept as written). That makes it the right type for ordered lists (the files of a documentation set, the chapters of a book, the modules a script must run in order), and for the expected outputs of a build that has not run yet.

docs/user guide.pdf does not exist, and it is in the list anyway, with a space in its name that the files attribute could not have carried. This is useful with pathconvert, as in the manual’s example that converts a filelist into a Unix path with a map, and with the available and uptodate tasks that check whether the listed files exist.

Source: Apache Ant manual: FileList

<zipfileset> and <tarfileset> are filesets with extra attributes for building archives. Each has two modes: with dir it is populated with files from the file system; with src (or, since Ant 1.7, a nested resource) it is populated with the entries of an archive, which is how one archive is merged into another. They take every attribute of <fileset> plus:

Only the tar and zip tasks use the permission attributes, and only tar uses the ownership attributes. A zipfileset can be defined once with an id and used in several archives with refid (since Ant 1.6; for tarfileset since Ant 1.7). The file archives.xml:

The reading guide: prefix moved data and lib under app/; fullpath renamed README.txt to app/docs/READ-ME.txt; the src form copied only the model classes out of tidepool-cli.jar (the build of the earlier section); filemode="640" shows as -rw-r----- and the tar entry for stations.csv carries mode 600 and the owner names. The default-excluded stations.csv~ is not in either archive. Two messages catch the usual mistakes:

The jar, war and ear tasks accept the same nested sets, plus zipgroupfileset to merge the contents of many archives, and are the subject of Packaging Archives. Merging is also how Ant builds an uber JAR; Maven uses the Shade or Assembly plugins and Gradle a Jar task with from(zipTree(...)) (see Uber JARs with Shade & Assembly).

Source: Apache Ant manual: ZipFileSet

Source: Apache Ant manual: TarFileSet

A <multirootfileset> (since Ant 1.9.4) is a union of filesets and dirsets that share the same patterns and selectors: the same include and exclude rules applied to several base directories. It takes everything a fileset takes except dir, and adds:

The result lists the model sources first and the engine sources next, the order of basedirs, and the Tide* exclude dropped TideEvent.java and TideEngine.java. The two-module union would otherwise need two filesets and the same patterns twice (or a shared patternset). Gradle’s equivalent is a files(...) collection of several fileTree objects, and Maven spreads the same filter over <sourceDirectory> and the build-helper plugin’s add-source goal.

Source: Apache Ant manual: MultiRootFileSet

A <classfileset> is a fileset computed from the dependencies of some root classes: given a directory of class files and one or more roots, it includes every class file the roots need. It is typically used to build a JAR containing only what an application requires. It needs the BCEL library, which Ant does not ship; the optional ant-apache-bcel.jar in ANT_HOME/lib is the glue, and the BCEL jar (and its own dependencies) must be on Ant’s class path.

The first run failed because BCEL is not installed in Ant 1.10.18 (the stack trace is long; this is its first line):

BCEL 6.12.0 (the version in Ant 1.10.18’s libraries.properties) was fetched from Maven Central into a directory extlib, which is passed to Ant with -lib. That run failed again, with NoClassDefFoundError: org/apache/commons/io/IOUtils, because BCEL 6.12.0 itself needs commons-lang3 (3.20.0) and commons-io (2.21.0). With the three jars in extlib:

The root StationReader needs Station and, through it, Constituent; the other classes of the model (Height, TideEvent) and of the engine were left out of minimal.jar. Static analysis of the byte code cannot see classes loaded by reflection or by name, so a classfileset is an optimisation for small applications, not a replacement for a dependency declaration. Gradle and Maven reach the same goal with the Shade plugin’s minimizeJar and ProGuard-style tools, not with a core feature.

Source: Apache Ant manual: ClassFileSet

Source: Apache Ant manual: Library dependencies

Any element of a project can be given an id, and an element of the same type can then be a reference to it with refid. The mechanism is not specific to sets and paths, but they are its main use, and a build file that defines compile.path once and refers to it from javac, java and junitlauncher is more than tidy: it cannot drift.

Rules that matter in practice:

The <arg> element of the tasks that start programs has a path form for the same purpose: <arg path="..."/> is one argument written with either separator and converted to the platform’s, and <arg pathref="..."/> takes a path reference. <arg line="..."/> splits on spaces and the manual recommends avoiding it. ${toString:ref} is the string form of any reference, used in the first example.

Source: Apache Ant manual: References

Source: Apache Ant manual: Command-line Arguments

Neither Maven nor Gradle has a type called "path": a class path is derived from the declared dependencies. The sets of files do have counterparts:

The Maven row for path is the one that most often surprises Ant users: in Maven the class path is not something one builds, it is a consequence of the POM, and a "wrong" class path is fixed by the dependency declaration and not by a pathelement. The Gradle page on Ant Integration shows how an Ant build’s paths and filesets can be reused from a Gradle build, and the page Migrating Between Tools takes the tidepool Ant build to the other two tools.

Source: Gradle User Manual: File collections

Source: Gradle User Manual: File trees

Source: Maven: POM Reference, Resources

The features are older than most builds, so the release numbers are mostly of interest when a build file must also run on a very old Ant:

Nothing on this page is deprecated in Ant 1.10.18. The nearby tasks copydir, jlink and jspc that the manual lists as implicit-fileset users are deprecated; the migration advice is on Deprecations & Migration.

Java Development with Ant (written for Ant 1.5, 2002) teaches paths and filesets as the only vocabulary, because resource collections, multirootfileset, cache on paths, erroronmissingdir and the includeantruntime warning did not exist yet; its examples compile without includeantruntime, which prints a warning today and is something to set to false on every javac. Its default-excludes list predates the Git, Mercurial and Bazaar entries, so the .gitignore surprise above cannot be found in it. The path and fileset rules themselves have not changed, and the book’s chapter on them is still a good introduction. The Maven 2 and Gradle books do not cover Ant paths.