Metadata

org.apache.tika.metadata.Metadata is a multi-valued map from a String key to one or more String values. It is the parameter every Parser.parse call fills, the second result of Tika.parse, and the per-document record of a RecursiveParserWrapper result. It carries three kinds of information:

The distinction is the whole point of the 4.x key redesign: in 4.x every key that Tika itself computes lives under the tk: prefix, and a file can no longer write into that namespace. A document that carries a custom property literally named like a Tika key cannot overwrite the value Tika computed. The rest of this page builds on that idea.

Metadata is not the document text. In the Parser API the text goes to the ContentHandler; only the RecursiveParserWrapper and Tika Pipes put the handler output into metadata, under the key tk:content. See Introduction & Architecture for the four pillars.

Metadata is small. Its useful surface is:

All values are stored as strings; typed access is convenience over parsing. Every string value is stored well-formed: a lone UTF-16 surrogate is replaced by U+FFFD at write time (except for tk:content, which already went through a safe handler), so no serializer downstream meets one.

Metadata is not thread safe. Create one per parse and do not share it between threads while a parse runs.

Since Tika 2.0 a key is not just a string: it is usually a org.apache.tika.metadata.Property constant that knows its name, whether it is single- or multi-valued and what the value means. Property has two orthogonal characteristics.

Public factories create properties in a parser module’s own namespace, for example Property.internalText("myformat:title"), Property.internalTextBag(...), Property.internalDate(...), Property.externalInteger(...) and Property.internalClosedChoice("tiff:Orientation", "1", ..., "8"). The PropertyType of a key matters at write time:

Both lines were run on Tika 4.1.0: the second add throws PropertyTypeException and the typed read returns 2. In 4.x some former String constants became SIMPLE properties instead of bags, so code that called add twice on them now fails loudly rather than silently accumulating. Use set for a genuinely single value, or a bag-typed key if it can repeat.

Dates are stored as ISO 8601 strings in UTC. metadata.set(TikaCoreProperties.CREATED, new Date(0)) stores 1970-01-01T00:00:00Z, and getDate parses it back. Parsers write dates through the Date and Calendar overloads, so you can rely on this shape for Office, PDF and e-mail dates, but a value can still be absent or unparsable for a damaged file.

Source: Property, Javadoc 4.1.0.

Tika did not invent a metadata vocabulary. It maps each format’s native fields onto a handful of existing standards and exposes them as constants in tika-core (all in org.apache.tika.metadata). The first four rows are the ones to learn.

The class list was taken from the org.apache.tika.metadata package of the 4.1.0 tika-core sources, and every key spelled out above was read from those constants. The set is far larger than this table; the official Adding a Metadata Key page notes that every key Tika can emit is a Property constant or, for names minted at run time, a KeyPrefix, and that a generated registry (three JSON files under tika-metadata-schema) lists all of them.

A key falls into one of four families, and the family decides who may write it and how:

The next sections cover each family: first the tk: family and its migration, then the guard, then the KeyPrefix route.

If downstream consumers (an index schema, a dashboard) cannot move to tk: names yet, Tika does not ask you to fork it. The opt-in LegacyKeyMigrationFilter metadata filter rewrites keys at the emit edge using the same migration table. It works in either direction and is configured in the parse-context section of tika-config.json:

The default direction is V4_TO_V3: Tika’s 4.x output is rewritten into 3.x spellings. For input that is 3.x shaped and must be read with 4.x names, set direction:

An optional table overrides the bundled table with a custom one in the same row shape ({"v3": "...", "v4": "..."}, with "v4": "DROPPED" for a key that has no successor). The filter is also usable programmatically, as in this run against the metadata of catalogue.pdf:

Output on Tika 4.1.0:

The reverse direction (the default) turned the 4.x metadata of catalogue.pdf into keys such as X-TIKA:Parsed-By, X-TIKA:Parsed-By-Full-Set, X-TIKA:digest:SHA256, X-TIKA:versionCount and access_permission:can_print. The filter changes the metadata in place.

There is one limit worth knowing: families whose 3.x keys were completely bare (HDF, NetCDF, residual GDAL keys, ISA-Tab, QuickTime, audio SPI keys) are restored only when going 4.x to 3.x, because on ingest a bare name does not identify its family. The filter passes them through when migrating forward.

Source: Restoring 3.x Key Names, Tika 4.1.x and LegacyKeyMigrationFilter, Javadoc 4.1.0.

In 3.x a call such as metadata.set("X-TIKA:Parsed-By", value) simply succeeded, so a custom property in a hostile document, written by a parser through a string key, could replace a value Tika had computed. In 4.0 the string-keyed writes set(String, String) and add(String, String) throw IllegalArgumentException for any reserved name: a tk: key, and, during the 4.x line, the legacy X-TIKA: prefix.

On Tika 4.1.0 the first call fails with the message "Writing reserved key 'tk:resource-name' via the String API is not allowed", which names the three legitimate alternatives: the Property constant, add(KeyPrefix, ...) for document-derived names, and putAll or setTrusted for intentional reconstruction. The same message appears for X-TIKA:Parsed-By. A null value (a removal) is always allowed.

Three further consequences matter in production:

Source: Write API Changes, Metadata Changes in Tika 4.x.

When key names come from the document (HTML <meta name=...> tags, custom Office properties, CSV headers, e-mail headers) they cannot be constants. A parser declares a KeyPrefix once, as a static final field, and writes through Metadata#add(KeyPrefix, String, String):

The route has these guarantees: it is append-only (a repeated name accumulates values), it never throws on hostile input (a blank name, an over-length name, or a flood of distinct names is skipped with a warning), and source-typed dates have an Instant overload that stores canonical ISO 8601. Use KeyPrefix.file(...) for names read from the document and KeyPrefix.tool(...) for names coined by an external tool. The same route is what produced these keys in runs on Tika 4.1.0: message:raw-header:Message-ID and message:raw-header:MIME-Version for the e-mail, and custom:Shelf when the Office.USER_DEFINED prefix was used directly.

A parser module author should know that keys with a known, bounded vocabulary belong in curated Property constants, and that adding a key to TikaCoreProperties requires the build’s registry regeneration (tika-metadata-schema/regen.sh). That is contributor-level detail; the Extending Tika page covers custom parsers.

Source: Adding a Metadata Key, Apache Tika 4.1.x and KeyPrefix, Javadoc 4.1.0.

A metadata filter rewrites metadata after parsing and before it is emitted or serialized: the last chance to drop a field, rename it, normalize it or add a derived one. Filters run in tika-server, Tika Pipes and (for its -J output only, in 4.1.0) tika-app, on the whole list of metadata objects (the container document first, then each embedded document).

metadata-filters is a top-level array in tika-config.json. Filters run in the listed order, each seeing the previous output. An element is a filter name (default settings) or an object keyed by the name with its configuration. There is no default filter and no implicit filtering: omit the section and the metadata is emitted unchanged.

Optional modules add filters when on the classpath: the language detectors (charsoup-metadata-filter, optimaize-metadata-filter, open-nlp-metadata-filter), tika-eval-metadata-filter for extraction-quality scores, and the deprecated embedding filters of tika-inference (deprecated in 4.1.0, removed in 4.2.0 according to the 4.1.x documentation).

In 4.1.0, tika-eval-metadata-filter cannot be named in tika-config.json: the load fails with "Component …​ is not registered" because the tika-eval-core jar does not register it. Add it from Java instead, as tika-eval & Extraction Quality shows.

A hostile or just unlucky file can carry a huge amount of metadata: a million custom properties, a 50 MB value. The metadata limiter bounds the size of each parse’s metadata at write time, before memory is spent. It is the StandardMetadataLimiterFactory, configured under parse-context as standard-metadata-limiter-factory:

The defaults, read from the 4.1.0 source, are:

Some keys bypass the limiter so that Tika’s own machinery keeps working: Content-Type, Content-Length, Content-Encoding, Content-Disposition, tk:content, tk:resource-name, the tk:content-type-* hints and two PDF access-permission keys always get through, with a minimum field size of 300 bytes. When data is dropped, Tika sets tk:warn:truncated-metadata to true; check it before trusting that a record is complete.

To apply the limiter in code, put the factory on the ParseContext and create the Metadata with Metadata.newInstance(context):

This compiled and ran on Tika 4.1.0 against the sample PDF. With a maximum of two values the two-valued tk:parsed-by fit, so nothing was dropped and the warning key was absent. Parsers that create metadata for embedded documents should use Metadata.newInstance(context) rather than new Metadata(), so the same limits apply to children.

Source: Setting Limits, Apache Tika 4.1.x and StandardMetadataLimiterFactory, Javadoc 4.1.0.

Tika can hash documents while parsing, for deduplication, integrity checks and forensics. Digests land in metadata under tk:digest:<ALGORITHM> (hex encoding, the default) or tk:digest:<ALGORITHM>:<ENCODING> for other encodings, for example tk:digest:SHA-256:BASE32. The key carries the algorithm’s JCA name (SHA-256, SHA3-512), while the configuration names it by the enum spelling (SHA256, SHA3_512). Two factories are configured in parse-context:

Encodings are HEX (default, lowercase), BASE32 and BASE64, both RFC 4648. In 3.x the key was X-TIKA:digest:SHA256, so a digest-keyed index needs the LegacyKeyMigrationFilter or a migration.

skipContainerDocumentDigest set to true digests only the embedded documents of a container (say, the PDFs inside a ZIP) and not the container. Programmatically:

Run on Tika 4.1.0 against the 785-byte catalogue.pdf, the keys were tk:digest:SHA-256 (a 64-character hex value) and tk:digest:MD5:BASE32 (a base32 value padded with =). The digest needs the bytes: Tika reads the stream, spooling it to a temporary file when the digester requires it, so enabling digests adds I/O. On the command line, tika-app's --digest=<algorithm> covers the Commons algorithms only; for SHA3 use a JSON configuration with the BouncyCastle factory.

Source: Configuring Digesters, Apache Tika 4.1.x.

The table lists the keys worth storing for each file of the intake scenario. Rows marked run were produced by a parse on Tika 4.1.0 of the stand-in sample; rows marked source come from constants in the 4.1.0 tika-core source and were not observed with a real file (the book-cover.jpg, scanned-loan-form.pdf and novel.epub samples were not run for this page).

The last row is the point of the example on the Media Types & Detection page. The same PDF, called renamed-invoice.png, produced the Content-Type application/pdf, kept tk:resource-name = renamed-invoice.png and added tk:content-type-magic-detected = application/pdf. An intake service should compare what the file name claims with Content-Type and reject or quarantine a mismatch. See Media Types & Detection for validating uploads by content.

Verified for this page:

Not verified: