tika-eval & Extraction Quality

Extraction rarely fails loudly. A parser that throws is easy to see; the expensive failures are the quiet ones:

There is usually no ground truth: nobody has a hand-checked transcript of every document in a corpus of a million files. tika-eval therefore measures proxies that do not need one. The main proxy is the share of common words: real text in a language contains many of that language’s most frequent words, while mojibake and junk do not. Others are the rate of U+FFFD replacement characters, a learned text-quality score, the detected language, the amount of text per page and the exceptions recorded during the parse. Compared between two runs over the same files, these proxies become a regression test.

The Tika wiki lists the questions tika-eval was built for: comparing two different tools, two versions of the same tool ("should we upgrade?"), two runs with different settings (for example OCR at 200 against 300 DPI), and a tool against a gold standard. The project runs it against about three million files as part of its regression testing; the wiki also states that the design does not scale to hundreds of millions of files.

Source: TikaEval, Tika wiki (written in the 1.x to 2.x era, last updated 2022) and Apache Tika 4.1.x, Regression evaluation with tika-app + tika-eval.

Like tika-app and tika-server-standard, tika-eval-app 4.x is a thin launcher: the jar on Maven Central has a manifest class path that points at lib/ and fails on its own. Download the zip, check it and unpack it into a new directory (the zip has no top-level folder):

Output:

The zip (41 MB) holds the launcher jar, LICENSE, NOTICE and 43 jars in lib/, among them tika-core, tika-eval-core, tika-langdetect-charsoup, tika-ml-junkdetect, tika-encoding-detector-mojibuster, tika-pipes-core, tika-pipes-file-system (the directory crawler), tika-serialization, h2 and poi-ooxml. It does not contain parsers: tika-eval never parses your documents. It reads the extracts another tool wrote. The 3.x line ships tika-eval-app-3.3.2.jar as a runnable fat jar instead.

Source: Apache Tika, Download and Maven Central, org.apache.tika (artifact listings checked on 2026-10-13).

Every tika-eval run has the same three stages: write extracts, load them into a database, write reports from the database. Profile loads one set of extracts; Compare loads two sets of extracts of the same corpus and pairs them file by file.

Two consequences of this design matter in practice. First, tika-eval evaluates any tool that can write the extract format, not only Tika: a directory of UTF-8 .txt files from another extractor can be one side of a comparison. Second, the database survives the run (when you name it with -d), so you can write new reports or run your own queries long after the extraction finished.

Source: Apache Tika 4.1.x, Regression evaluation, Concept.

Profile answers "what did this run produce?". It needs the extracts directory (-e) and a database name (-d); the original files (-i) are optional but recommended, because with them tika-eval also records the files that produced no extract at all.

Profile creates bookshelf-profile.mv.db in the current directory and logs:

Report then writes 16 workbooks:

Each workbook has the result on its first sheet and, because the report definitions set includeSql="true", the SQL that produced it on a second sheet: a good way to learn the schema. A profile database gets no summary.md; the log says Not a comparison database; skipping markdown summary.

The per-document numbers behind those workbooks, read straight from the database (the query is in Querying the H2 database):

(The long Office media type is shortened here.) Thirteen documents from eight files: the e-mail contributes its five embedded documents as their own rows, each with its own metrics. empty.bin has a profile row but no content row, and an exception row with type RUNTIME (the ZeroByteFileException the parse recorded). Language ids are ISO 639-3 codes from CharSoup: spa, fra, eng.

Source: TikaEval, Tika wiki, Single Output from One Tool (Profile); command forms from the ExtractProfileRunner options in the tika-eval-app 4.1.0 source jar.

tika-eval computes the same text profile for every document in Profile and for each side in Compare, then adds the pairwise contrast in Compare. The column names are those of the contents, profiles and content_comparisons tables (contents_a and contents_b in a comparison database).

This is the official worked example of the 4.1.x documentation, applied to the Bookshelf corpus. The documentation compares the default encoding-detector chain against an explicit chain over 29,000 HTML pages from Common Crawl. Here the variant is deliberately bad, so that the reports have something to find: configuration B replaces the detector chain with a BOM check and an override that always says UTF-8. The two configurations differ in exactly one thing, the field under test; the digester is identical.

tika-config-A.json is the digester-only file shown above. tika-config-B.json:

An explicit encoding-detectors array replaces the SPI-discovered chain completely; see Character Encoding Detection for the detector names.

The Compare keyword is optional: with both -a and -b the CLI infers it. -r runs Report right after Compare and packs the reports directory into reports.tgz; -d keeps the database as bookshelf-A-vs-B.mv.db (94 KB here). The whole comparison took 11 seconds of wall time, most of it JVM start-up and the 67 report queries.

The quickest overview is reports/summary.md, new in 4.x. Extracts from the run for this page (empty sections and the long paths of the provenance tables removed):

Every signal points at the same file, and each one tells a different part of the story:

The same summary also lists a parse-time table. Ignore it on a corpus this small; see Memory, performance and scaling.

Source: Regression evaluation, Worked example: encoding-detector chain comparison.

The question tika-eval was first built for is "should we upgrade?". The recipe is the same as above, with the version as the variable instead of the configuration. Use the same corpus, the same output form (plain text on both sides here, since 3.x has no Markdown handler) and, where possible, the same digests and timeouts.

Thanks to the legacy-key normalization, the 3.3.2 extracts (X-TIKA:content, X-TIKA:digest:MD5) and the 4.1.0 extracts (tk:content, tk:digest:SHA-256) were paired without any preprocessing. What the comparison found:

A query for the two flipped documents (the SQL is in the next section) shows that the change is a fix:

3.3.2 decoded legacy-catalogue.txt as ISO-8859-1. The decoder itself does not fail: it maps byte 0x80 of each € sign to the C1 control character U+0080. Tika’s XHTMLContentHandler treats U+007F to U+009F as invalid in XHTML and writes U+FFFD in their place, so the two € signs came out as replacement characters (checked by decoding the file with AutoDetectReader, which returns U+0080, and with tika-app-3.3.2.jar -t, which prints U+FFFD). 4.1.0 detected windows-1252 and decoded them correctly. Note what did not move: OOV, the token counts and the dice coefficient (1.0) are identical, because € is not a word token. Token metrics are blind to symbols, currency signs and punctuation; the replacement-character count and the charset reports are not. Read them together.

Three more observations from the same run, each of them a lesson for real upgrades:

A practical pre-upgrade checklist:

Source: TikaEval, Tika wiki, Background (the upgrade use case) and the ExtractReader legacy-key table in the tika-eval-app 4.1.0 source jar. See What’s Changed & Migration for what else to check when you move from 3.x to 4.x.

The reports are pre-canned views. For anything else, query the database directly. The rules, all confirmed in the runs for this page:

A comparison database has 29 tables: containers, profiles_a/_b, contents_a/_b, content_comparisons, encodings_a/_b, exceptions_a/_b, exceptions_compared, extract_exceptions_a/_b, emb_path_a/_b, tags_a/_b, mimes, pair_names, parse_time_compared, token_counts_compared, pipes_class_a/_b, run_info/_a/_b and the three reference tables. A profile database has the same tables without the suffixes (14 in all). A and B are paired by id: the same id is the same document in both runs, and every join between the two sides uses it.

The OOV direction count from the official documentation, on the encoding comparison:

The documents whose text changed, or that gained replacement characters, with all the signals side by side:

Sixteen replacement characters out of sixteen non-ASCII characters: a 100 % decode-failure rate for the non-ASCII part. Divide by num_non_ascii, not by the content length, which dilutes the rate to almost zero on mostly-ASCII text.

The charset flips with the detector that made each decision:

The common-token headline ("did B recover more real text?"):

And, on the profile database, the per-document listing shown in the Profile section:

The documentation adds a caution worth repeating: OOV is one signal, not the verdict. A language-id shift (for example a CJK text that B decodes correctly for the first time) can raise OOV although B is right, and a wrong decode that fragments words into short common-looking tokens can lower it. Languageness has the opposite blind spots. When the two disagree on a file, look at the file.

StartDB starts the H2 web console (org.h2.tools.Console -web, by default on http://localhost:8082) for browsing a database in a browser; enter the JDBC URL with the full path and empty credentials. It runs until you stop it.

Source: Regression evaluation, Querying the H2 database directly and H2, Shell tool.

A crash (OOM, TIMEOUT, UNSPECIFIED_CRASH) leaves no extract, so on its own tika-eval reports it as NO_EXTRACT_FILE, the same as a file that never ran. Since 4.1.0 two inputs close the gap:

When these flags are absent, tika-eval looks in <extracts>/.run-info/ for crashes-*.jsonl and run-info-*.json (it refuses to start if that directory holds more than one of either, since that means two runs wrote into the same extracts directory). The .run-info directory is skipped by the crawler and excluded from the extracts fingerprint. A reporter block that writes the ledger where tika-eval looks for it, with successes left out:

On the Bookshelf corpus nothing crashed, so the ledger held a single line, for the zero-byte file:

Profile found and joined it without any flag (loaded 1 pipes report rows (0 pipeline error lines) in the log), and the result lands in containers.pipes_status and pipes_message (_a and _b on Compare), and in run_info as pipes_report.rows = 1 and pipes_report.joined = 1. A joined of zero with a non-empty ledger means the wrong ledger, or a crawl without -i, which never sees the files that crashed. The reports classify each container:

The run_info tables also record, without being asked, the tika-eval version, the SHA-256 of its jar, Java, host, user, the arguments, the effective EvalConfig, start and end times, and for each extracts directory the file count and a fingerprint (SHA-256 over the sorted names and sizes). summary.md opens with them, so a report always says what produced it. Password-like values in the JDBC string, arguments and configuration are masked, since these tables travel with the reports.

The Tika source tree has a helper, .skills/devs/tika-eval-compare/run-batch.sh, that runs tika-app and writes both files (run-info-<run.id>.json with the SHA-256 of every jar and of the configuration, and the ledger). It is not part of the distribution zips and was not run for this page. If a run-info.json carries run.id, the ledger must be named crashes-<run.id>.jsonl or tika-eval refuses to start.

Source: Regression evaluation, Joining the batch crash ledger and run provenance and Apache Tika 4.1.x, Pipes Reporters; reporter keys on the Tika Pipes page.

You do not need the whole tika-eval-app to use its metrics. TikaEvalMetadataFilter (package org.apache.tika.eval.core.metadata, artifact tika-eval-core) is a metadata filter: it reads tk:content from a Metadata object and writes the scores next to it. That turns quality into a per-document field your pipeline can act on before the text reaches an index or an embedding model.

The filter does nothing when tk:content is blank. The 3.x filter writes the same keys except languageness, and uses OpenNLP for the language.

Token metrics cannot see order. A parser change that reorders the columns of a PDF, splits a paragraph or glues two words keeps dice near 1.0. tika-eval-structure (new in 4.1.0, Maven Central only) compares two directories of XHTML extracts block by block: it matches paragraphs, headings, list items and cells one to one by token overlap and reports Kendall tau and adjacent-pair agreement for the order, splits and merges, and words glued on one side and split on the other. Its summary groups files by a metadata key of A (-g, default pdf:producer).

The jar expects its dependencies in lib/ next to it; the lib/ directory of tika-eval-app 4.1.0 contains all of them (tika-core, tika-serialization, commons-cli, Jackson, log4j), which is how it was run here:

As expected for a charset-only change, every file kept a tau of 1.000. The unreadable file is empty.bin, whose extract has no XHTML. The options are -a, -b, -o, -g, -d (the least token dice for two blocks to match, default 0.5) and -s (a suffix to drop from B names, for comparing against labels made from a source document). Use the tool when the variable under test is a PDF parser, layout or reading-order setting; see PDF Parsing.

Source: Regression evaluation, Structure: order, segmentation and word boundaries and the --help output of tika-eval-structure-4.1.0.jar.

The first argument names the tool. When it is a flag instead, the tool is inferred: -a with -b means Compare, -e means Profile, -d alone means Report.

The -c file maps onto EvalConfig. Its keys are minExtractLength (default 0), maxExtractLength (100,000,000), numWorkers (4) and jdbcDriverClass; the class also declares jdbcString, forceDrop, maxFilesToAdd, maxTokens and maxContentLength, but in the 4.1.0 source nothing reads them. Command-line -n and -m override the file:

The run with this file recorded EvalConfig{minExtractLength=10, maxExtractLength=20000000, ..., numThreads=2, ...} in run_info.

Options of the 3.x tools that no longer exist in 4.1.0: -drop, -alterExtract (as_is, first_only, concatenate_content), -minExtractLength as a flag, -maxTokens, -maxContentLength, -maxContentLengthForLangId, -defaultLangCode, -bc and -numConsumers (tika-batch), -tablePrefixA/-tablePrefixB on Compare, and the FileProfile tool. See Pitfalls for what happens if you still type them.

Source: the option definitions of TikaEvalCLI, ExtractProfileRunner, ExtractComparerRunner and ResultsReporter in the tika-eval-app 4.1.0 and 3.3.2 source jars on Maven Central; the Report usage text printed by the 4.1.0 jar.

The 4.0.0 release was not inspected separately; the "4.0 and 4.1" row describes 4.1.0.

Verified on 2026-10-13 by running it:

Not verified: