Character Encoding Detection

The publisher’s price list is a five-line text file saved in windows-1252 with Windows line endings. It was generated for this page, with only characters that exist in windows-1252 and that make the file invalid UTF-8:

The first bytes show why guessing is hard:

E1 is á, 97 is the em dash, F3 is ó, AB is « and 80 (on a later line) is the Euro sign. None of these bytes forms a valid UTF-8 sequence, and 80 and 97 are control characters in ISO-8859-1, which is why labelling such files "Latin-1" is a common but wrong shortcut. Windows-1252 is the superset in which they are printable.

The first Java experiment decodes the same bytes with two wrong guesses and then asks Tika:

Output on Tika 4.1.0 (the question marks are the Unicode replacement character that the terminal shows for bytes that are not valid UTF-8):

Reading the output: decoding as UTF-8 destroys every non-ASCII byte; the default chain returns a single candidate, windows-1252, with confidence 1.00 and the result type STATISTICAL; the metadata records the winner, the detector that proposed it and a trace of all detectors that took part. The rest of this page explains each of those words.

Source: DefaultEncodingDetector, Javadoc 4.1.0 and AutoDetectReader, Javadoc 4.1.0.

Each candidate is an EncodingResult:

The three types matter because the default chain does not simply take a declaration at its word; see JunkFilterEncodingDetector: the arbiter.

MetaEncodingDetector is a marker sub-interface of EncodingDetector. A detector that implements it does not look at bytes first; it receives the candidates of all the other detectors and picks among them. Only one meta detector is supported in a chain. If a second is found, CompositeEncodingDetector logs a warning and ignores it. In the default chain the meta detector is JunkFilterEncodingDetector.

The 3.x interface is much smaller, and this is the form the book Tika in Action (chapter 4) describes for 1.x:

It returns a single Charset or null, with no confidence and no result type, and a composite takes the first non-null answer. Custom detectors written for 3.x must be rewritten for 4.x: the stream becomes a TikaInputStream, a ParseContext is added and the return type becomes a list of EncodingResult.

Source: EncodingDetector, Javadoc 4.1.0, EncodingResult, Javadoc 4.1.0, MetaEncodingDetector, Javadoc 4.1.0 and EncodingDetector, Javadoc 3.3.2.

How the members cooperate depends on one question: is a MetaEncodingDetector in the chain?

The 4.x default is collect-all on purpose: the Tika documentation states that the trade-off is "occasional overrides of correct declarations on short probes for resilience against lying declarations and wrong-codec decodings". Section Configuring the chain shows how to choose the other mode.

Source: Apache Tika 4.1.x, Configuring Encoding Detectors and Apache Tika 4.1.x, Charset Detection Pipeline.

A byte-order mark is a few bytes at the start of a file that announce its Unicode encoding. BOMDetector reads the first four bytes (it needs at least two) and compares them with this table:

The result is DECLARATIVE with confidence 1.0. The HTML detectors deliberately ignore BOMs, so BOMDetector is the only source of BOM evidence, which lets the arbiter decide when a BOM and a <meta> tag disagree. A UTF-8 BOM is optional and many files lack it; a file that has one is not necessarily valid UTF-8 afterwards, so the BOM is evidence, not a guarantee. AutoDetectReader drops a leading U+FEFF character after decoding (the code comments refer to TIKA-240) so that the BOM does not show up as a stray character in the text.

Run on a file that starts with EF BB BF, the default chain reported (4.1.0):

Both detectors agreed on UTF-8, and the trace credits the BOM detector, the first to propose the winning charset.

This detector looks at the Metadata before any byte is read and checks three keys in this order:

It normalises labels with the WHATWG rules. In particular ISO-8859-1 and US-ASCII are normalised to windows-1252, because browsers and the HTML standard treat them as aliases in practice. The result is DECLARATIVE.

This is how you pass the information that your caller already has. An upload handler that knows the HTTP Content-Type header, or a mail parser that has read a charset= parameter, should put it into the metadata rather than hope the detector re-derives it:

Output on Tika 4.1.0:

The header said ISO-8859-1, Tika normalised it to windows-1252, the statistical detector independently agreed, and the declaration is credited. Note what the header would have been worth if it had been wrong: in the default chain the arbiter can still override it, as the next section shows.

Source: MetadataCharsetDetector, Javadoc 4.1.0 and Apache Tika 4.1.x, Configuring Encoding Detectors.

HTML declares its charset inside the document, in <meta charset="windows-1252"> or in the older <meta http-equiv="Content-Type" content="text/html; charset=windows-1252">. HtmlEncodingDetector finds it with a fast, lenient regular expression over the first markLimit bytes (8,192 by default, see TIKA-2485 for why scripts before the tag are a problem), and maps the label through a curated subset of the WHATWG alias table (TikaHtmlCharsetAliases).

StandardHtmlEncodingDetector implements the full WHATWG pre-scan algorithm. Use it when leniency produces false matches, for example when a charset declaration appears inside an HTML comment that the regular expression should have skipped. It is not in the default chain.

Mojibuster is the statistical heart of the 4.x chain. Its layers run in order, and the first confident structural answer ends the run; otherwise the bytes fall through to the classifier:

The classifier has several safeguards that explain behaviour you will see on small files:

Run on 4.1.0, the small cases show these safeguards:

The model sources and the training procedure are described in the design document. The same document states plainly that no accuracy figures are published, because the last full evaluation predates a requantization of the model and no test asserts an accuracy floor; it gives only a qualitative comparison, which is that Mojibuster roughly doubles the strict accuracy of ICU4J and roughly triples that of juniversalchardet, at an order of magnitude less latency than ICU4J. Treat that as the project’s own claim, which this page did not reproduce. The design document also names EvalCharsetDetectors as the tool for measuring the shipped model on your own corpus.

JunkFilterEncodingDetector is a MetaEncodingDetector backed by JunkDetector, a script-aware scorer that tells clean text from mojibake, wrong-codec decodings and corrupted bytes. Its presence switches the composite into collect-all mode. It arbitrates over all candidate types, DECLARATIVE included, because real-world declarations are unreliable. The outline of its decision:

Because declarations are only one input, a lying <meta> tag does not defeat the default chain. The second sample is reviews-es.html's cousin, a Spanish review page whose bytes are windows-1252 but whose <meta> tag says UTF-8:

Output on Tika 4.1.0:

The two detectors disagreed, the arbiter decoded both ways, found that UTF-8 produced replacement characters and windows-1252 produced clean Spanish, and chose windows-1252 (junk-filter-selected in the trace). With an honest declaration the outcome is the same, and the declaration is credited:

Source: Apache Tika 4.1.x, Charset Detection Pipeline and Apache Tika 4.1.x, Text Quality Scoring (Junk Detection).

default-encoding-detector is the SPI-discovered chain, and exclude drops members of it:

Run on the HTML sample, the chain then consisted of Mojibuster only (MojibusterEncodingDetector->windows-1252), the <meta> tag being ignored. Do not put default-encoding-detector and explicit detectors in the same list: the loader wraps everything in an outer composite that has no meta detector at its top level, so the collect-all arbitration is lost silently and the explicit detectors are never reached (the 4.1.x documentation warns about this).

An explicit ordered list replaces the discovered chain. End it with junk-filter-encoding-detector to keep collect-all arbitration; omit it for first-match-wins. This is also the only way to set an option of an individual detector, such as the HTML detector’s read limit:

markLimit is the number of bytes the HTML detector scans for the <meta> tag. The default is 8,192. Raise it if your pages put large <script> blocks before the tag. Mojibuster reads a larger probe of its own, so with the default chain the HTML limit matters mainly for very large preambles. Loaded with TikaLoader on 4.1.0, this configuration produced the expected three-detector chain (the HTML candidate and the Mojibuster candidate both appeared in the trace); the effect of a raised markLimit itself was not measured.

When you can trust the declarations (a controlled upstream that labels its files correctly) and want predictable, cheaper behaviour, drop the meta detector and list the detectors in priority order:

The classifier then runs only when no declaration fires. The cost shows with the lying page from the previous section. Parsing lying.html with a TikaLoader-built parser using this chain gave:

The false declaration won and the text is corrupted, which is exactly the trade-off the documentation describes: "lying declarations propagate unfiltered". Choose this mode only for sources you control.

For a corpus whose encoding you already know, configure the override detector, which always returns its charset (default UTF-8):

On 4.1.0 this chain returned windows-1252 as DECLARATIVE for the HTML sample regardless of content.

Tika 3.x used a first-match-wins chain of the HTML detector, juniversalchardet (UniversalEncodingDetector) and ICU4J (Icu4jEncodingDetector). Both of the latter are still published in 4.1.0 (tika-encoding-detector-universal and tika-encoding-detector-icu4j; the ICU4J module is a vendored copy of ICU4J’s CharsetDetector), but they are neither bundled in the tika-app or tika-server-standard packages nor auto-discovered. You have to do two things, and either one alone is not enough:

Run on 4.1.0 with both jars added, for legacy-catalogue.txt the chain answered windows-1252, credited to UniversalEncodingDetector, at a confidence of 0.50 (against 1.00 for Mojibuster); for the Cyrillic file it answered windows-1251, also at 0.50. With the jars on the classpath but no configuration, the default chain was unchanged and Mojibuster still decided, confirming that these two are config-only. Whether to go back to the 3.x chain is a question for your corpus, not for this page: measure both on a labelled sample of your files. The only published comparison is the project’s design document, quoted above.

The 3.x equivalent of the exclude example used tika-config.xml with an <encodingDetectors> element and the TikaConfig class, which is removed in 4.x. The 4.x conversion tool, --convert-config-xml-to-json, migrates only the parsers section (see the migration page of the 4.1.x documentation); recreate the encoding-detectors array by hand.

Source: Apache Tika 4.1.x, Configuring Encoding Detectors, DefaultEncodingDetector, Javadoc 4.1.0 and Apache Tika 4.1.x, Migrating to Tika 4.x.