Language Detection & Translation

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

Baseline date: 2026-10-12. This page was written against Apache Tika 4.1.0 (the latest release on Maven Central on that date) with the 3.3.x differences noted, Java 21 and later for the examples (Tika 4.x needs Java 17), Spring Boot 4.x on Spring Framework 7 and the Docker image apache/tika:4.1.0. Language identification is the fourth of the four pillars of Tika (detection, parsing, metadata and language): once the text is out of the file, Tika can tell you which human language it is written in. This page explains how that works (character n-grams and a statistical model), which detector implementations ship with 4.1.x, how to call them from Java, from the metadata pipeline, from the command line and from the server, how far to trust the answer, and what is, and is not, left of the translation API.

The language-detection part is current and recommended. The translation part is a different story: the tika-translate module still ships in 4.1.0, but nothing in the 4.x documentation describes it, the server endpoints for it were removed in 4.0, and every bundled implementation wraps either a third-party commercial API that this documentation could not exercise or a local tool you must install yourself. The honest summary is in The bundled translators and their maintenance status. Read that section before you plan anything on top of it.

The detectors and metadata keys used here are the same ones the rest of the section relies on: the output formats and handler types are in Content Handlers & Output Formats, the tk: metadata namespace is in Metadata, the encoding step that runs before any language detection is in Character Encoding Detection, the parse call that feeds the text is in Parser API & AutoDetectParser, and dependencies are in Setup & Installation.

All the Java examples on this page were compiled and run against Tika 4.1.0 on Java 25 in a throwaway Maven project (tika-core, tika-parsers-standard-package as a pom dependency, tika-langdetect-charsoup, tika-langdetect-opennlp, tika-translate and tika-serialization). The output shown is the real output of those runs, with the exceptions listed explicitly in What this page did not verify. The sample files are small stand-ins for the Bookshelf files: reviews-es.html (two paragraphs in Spanish), reseñas-fr.txt (four lines in French), review-en.txt (two sentences in English) and mixed.txt (three sentences, one in each language).

Why the intake service wants a language

The Bookshelf intake service receives reviews and catalogue descriptions from readers and publishers in several languages. The language of a text is not a curiosity: it drives decisions further down the pipeline.

  • Search analysis. A Solr or Elasticsearch field needs a language-specific analyzer (stemming, stop words); choosing it needs the language of the text. See the existing pages Solr indexing and updates for where the analyzer is chosen.

  • RAG and embeddings. Chunks of a multilingual corpus are tagged with their language so retrieval can filter or re-rank by it, and so a monolingual embedding model is not fed text it was not trained on. See Document parsing for RAG.

  • Routing and review. A French review can go to the French-speaking editor; text in an unexpected language can be flagged for a human.

  • Quality control. A file that claims to be an English manuscript but whose extracted text is detected as another language, or as nothing at all, is a good candidate for an extraction problem (an encoding mistake or garbled OCR).

  • Translation. You must know the source language before you ask a translator for a target one (see The Translator API).

The other thing language is not is a property of the file format. A Word document can carry a language tag in its styles, an HTML page can say lang="es" and a PDF can have a /Lang entry, but those are declarations made by the author or the software that produced the file. They are frequently missing, frequently wrong (a template left in English around Spanish text) and always a single value for a document that may contain several languages. Language detection looks at the text itself. The two are complementary and a good intake pipeline stores both, as shown in Declared versus detected language.

N-gram language identification: the theory

This section is the idea behind every detector below. It is the content of chapter 7 of Tika in Action, whose listings (LanguageIdentifier, LanguageProfile, ProfilingHandler) no longer exist, brought up to date.

Character n-grams and profiles

An n-gram is a run of n consecutive characters. The text the cat contains the trigrams the, he ` (with the space), `e c, ` ca` and cat. Every language has characteristic n-grams: th, the, ing and ion are frequent in English, ción and que in Spanish, eau and ent ` in French, `sch and ung in German, and whole scripts have their own character ranges (Cyrillic, Hangul, Devanagari). Counting how often each n-gram occurs gives a profile, a sort of fingerprint, of a language.

The classic method, from the 1994 paper by Cavnar and Trenkle that both the book and the 4.1.x documentation cite, builds a ranked profile of the most frequent n-grams for each language from training text, builds the same profile for the unknown text, and picks the language whose profile is nearest by an "out-of-place" distance (how far each n-gram has moved in the ranking). It is simple, needs no dictionaries, works on any script and is robust to spelling errors, which is why it became the standard first approach to the problem.

From rank distance to a trained classifier

Modern detectors keep the n-gram idea but replace the hand-built ranking by a trained statistical model. Instead of comparing ranked lists, the detector turns the text into a numeric feature vector (how often each hashed n-gram occurs), and a linear classifier trained on a large multilingual corpus turns the vector into one score per language. The 4.1.x documentation describes this for the default detector: character n-grams, short complete words, word bigrams and per-script counts are hashed into a fixed-size vector, normalised, and multiplied by a weight matrix with one row per language.

The scores are raw logits. They have no absolute meaning on their own; what matters is how far the best language is from the runner-up. This leads to the confidence definition that the 4.1.x documentation gives for the default detector:

where \(z_1\) and \(z_2\) are the scores of the best and second-best languages (after groups of confusable languages are merged). The value is close to 0.5 when two languages are almost equally likely and approaches 1 when one clearly wins. Because it depends on a difference, the same threshold means the same thing for a one-line snippet and for a whole document, and for a detector that knows 100 or 200 languages.

The pipeline in one picture

flowchart LR text["Text: characters from a handler, a reader or a string"] pre["Preprocessing: truncate to 100,000 characters, strip URLs and emails, NFC normalise, skip marks, case fold"] feat["Feature extraction: character n-grams, short words, word bigrams, script counts"] hash["Hashing into a fixed vector: 32,768 buckets, L2 normalised"] model["Model: linear classifier with INT8 weights, one score per language"] merge["Merge confusable groups: best score minus runner-up"] res["LanguageResult: language, LanguageConfidence, raw score"] text --> pre --> feat --> hash --> model --> merge --> res res --> use{"isReasonablyCertain?"} use -- yes --> keep["Store the language and route by it"] use -- no --> flag["Store as unknown or flag for review"]

Source: Apache Tika 4.1.x, Language Detection (the stages, the 100,000-character truncation, the 32,768 buckets and the confidence definition are from this page; the final decision box is the intake service’s own policy, shown in Routing by language).

What the theory predicts, and where it breaks

  • Short text is hard. An n-gram model needs evidence. A single word, a name or a code has few n-grams and many of them occur in several languages. The 4.1.0 run below detects Hola as Tetum (tet, LOW) rather than Spanish, and the catalogue number ISBN 978-84-376-0494-7 as Maltese (mlt, LOW). Both are "reasonable" guesses for a statistical model and both are wrong for a human; the confidence is the signal that tells you so.

  • Closely related languages are hard. Spanish and Galician, Serbian and Croatian, Malay and Indonesian, Norwegian Bokmål and Danish differ in few n-grams. The CharSoup detector ships a list of confusable groups (confusables.txt inside the tika-langdetect-charsoup jar) and merges them for confidence purposes, so that a genuinely ambiguous pair gets a low confidence instead of a falsely sharp one.

  • Mixed text has no single answer. A page that contains a French paragraph, a Spanish one and an English one has no one language. detect returns the best; detectAll returns the ranked list so you can see the mixture (an example is below).

  • Boilerplate pollutes the sample. Menus, legal footers and cookie banners are written in the site’s language, not the article’s. Detect on the main text where you can.

  • Text that is not language. Numbers, code, tables of ISBNs and lists of names produce unstable results.

  • Script is a strong, cheap signal. A text in Hangul or Thai is almost never confused with a Latin-script language, which is why the model gives the classifier explicit per-script counts.

Dependencies

Language detection lives in separate modules so you only pull in what you use.

Artifact What it provides Notes

tika-core

The SPI types in org.apache.tika.language.detect: LanguageDetector, LanguageResult, LanguageConfidence, LanguageHandler, LanguageWriter, LanguageNames; and org.apache.tika.language.translate: Translator, DefaultTranslator, EmptyTranslator

No implementation. On its own LanguageDetector.getDefaultLanguageDetector() has nothing to load.

tika-langdetect-charsoup

CharSoupLanguageDetector and CharSoupMetadataFilter (friendly name charsoup-metadata-filter)

Recommended. Brings tika-langdetect-charsoup-core (the model, langdetect-20260320.bin, 204 languages) with it. In the test project it also arrived through tika-parsers-standard-package.

tika-langdetect-opennlp

OpenNLPDetector and OpenNLPMetadataFilter (friendly name open-nlp-metadata-filter)

The alternative. Model opennlp-langdetect-20210413.bin inside the jar. It was not pulled in by tika-parsers-standard-package in the test project, so add it explicitly.

tika-translate

The translator implementations in org.apache.tika.language.translate.impl

Optional, see The bundled translators and their maintenance status. Depends on tika-langdetect-charsoup plus several HTTP and client libraries.

 
<dependencies>
  <dependency>
    <groupId>org.apache.tika</groupId>
    <artifactId>tika-core</artifactId>
    <version>4.1.0</version>
  </dependency>
  <dependency>
    <groupId>org.apache.tika</groupId>
    <artifactId>tika-langdetect-charsoup</artifactId>
    <version>4.1.0</version>
  </dependency>
  <!-- optional: the OpenNLP detector as a second opinion -->
  <dependency>
    <groupId>org.apache.tika</groupId>
    <artifactId>tika-langdetect-opennlp</artifactId>
    <version>4.1.0</version>
  </dependency>
</dependencies>
dependencies {
    implementation("org.apache.tika:tika-core:4.1.0")
    implementation("org.apache.tika:tika-langdetect-charsoup:4.1.0")
    // optional: the OpenNLP detector as a second opinion
    implementation("org.apache.tika:tika-langdetect-opennlp:4.1.0")
}

Source: Apache Tika 4.1.x, Language Detection and Maven Central, tika-langdetect-charsoup 4.1.0.

Detector discovery uses the service mechanism, so what is on the classpath decides what is found. With both detector modules present, LanguageDetector.getLanguageDetectors() returned the CharSoup detector first and the OpenNLP detector second, and LanguageDetector.getDefaultLanguageDetector() returned the CharSoup one. The default is therefore whichever detector discovery finds first; with only one of the two modules on the classpath you get that one (the single-module case was not run), and with neither there is no implementation to load.

The LanguageDetector SPI

org.apache.tika.language.detect.LanguageDetector is an abstract class in tika-core. A detector accumulates text and then answers.

Member Purpose

static LanguageDetector getDefaultLanguageDetector()

The first detector found by service discovery. Not loaded yet.

static List<LanguageDetector> getLanguageDetectors()

All detectors found on the classpath, in discovery order.

loadModels()

Loads the model for every language the detector supports. Must be called once before use. Returns the detector, so it chains.

loadModels(Set<String> languages)

Loads only the listed languages. This restricts the candidate set and can shrink memory for detectors that load a model per language.

hasModel(String language)

Whether the detector has a model for a language code.

reset() and reset(ParseContext)

Clears the accumulated text so the detector can be reused. Detectors are stateful; reset between documents.

addText(char[], int, int) and addText(CharSequence)

Adds text to the buffer. Can be called repeatedly (incremental use).

hasEnoughText()

true once enough text has been added for a reliable answer. A streaming caller can stop feeding then.

detect()

The best match for the accumulated text, as a LanguageResult.

detectAll()

All candidates, best first.

detect(CharSequence) and detectAll(String)

Convenience: reset, add the text, detect.

setMixedLanguages(boolean) and isMixedLanguages()

A hint that the text may mix languages.

setShortText(boolean) and isShortText()

A hint that the text is short.

setPriors(Map<String, Float>)

Language priors, where the implementation supports them (see the warning below).

Two details matter in practice. First, detect(CharSequence) calls reset() for you, whereas addText followed by detect() does not, so a long-lived detector shared between documents must be reset explicitly. Second, the detector object is not thread safe: it holds the accumulated text. Create one detector per thread, or one per task, or guard it. The models are loaded per detector instance and are the expensive part (the CharSoup model is a 6.4 MB file), so create detectors once and reuse them for many documents rather than per file.

The 4.1.x documentation lists setPriors(Map) as a tuning knob of CharSoupLanguageDetector. In the 4.1.0 run for this page, CharSoupLanguageDetector.setPriors(...) threw UnsupportedOperationException: Priors are not supported. The method exists because the base class declares it, not because the CharSoup implementation honours it. The tuning that works for CharSoup is setMaxLength(int) (the size of the rolling text buffer) and the choice of loadModels(Set). Priors on OpenNLPDetector were not tested.

LanguageResult and LanguageConfidence

LanguageResult is what a detector returns.

Member Meaning

getLanguage()

The language code. For the bundled detectors these are ISO 639-3 codes (eng, spa, fra), apart from three orthography variants of the CharSoup model that carry a private-use subtag (cdo-x-rom, hak-x-rom, be-x-old). An empty string for no answer. tika-server /language converts these codes to two-letter ISO 639-1 codes (see below).

getConfidence()

A coarse LanguageConfidence: HIGH, MEDIUM, LOW or NONE.

getRawScore()

The detector’s raw probability-like score for the language, between 0 and 1.

getConfidenceScore()

A normalised score, derived from how far ahead of the runner-up this language is.

isReasonablyCertain()

A convenience that is true for the confidence levels the library considers reliable. In the runs here it was true for HIGH and false for LOW.

isUnknown()

true for the "no language" result (for example from empty text).

isLanguage(String)

Compares with a code, using LanguageNames to treat macro-languages and variants as equal where appropriate.

Two facts about the numbers. getRawScore() and getConfidence() are not the same quantity: in the French run below the raw score is 0.998 and the confidence score is 0.886, while for the very short text Hola, buenos días the raw score is 0.517 and the confidence is LOW. And scores are comparable only within one detector: the OpenNLP detector reports much smaller raw scores for the same texts (see CharSoup versus OpenNLP), so never apply one detector’s threshold to the other.

A first detection

import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Locale;
import org.apache.tika.language.detect.LanguageDetector;
import org.apache.tika.language.detect.LanguageResult;

public class DetectText {

    public static void main(String[] args) throws IOException {
        LanguageDetector detector = LanguageDetector.getDefaultLanguageDetector().loadModels();
        System.out.println("detector: " + detector.getClass().getName());

        String french = Files.readString(Path.of("reseñas-fr.txt"), StandardCharsets.UTF_8);
        String english = Files.readString(Path.of("review-en.txt"), StandardCharsets.UTF_8);

        for (String text : new String[] {french, english, "Hola, buenos días", "ok"}) {
            LanguageResult result = detector.detect(text);
            System.out.printf(Locale.ROOT, "%-6s %-7s raw=%.3f score=%.3f certain=%s%n",
                    result.getLanguage(), result.getConfidence(), result.getRawScore(),
                    result.getConfidenceScore(), result.isReasonablyCertain());
        }
        for (LanguageResult r : detector.detectAll(french).subList(0, 3)) {
            System.out.println("candidate " + r);
        }
    }
}

Output (Tika 4.1.0):

detector: org.apache.tika.langdetect.charsoup.CharSoupLanguageDetector
fra    HIGH    raw=0.998 score=0.886 certain=true
eng    HIGH    raw=0.995 score=0.664 certain=true
spa    LOW     raw=0.517 score=0.254 certain=false
diq    LOW     raw=0.779 score=0.184 certain=false
candidate fra: HIGH (0.997937)
candidate wln: NONE (0.002063)
candidate cat: NONE (0.001633)

Notice what the last two lines of the first block say. The four-line French review and the English review are detected with HIGH confidence. The three-word Spanish greeting is correct but LOW. And the two-letter text ok is detected as Zazaki (diq) with a raw of 0.779 that looks impressive and is meaningless: the raw score of a short text is not a reason to trust it. The decision belongs to getConfidence() or isReasonablyCertain(), not to the raw number. The candidate lines show detectAll: the top candidate, then the runners-up with tiny scores. The toString of a result is language: CONFIDENCE (rawScore).

Source: LanguageDetectorExample in tika-example and LanguageDetector, Javadoc 4.1.0.

Mixed-language text with detectAll

mixed.txt holds one English, one Spanish and one French sentence. detect returns a single answer, and detectAll shows how the evidence is spread.

LanguageDetector detector = new CharSoupLanguageDetector().loadModels();
String mixed = Files.readString(Path.of("mixed.txt"));

System.out.println("detect : " + detector.detect(mixed));
for (LanguageResult r : detector.detectAll(mixed).subList(0, 4)) {
    System.out.println("candidate " + r + " raw=" + r.getRawScore());
}

Output (Tika 4.1.0):

detect : fra: LOW (0.636762)
candidate fra: LOW (0.636762) raw=0.6367616
candidate eng: NONE (0.363238) raw=0.3632384
candidate spa: NONE (0.353880) raw=0.35388032
candidate glg: NONE (0.297766) raw=0.2977663

The text has three languages and the answer is "French, low confidence", with English and Spanish next in the list. That is a correct reading of a mixed text: the detector is telling you that it is not a single-language text. The fourth candidate, Galician, is the Spanish confusable showing up (Galician and Spanish share many n-grams). If your intake service must handle mixed documents, split the text into paragraphs, detect each, and store a language per chunk (see Language per chunk for mixed documents and RAG), instead of forcing one language on the whole file. setMixedLanguages(true) exists as a hint on the base class; its effect on CharSoup was not tested, and the safe approach for mixed text is to split it yourself.

Source: LanguageDetector, Javadoc 4.1.0.

Detecting from a parse: LanguageHandler and LanguageWriter

In the book (and for as long as ProfilingHandler existed) the idiom was "feed the language identifier from a SAX handler". The 4.x equivalents are LanguageHandler, a content handler, and LanguageWriter, a java.io.Writer. Both hand the characters straight to a LanguageDetector as they stream past, so no complete text is built in memory just for language detection.

LanguageHandler: language as a by-product of the parse

LanguageHandler extends WriteOutContentHandler. Construct it with no argument (it uses the default detector and loads its models), with a LanguageDetector you prepared, or with a LanguageWriter.

import java.io.IOException;
import java.nio.file.Path;
import org.apache.tika.exception.TikaException;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.language.detect.LanguageHandler;
import org.apache.tika.language.detect.LanguageResult;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.parser.AutoDetectParser;
import org.apache.tika.parser.ParseContext;
import org.xml.sax.SAXException;

public class Handler {

    public static void main(String[] args) throws IOException, SAXException, TikaException {
        AutoDetectParser parser = new AutoDetectParser();
        LanguageHandler handler = new LanguageHandler();
        Metadata metadata = new Metadata();
        try (TikaInputStream in = TikaInputStream.get(Path.of("reviews-es.html"))) {
            parser.parse(in, handler, metadata, new ParseContext());
        }
        LanguageResult result = handler.getLanguage();
        System.out.println("detector : " + handler.getDetector().getClass().getSimpleName());
        System.out.println("language : " + result.getLanguage() + " " + result.getConfidence());
        System.out.println("declared : " + metadata.get("Content-Language"));
    }
}

Output (Tika 4.1.0):

detector : CharSoupLanguageDetector
language : spa HIGH
declared : es

The handler consumed the characters of the page as the HTML parser produced them, and then getLanguage() returned the detected result. The declared line is the other source of language information: the HTML parser copied the lang="es" attribute from the <html> element into the metadata key Content-Language. Note that the declared value is the two-letter ISO 639-1 code (es) and the detected value is the three-letter ISO 639-3 code (spa); compare them through LanguageNames or a mapping table, not with equals.

Because the handler is the only handler passed to the parse, the extracted text is not kept; you get the language and the metadata but no text. If you need both the text and the language, either parse once into a text handler and detect on the string, or tee the content to two handlers. The metadata filter (next section) is the cleaner way to get both in one pass.

Source: LanguageHandler, Javadoc 4.1.0.

LanguageWriter: language from any stream of characters

LanguageWriter is the writer-shaped version. It is useful when the characters do not come from a Tika parse at all (a Reader over a database column, a message queue payload, a decoded stream).

import java.io.IOException;
import java.io.Reader;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import org.apache.tika.language.detect.LanguageDetector;
import org.apache.tika.language.detect.LanguageResult;
import org.apache.tika.language.detect.LanguageWriter;

public class StreamingDetect {

    public static void main(String[] args) throws IOException {
        LanguageDetector detector = LanguageDetector.getDefaultLanguageDetector().loadModels();
        try (LanguageWriter writer = new LanguageWriter(detector);
                Reader reader = Files.newBufferedReader(Path.of("reseñas-fr.txt"), StandardCharsets.UTF_8)) {
            char[] buffer = new char[64];
            int read;
            int chunks = 0;
            while ((read = reader.read(buffer)) != -1) {
                writer.write(buffer, 0, read);
                chunks++;
                if (detector.hasEnoughText()) {
                    break;
                }
            }
            LanguageResult result = writer.getLanguage();
            System.out.println("chunks=" + chunks + " enough=" + detector.hasEnoughText()
                    + " -> " + result.getLanguage() + " " + result.getConfidence());
        }
    }
}

Output (Tika 4.1.0):

chunks=6 enough=false -> fra HIGH

The loop stops early if hasEnoughText() becomes true. For CharSoup that threshold is 10,000 buffered characters (from the 4.1.x documentation), so this small file is read to the end (enough=false) and the answer is still HIGH: enough text for a confident answer is much less than the early-stop threshold. On a large document the early stop means the reader does not need to consume the whole file, which is the point of the incremental API. The detector also truncates its input at 100,000 characters regardless, so a gigabyte-long text does not cost a gigabyte of work.

Source: LanguageWriter, Javadoc 4.1.0.

CharSoup: the recommended detector

org.apache.tika.langdetect.charsoup.CharSoupLanguageDetector (module tika-langdetect-charsoup) is the detector the 4.1.x documentation recommends for new deployments. Everything below, apart from what is marked as observed, is from the 4.1.x page "Language Detection" and the CharSoup supported-languages page.

Property Value

Languages

204, ISO 639-3 codes (apart from cdo-x-rom, hak-x-rom and be-x-old). The full list is on the "CharSoup Supported Languages" page. Observed: CharSoupLanguageDetector.getSupportedLanguages() returned 204 entries.

Training data

Wikipedia dumps, supplemented by the MADLAD-400 corpus where Wikipedia coverage is thin.

Model

One model for all text lengths: langdetect-20260320.bin inside tika-langdetect-charsoup-core, about 6.4 MB, 32,768 hash buckets, INT8 quantised weights.

Runtime dependencies

None beyond tika-core.

Preprocessing

Truncate to 100,000 characters, strip URLs and emails, NFC normalise, skip transparent characters, case fold.

Transparent characters

Unicode nonspacing marks (Arabic harakat, Hebrew niqqud), the Arabic tatweel, and the zero-width joiners, so that Arabic, Hebrew and Persian words are not broken into fragments.

Features

Character bigrams, trigrams and 4-grams with a word-position salt, short complete words, word bigrams, CJK and kana unigrams, per-script counts.

Confidence

sigmoid(top - second) of the raw logits after confusable groups are collapsed; the per-language raw score is sigmoid(logit - best_competitor_logit).

Early stop

hasEnoughText() is true at 10,000 buffered characters.

Friendly name

charsoup-language-detector (a @TikaComponent, so it resolves by name in the JSON configuration).

Tuning knobs

setMaxLength(int) (rolling buffer cap). setPriors(Map) is documented but threw UnsupportedOperationException in 4.1.0 (see the warning above). Also getDistributionEntropy(), compareLanguageSignal(Map) and the static topShortTextLanguages(String, int), whose uses are not documented.

Strengths and limits

CharSoup’s strengths are the breadth (204 languages from one 6.4 MB file), the absence of dependencies, the handling of right-to-left and joining scripts, and a confidence that means the same thing for short and long text. Its limits are those of every n-gram detector: very short strings are guesses (the Hola run above), closely related languages overlap (mitigated by the confusable groups, not removed), and the model knows exactly its 204 languages. A text in a language outside that set is assigned to the nearest known one, usually with a low confidence. The detector also does not detect dialects (Brazilian versus European Portuguese is the same por).

The model is trained from Wikipedia and MADLAD-400, which means formal and web register is better covered than slang, chat abbreviations or historical spelling. Test it on a sample of your own documents (the labelled sample in Evaluating on your own documents) before relying on it for anything that matters.

Restricting the candidate languages

If the Bookshelf catalogue only ever contains English, Spanish and French, you can tell the detector to consider only those, which both speeds up the scoring and removes a whole class of absurd answers (Tetum for Hola).

LanguageDetector restricted = new CharSoupLanguageDetector()
        .loadModels(Set.of("eng", "spa", "fra"));

System.out.println(restricted.detect("Hola"));

This fragment was not run for this page. Whether and how loadModels(Set) restricts the candidates for the single-matrix CharSoup model (as opposed to detectors with a model per language) is not described in the 4.1.x documentation, so test the behaviour on your own samples before depending on it, and prefer the explicit post-filter shown in Routing by language (accept only a known set of languages with sufficient confidence), which behaves the same way with every detector.

CharSoup versus OpenNLP

org.apache.tika.langdetect.opennlp.OpenNLPDetector (module tika-langdetect-opennlp) wraps the language-detection model of Apache OpenNLP. The model inside the 4.1.0 jar is dated 2021-04-13 (opennlp-langdetect-20210413.bin). The CharSoup model is dated 2026-03-20. The observed comparison on the Bookshelf samples (Tika 4.1.0, both detectors in the same JVM, one warm-up not performed so the times are only indicative):

List<LanguageDetector> detectors = List.of(
        new CharSoupLanguageDetector().loadModels(),
        new OpenNLPDetector().loadModels());

for (String file : List.of("reviews-es.html", "reseñas-fr.txt", "review-en.txt", "mixed.txt")) {
    String text = Files.readString(Path.of(file));
    if (file.endsWith(".html")) {
        text = text.replaceAll("<[^>]+>", " ");
    }
    for (LanguageDetector d : detectors) {
        LanguageResult r = d.detect(text);
        System.out.printf(Locale.ROOT, "%-16s %-24s %s %-6s raw=%.3f%n",
                file, d.getClass().getSimpleName(), r.getLanguage(), r.getConfidence(), r.getRawScore());
    }
}

Output (Tika 4.1.0; the time column of the original run is omitted):

reviews-es.html  CharSoupLanguageDetector spa HIGH   raw=0.961
reviews-es.html  OpenNLPDetector          spa LOW    raw=0.267
reseñas-fr.txt   CharSoupLanguageDetector fra HIGH   raw=0.998
reseñas-fr.txt   OpenNLPDetector          fra NONE   raw=0.179
review-en.txt    CharSoupLanguageDetector eng HIGH   raw=0.995
review-en.txt    OpenNLPDetector          eng NONE   raw=0.047
mixed.txt        CharSoupLanguageDetector fra LOW    raw=0.637
mixed.txt        OpenNLPDetector          spa NONE   raw=0.079

And on the short strings:

Hola                     CharSoupLanguageDetector tet LOW
Hola                     OpenNLPDetector          mlg NONE
Bonjour tout le monde    CharSoupLanguageDetector fra MEDIUM
Bonjour tout le monde    OpenNLPDetector          mlg NONE
Gracias por el libro     CharSoupLanguageDetector spa HIGH
Gracias por el libro     OpenNLPDetector          spa NONE
Aspect CharSoup OpenNLP

Module

tika-langdetect-charsoup

tika-langdetect-opennlp

Languages

204 (observed via getSupportedLanguages())

148 (observed: the array returned by getSupportedLanguages())

Model date

2026-03-20

2021-04-13

Dependencies

None beyond tika-core

Apache OpenNLP

Position in service discovery

First (the default)

Second

Confidence on these samples

HIGH for every clear single-language text

Raw scores 0.05 to 0.27 and NONE or LOW even when the language is right

Short text

Wrong more often than it looks but flagged LOW or MEDIUM

Hola and Bonjour tout le monde as Malagasy (mlg), flagged NONE

Tuning

setMaxLength, loadModels(Set)

setMaxLength, loadModels(Set)

Status in the 4.1.x docs

Recommended

Documented as the alternative

Both detectors returned the correct language on every clear sample. The difference is in how sure each one is. The OpenNLP detector’s scores are low even on correct answers, which means that, with the confidence mapping used by Tika, you cannot use isReasonablyCertain() as a gate with it on texts of this size: everything would be rejected. That is a property of how the two models and Tika’s score mapping behave on these samples, not a statement about the quality of OpenNLP in general. The practical conclusions are: use CharSoup unless you have a reason not to; never mix their scores; and, if you run both as a second opinion, compare the languages they return, not their confidences.

Source: OpenNLPDetector, Javadoc 4.1.0 and CharSoupLanguageDetector, Javadoc 4.1.0.

The other detector modules

The 4.1.x language-detection page documents two implementations. The earlier detector modules are not in that page, but they were still being published at the time of writing: Maven Central lists tika-langdetect-optimaize, tika-langdetect-lingo24 and tika-langdetect-mitll-text at version 4.1.0 (the maven-metadata.xml of each says the release is 4.1.0). The Optimaize detector (a Java port of a well-known n-gram detector) was the usual choice in the 2.x and 3.x lines. None of these three modules was loaded or tested for this page, and the 4.x documentation does not describe them, so treat them as legacy: they are not part of the 4.x recommendation, which is CharSoup (default) or OpenNLP. If you are upgrading from a 3.x deployment that selected Optimaize, plan to move to CharSoup and to re-validate thresholds, because the confidence numbers are not comparable.

Language in the metadata pipeline: charsoup-metadata-filter

The LanguageHandler approach has a limitation: it takes over the content handler. The 4.x answer is a metadata filter: a component that runs after the parse and reads the extracted content that is already in the metadata. The filter detects the language of the text and writes the result back as metadata keys, so the content is still available.

The tika-langdetect-charsoup module provides CharSoupMetadataFilter under the friendly name charsoup-metadata-filter, and the OpenNLP module provides OpenNLPMetadataFilter as open-nlp-metadata-filter. Metadata filters are configured in tika-config.json under the top-level metadata-filters array and applied through TikaLoader. The configuration below keeps only the keys the Bookshelf indexer needs, plus the three language keys.

{
  "metadata-filters": [
    "charsoup-metadata-filter",
    {
      "include-field-metadata-filter": {
        "include": [
          "tk:content",
          "Content-Type",
          "dc:title",
          "tk:detected-language",
          "tk:detected-language-confidence",
          "tk:detected-language-confidence-raw"
        ]
      }
    }
  ]
}

The filter has to run before the include filter, because the include filter drops everything it does not list, and the language keys do not exist until the language filter has run. The order of the array is the order of execution.

import java.nio.file.Path;
import java.util.List;
import org.apache.tika.config.loader.TikaLoader;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.parser.RecursiveParserWrapper;
import org.apache.tika.sax.BasicContentHandlerFactory;
import org.apache.tika.sax.RecursiveParserWrapperHandler;

public class FilterDemo {

    public static void main(String[] args) throws Exception {
        TikaLoader loader = TikaLoader.load(Path.of("tika-config.json"));
        RecursiveParserWrapper wrapper = new RecursiveParserWrapper(loader.loadAutoDetectParser());
        RecursiveParserWrapperHandler handler = new RecursiveParserWrapperHandler(
                new BasicContentHandlerFactory(BasicContentHandlerFactory.HANDLER_TYPE.TEXT, -1));
        try (TikaInputStream in = TikaInputStream.get(Path.of("reviews-es.html"))) {
            wrapper.parse(in, handler, new Metadata(), new ParseContext());
        }
        List<Metadata> list = handler.getMetadataList();
        loader.loadMetadataFilters().filter(list);
        for (Metadata m : list) {
            for (String name : m.names()) {
                String v = m.get(name);
                System.out.println(name + " = " + (v.length() > 60 ? v.substring(0, 60).replace('\n', ' ') + "..." : v));
            }
        }
    }
}

Output (Tika 4.1.0):

tk:content =             Reseñas de lectores   Reseñas de lectores  El gu...
dc:title = Reseñas de lectores
tk:detected-language = spa
tk:detected-language-confidence-raw = 0.9610374569892883
Content-Type = text/html; charset=UTF-8
tk:detected-language-confidence = HIGH

The key names are verified: they are the constants TikaCoreProperties.TIKA_DETECTED_LANGUAGE, TIKA_DETECTED_LANGUAGE_CONFIDENCE and TIKA_DETECTED_LANGUAGE_CONFIDENCE_RAW, written in the 4.x tk: namespace. The 4.x migration material says metadata keys moved from the X-TIKA: prefix to tk:, so a 3.x deployment will have stored these as X-TIKA:detected-language and so on (3.x was not run for this page to confirm the exact 3.x key names); the LegacyKeyMigrationFilter mentioned in Metadata can restore the 3.x names for downstream consumers that were not updated. The key tk:detected-language-confidence is the coarse HIGH/MEDIUM/LOW/NONE, and tk:detected-language-confidence-raw is the raw score. The filter processes the extracted content, and the filter class contains the message "no content to detect language from; filter is a no-op for this document", so a document without text simply gets no language keys rather than a made-up one.

Because the filter works on the content of each Metadata in the list, it applies to every embedded document too when you use the recursive wrapper. In the publisher-email.eml and samples.zip walkthrough of Embedded Documents & Recursive Parsing, each embedded file therefore gets its own language, which is what you want for a zip that contains a French review, a Spanish review and an English manuscript.

Source: Apache Tika 4.1.x, Metadata filters, CharSoupMetadataFilter, Javadoc 4.1.0 and Apache Tika 4.1.x, Metadata changes in 4.x.

Declared versus detected language

The Handler example already showed both sources: Content-Language = es from the HTML and spa from the detector. The metadata the parsers produce for the declared side varies by format (an HTML lang attribute or Content-Language meta tag, the dc:language of an EPUB, a Word language tag, a PDF /Lang), so the intake service should store both in separate fields and compare them:

Situation What to do

Declared and detected agree

Store the detected value with its confidence; high trust.

Declared is missing

Use the detected value if its confidence is HIGH; mark the source as "detected".

Declared and detected disagree, detection is HIGH

Trust the detection, log the disagreement (templates left in English around Spanish text are common). Keep the declared value for audit.

Declared and detected disagree, detection is LOW or NONE

Keep the declared value if present, flag for review. A short text is not evidence against a declaration.

Detection is NONE on a long text

Suspect the extraction (an encoding error, an OCR failure, a scanned page with no text layer), not the language.

The comparison needs a normalisation step because the vocabularies differ: es against spa, fr against fra. The LanguageNames class in tika-core (normalizeName, equals, isMacroLanguage, getMacroLanguage) provides the helpers Tika itself uses; a small explicit map for the handful of languages your service accepts is also perfectly reasonable.

Routing by language

The decision box of the pipeline figure, in code: accept a language only when the detector is confident and the language is one the service handles; otherwise fall back to a neutral treatment.

import java.util.Locale;
import java.util.Map;
import org.apache.tika.language.detect.LanguageDetector;
import org.apache.tika.language.detect.LanguageResult;

public class Route {

    private static final Map<String, String> ANALYZERS =
            Map.of("eng", "english", "spa", "spanish", "fra", "french");

    public static String analyzerFor(LanguageDetector detector, String text) {
        LanguageResult result = detector.detect(text);
        if (result.isReasonablyCertain() && ANALYZERS.containsKey(result.getLanguage())) {
            return ANALYZERS.get(result.getLanguage());
        }
        return "standard";
    }

    public static void main(String[] args) throws Exception {
        LanguageDetector detector = LanguageDetector.getDefaultLanguageDetector().loadModels();
        for (String s : new String[] {
                "El guardián del faro es una novela pausada y luminosa.",
                "Le gardien du phare est un roman lent et lumineux.",
                "The Lighthouse Keeper is a slow, luminous novel.",
                "Hola", "ISBN 978-84-376-0494-7"}) {
            LanguageResult r = detector.detect(s);
            System.out.printf(Locale.ROOT, "%-58s %-4s %-6s -> %s%n",
                    s, r.getLanguage(), r.getConfidence(), analyzerFor(detector, s));
        }
    }
}

Output (Tika 4.1.0):

El guardián del faro es una novela pausada y luminosa.     spa  LOW    -> standard
Le gardien du phare est un roman lent et lumineux.         fra  HIGH   -> french
The Lighthouse Keeper is a slow, luminous novel.           eng  HIGH   -> english
Hola                                                       tet  LOW    -> standard
ISBN 978-84-376-0494-7                                     mlt  LOW    -> standard

The first row is the instructive one. A correct, single-sentence Spanish text comes back as spa with LOW confidence, so the policy sends it to the neutral standard analyzer even though the answer was right. That is the cost of a conservative gate: a sentence of about nine words is not enough for HIGH, whereas the two-paragraph review in reviews-es.html is (spa HIGH). Two ways to reduce the loss are to detect on the whole document’s text rather than on a sentence, and to accept spa at LOW when the document’s declared language also says Spanish (the combined rule in Declared versus detected language). Do not lower the threshold globally: the last two rows show what comes through if you do.

Source: LanguageResult, Javadoc 4.1.0.

Language per chunk for mixed documents and RAG

When a document is mixed, or when the RAG pipeline chunks text anyway, detect the language per chunk and keep it as chunk metadata. The chunk is the unit that is embedded and retrieved, so it is the unit whose language matters. The same Route-style code applies to each chunk; the practical rules are:

  • Detect after chunking, not before, and skip chunks shorter than a threshold (a few hundred characters is a sensible start; below that, inherit the language of the neighbouring chunks or of the document).

  • Store the language and the confidence as chunk metadata so retrieval can filter (language == "spa") or boost.

  • Do not translate before chunking unless you have to: a translation pass is slow, costs money (see The bundled translators and their maintenance status) and loses the original wording.

The chunking and metadata design is in Document parsing for RAG and in document ingestion and chunking.

From the command line: tika-app -l

The 4.1.x command-line page lists the -l / --language option in its "Detection, Language and Content" group: "Detect and print only the language." The same group has -d / --detect, which prints only the media type. The text a language detection runs on is extracted by the same parser as any other tika-app run.

java -jar tika-app-4.1.0.jar -l reviews-es.html
java -jar tika-app-4.1.0.jar -l "reseñas-fr.txt"

Output, from the 4.1.0 zip distribution:

spa
fra

tika-app prints the bare code of the bundled CharSoup detector on one line, so the codes are the ISO 639-3 codes of LanguageResult.getLanguage(). The run, the zip layout and the in-flux status of the 4.x command line are on the Tika App (Command Line & GUI) page.

Source: Apache Tika 4.1.x, Command Line.

From the server: the /language endpoint

tika-server 4.1.x lists a /language endpoint (config name language) described as "Language detection". Two facts from the 4.1.x server page matter operationally.

  • It runs in the server’s own JVM, not in a fork. The parsing endpoints (and /detect and /unpack) run in forked JVMs through Tika Pipes, so a crashing or runaway parse kills only a fork. /language is outside that isolation: a crash or memory exhaustion in it takes the whole server down. The page says so in a warning, and adds that if you do not need /language you should omit it from the list of enabled endpoints.

  • It caps the work. Detection stops after the first 100,000 characters, and the server does not read more than that from the body into memory. The request size limit (maxRequestSizeBytes) still bounds what a client may send.

The endpoint takes the raw body (PUT or POST) and detects on it as text, without parsing it first:

curl -T reviews-es.html http://localhost:9998/language
curl -T reseñas-fr.txt http://localhost:9998/language
curl -X POST -d 'The catalogue lists forty titles.' http://localhost:9998/language

Output, from the 4.1.0 tika-server-standard zip:

es
fr
en

Unlike tika-app, the server answers with two-letter ISO 639-1 codes. Its LanguageResource runs the same default detector and then maps the three-letter code to ISO 639-1 "for backward compatibility with existing API consumers", keeping the three-letter code only for a language that has no two-letter one. The 3.x /language/stream and /language/string paths answer 404. Normalise the codes before you compare server answers with the tk:detected-language values of the metadata filter, which stay ISO 639-3. Prefer that filter anyway (language computed inside the pipes fork, as part of the same parse) because it keeps the isolation. The endpoint-by-endpoint reference is on the Tika Server (REST API) page.

Source: Apache Tika 4.1.x, Tika Server and the 4.1.0 LanguageResource source in tika-server on GitHub.

The removed 1.x LanguageIdentifier

Readers of Tika in Action (written for Tika 1.0) and of the 3.3.2 documentation’s examples.html page will find this idiom:

// Tika 1.x. Removed in 2.0; shown here only so you can recognise it.
LanguageIdentifier identifier = new LanguageIdentifier(text);
String language = identifier.getLanguage();

Source: Apache Tika 3.3.2, Examples (which still shows this snippet, although the class no longer exists in 3.x).

org.apache.tika.language.LanguageIdentifier, LanguageProfile and ProfilingHandler were removed in Tika 2.0 and replaced by the LanguageDetector SPI described above. Observed in this run: the 4.1.0 tika-core jar contains only org/apache/tika/language/detect/* and org/apache/tika/language/translate/* in the language package, so the 1.x classes are definitely gone. The mapping is mechanical:

Tika 1.x (the book, chapter 7) Tika 2.x, 3.x, 4.x Notes

new LanguageIdentifier(text).getLanguage()

LanguageDetector.getDefaultLanguageDetector().loadModels().detect(text).getLanguage()

Models are loaded explicitly. Codes become ISO 639-3 (eng) rather than the two-letter codes of the old profiles.

identifier.isReasonablyCertain()

result.isReasonablyCertain() on LanguageResult

Same name, different object; the criterion is the detector’s.

LanguageProfile (n-gram frequency table)

Internal to each detector’s model

You no longer build or load profiles yourself. To train a new CharSoup model see "Building the Language Detector" in the 4.1.x docs.

ProfilingHandler

LanguageHandler

Both are content handlers fed during the parse.

Profiles for about 18 languages in the book’s era

204 (CharSoup) or 148 (OpenNLP, observed)

The coverage is a different order of magnitude.

X-TIKA:language style keys

tk:detected-language in 4.x (X-TIKA:detected-language in 3.x)

See the metadata filter section above.

Source: Chris A. Mattmann and Jukka L. Zitting, Tika in Action (Manning, 2011), chapter 7 on language detection, written for Tika 1.0, and Apache Tika 4.1.x, Migrating to Tika 4.x.

The Translator API

Tika’s translation support is a small, pluggable abstraction: given text, a source language and a target language, a Translator returns the translated text, usually by calling an external service. It is deliberately not a translation engine. The API lives in tika-core and is intentionally tiny.

Member Meaning

String translate(String text, String sourceLanguage, String targetLanguage)

Translate from a known source language. Throws TikaException and IOException.

String translate(String text, String targetLanguage)

Translate, letting the service detect the source language.

boolean isAvailable()

Whether the translator is configured and usable (credentials present, tool installed, service reachable).

DefaultTranslator

The service-loader aggregate: it asks the available implementations in discovery order and uses the first one whose isAvailable() is true. Also getTranslators() and getTranslator().

EmptyTranslator

The do-nothing implementation.

Tika.translate(String, String) and related

The facade methods that call the facade’s translator (by default a DefaultTranslator).

The interface extends SelfConfiguring, the 4.x marker that lets a component be configured from the JSON configuration, and TikaLoader.loadTranslator() returns the configured translator (in 4.1.0 with no configuration it returned a DefaultTranslator). What a particular implementation reads (a properties file on the classpath and its keys, or setters) is listed in the next section.

Writing your own Translator

Because the contract is three methods, the practical way to use translation in 4.x is often to implement Translator for a service you already use, rather than to rely on the bundled ones. The toy implementation below replaces whole words from a glossary. It is not a translation engine: it exists to show the contract and the facade, and it runs offline.

import java.util.Map;
import org.apache.tika.Tika;
import org.apache.tika.detect.DefaultDetector;
import org.apache.tika.language.translate.Translator;
import org.apache.tika.parser.AutoDetectParser;

/** Toy translator: replaces whole words from a glossary. Shows the Translator contract only. */
public class GlossaryTranslator implements Translator {

    private final Map<String, String> glossary;

    public GlossaryTranslator(Map<String, String> glossary) {
        this.glossary = glossary;
    }

    @Override
    public String translate(String text, String sourceLanguage, String targetLanguage) {
        StringBuilder out = new StringBuilder();
        for (String word : text.split("(?<=\\s)|(?=\\s)")) {
            out.append(glossary.getOrDefault(word.toLowerCase(), word));
        }
        return out.toString();
    }

    @Override
    public String translate(String text, String targetLanguage) {
        return translate(text, "und", targetLanguage);
    }

    @Override
    public boolean isAvailable() {
        return true;
    }

    public static void main(String[] args) throws Exception {
        Translator t = new GlossaryTranslator(
                Map.of("le", "the", "gardien", "keeper", "du", "of the", "phare", "lighthouse"));
        System.out.println(t.translate("Le gardien du phare", "fr", "en"));
        Tika tika = new Tika(new DefaultDetector(), new AutoDetectParser(), t);
        System.out.println(tika.translate("le phare", "en"));
    }
}

Output (Tika 4.1.0):

the keeper of the lighthouse
the lighthouse

The second line shows that the Tika facade’s constructor accepts a translator and delegates to it. In a real implementation translate would call your translation service’s SDK, add authentication from the environment (never hard-code keys), set timeouts, and translate in batches that fit the service’s size limit. Register it as a service (META-INF/services/org.apache.tika.language.translate.Translator) only if you want DefaultTranslator to find it; otherwise pass it explicitly as above. The service-loading and @TikaComponent mechanics are covered in Configuration and Extending Tika.

Source: TranslatorExample in tika-example and Translator, Javadoc 4.1.0.

The bundled translators and their maintenance status

This is the part of the page that has to be blunt. The tika-translate module is published at 4.1.0 (Maven Central lists it, and it is in the tika repository), so it is not removed. But "published" is not "maintained", and the evidence points in the direction of "kept, not developed". The facts, each with how it was established:

Fact How it was established

tika-translate 4.1.0 exists and tika-core 4.1.0 still contains Translator, DefaultTranslator and EmptyTranslator

Maven Central metadata; the contents of the 4.1.0 jars (observed).

There is no translation page in the 4.1.x documentation

The 4.1.x navigation (Using Tika, Pipes, Configuration, Migration, Advanced, Developers, FAQ) has no translation entry. The only 4.x statement about translation is in the server migration page (next row).

tika-server’s /translate/* endpoints were removed in 4.x, together with the server’s dependency on tika-translate

The 4.1.x page "Tika Server Migration". Its stated reason: the endpoints "were never usable as shipped": a request had to name a translator implementation class in the URL, each bundled implementation wraps a commercial API that needs credentials, and the server had no way to supply them. The page’s advice: call the translation service directly, or invoke tika-translate from your own code, and that "Tika’s translation support is unaffected".

The older documentation that mentions translation is stale

The 3.3.2 examples.html page still shows a Microsoft Translation API example linked to an old MSDN address, and in the same page a LanguageIdentifier snippet whose class was removed in 2.0.

A default setup has no usable translator

Observed in 4.1.0: DefaultTranslator.isAvailable() returned false, getTranslator() returned null, every candidate reported available=false, and translate(...) threw TikaException: No translators currently available.

The module logs errors for unconfigured translators

Observed: JoshuaNetworkTranslator.isAvailable() logged an ERROR with a MalformedURLException stack trace (property joshua.server.url not set) each time availability was checked. Harmless, but noisy.

The bundled implementations depend on third-party services or old client libraries

See the table below. The parent POM pins microsoft-translator-java-api at 0.6.2.

The implementations registered by the 4.1.0 module’s service file, and what each one needs (read from the class constants, not from documentation, because there is none in 4.x):

Class What it calls Configuration it reads

GoogleTranslator

The HTTP endpoint https://www.googleapis.com/language/translate/v2

translator.google.properties, key read from translator.client-secret (a string constant in the class; no public setter was found); default placeholder dummy-secret.

MicrosoftTranslator

Microsoft translator client library 0.6.2

translator.microsoft.properties with translator.client-id and translator.client-secret; placeholders dummy-id and dummy-secret.

YandexTranslator

The endpoint https://translate.yandex.net/api/v1.5/tr.json/translate

translator.yandex.properties with translator.api-key; translator.text.format.

Lingo24Translator

The endpoint https://api.lingo24.com/mt/v1/translate

translator.lingo24.properties with translator.user-key.

MosesTranslator

A local Moses statistical machine translation installation (external process)

translator.moses.properties with translator.smt_path and translator.script_path.

MarianTranslator

A local Marian NMT installation (external process or server)

translator.marian.properties with translator.marian.path and several tuning keys.

JoshuaNetworkTranslator

A Joshua translation server over HTTP

translator.joshua.properties with joshua.server.url.

RTGTranslator

An RTG translation server over HTTP

translator.rtg.properties with rtg.base.url.

CachedTranslator

Wraps another translator and caches its answers

setTranslator(...); no external service of its own.

What this evidence supports, and what it does not:

  • Supported: the module compiles and is published with Tika 4.1.0; no translator is usable without extra configuration; the server no longer offers translation; the project’s own migration page tells you to call your translation service directly; and there is no 4.x narrative documentation for the module.

  • Not established: whether any of the four commercial endpoints above still accepts requests in the form this code sends. Some of them name API versions (Yandex "v1.5", Google "v2") and a vendor library from a period when these services used other authentication schemes, and at least some of those services have changed since. None of them was called for this page (that would need real accounts and credentials, which this page must not carry), so the page makes no claim that they work and no claim that they are broken. Moses, Marian, Joshua and RTG, which need their own installations, were not exercised either.

  • No statement from the project in the 4.1.x documentation says the module is deprecated or scheduled for removal. Calling it "unmaintained" would be an inference; the neutral statement is that nothing in 4.x documents, tests (for this page) or recommends it.

The recommendation for the Bookshelf intake service follows from this. Do not make a translation step through tika-translate a hard dependency of the pipeline. If you need translation:

  1. Keep language detection in Tika, which is current, and treat the translation step as a separate stage that takes (text, detected language).

  2. Call your translation provider’s own SDK or REST API directly, as the migration page says; if you want it behind Tika’s interface, implement Translator as shown above.

  3. Put credentials in environment variables or a secrets manager, set a timeout and a size limit per request, translate chunks rather than whole files, and cache by content hash (the CachedTranslator idea) so that re-ingesting a file does not pay twice.

  4. Record that a text was machine translated, and keep the original: a translation is a derived value, not the document.

  5. Mind privacy: a hosted translation service receives the text. Reviews may contain personal data, and manuscripts are unpublished commercial material; check the provider’s data-handling terms before sending them.

Source: Apache Tika 4.1.x, Tika Server Migration, tika-translate module on GitHub and Apache Tika 3.3.2, Examples.

Observing the state of translation in your own build

The check below tells you in a few lines what your classpath offers; run it before designing anything around the module.

DefaultTranslator translator = new DefaultTranslator();
System.out.println("available = " + translator.isAvailable());
for (Translator t : translator.getTranslators()) {
    System.out.println("  candidate " + t.getClass().getSimpleName() + " available=" + t.isAvailable());
}
try {
    System.out.println(translator.translate("Le gardien du phare est un roman lent et lumineux.", "fr", "en"));
} catch (Exception e) {
    System.out.println(e.getClass().getName() + ": " + e.getMessage());
}

Output (Tika 4.1.0, with no translator configured; the logged ERROR stack traces from JoshuaNetworkTranslator are omitted):

available = false
  candidate CachedTranslator available=false
  candidate GoogleTranslator available=false
  candidate JoshuaNetworkTranslator available=false
  candidate Lingo24Translator available=false
  candidate MarianTranslator available=false
  candidate MicrosoftTranslator available=false
  candidate MosesTranslator available=false
  candidate RTGTranslator available=false
  candidate YandexTranslator available=false
org.apache.tika.exception.TikaException: No translators currently available

Source: DefaultTranslator, Javadoc 4.1.0.

Evaluating on your own documents

The numbers in the 4.1.x documentation (204 languages, benchmark comparisons against other detectors) describe the model on its test corpus. Your documents are different: shorter, noisier, with domain vocabulary, boilerplate and extraction artefacts. Before you pick thresholds, label a sample of a few hundred of your own extracted texts with the true language (a native speaker or a trusted source), run the detector, and look at:

  • accuracy by length bucket (under 50 characters, 50 to 200, 200 to 1,000, over 1,000), since length is the main driver;

  • the precision of HIGH and of HIGH plus MEDIUM, which tells you where to put the gate;

  • the confusions (Spanish with Galician or Catalan, Portuguese with Spanish, Croatian with Serbian);

  • the share of NONE on non-trivial text, which is a symptom of extraction problems.

The project’s own tool for measuring extraction quality, including the language of the extracted text, is described in the tika-eval & Extraction Quality page. Detectors and thresholds are not something to set once: re-run the check when you change the extraction (an OCR engine, a parser upgrade) because that changes the text the detector sees.

Memory, speed and thread safety

  • Memory. The CharSoup model is a 6.4 MB file kept as one INT8 matrix, so a detector costs roughly that plus a rolling buffer, much smaller than a per-language model set. OpenNLP loads its own model. Create detectors once per thread and reuse them; do not call loadModels() per document.

  • Speed. In the comparison run, CharSoup took about 1 to 10 milliseconds per document in a cold JVM (the first call, including class loading, was the slowest) against 1 to 15 milliseconds for OpenNLP. These are single, unwarmed measurements on one laptop and only indicate order of magnitude. The 100,000-character truncation bounds the cost per document.

  • Thread safety. A LanguageDetector instance holds the accumulated text and is not safe to share between threads without external locking. Use a detector per worker, or a pool.

  • Inside a Pipes fork. The metadata filter runs in the process that applies filters; for the forked parsing of the server and PipesForkParser, the isolation and limits are covered in the Robustness, Limits & Timeouts page. The /language endpoint is the exception because it is not forked.

Pitfalls

  • Trusting getRawScore() on short text. The ok run above had a raw score of 0.779 for Zazaki. Gate on the confidence level.

  • Treating HIGH as a guarantee. It is a statistical statement about the text. A Spanish text full of French quotations can still be HIGH for Spanish and wrong for the quotation.

  • Comparing declared and detected codes as strings. es is not spa. Normalise first.

  • Forgetting reset(). Reusing a detector with addText and detect() without a reset mixes documents.

  • Running detection on boilerplate. Detect on the body text; strip navigation and footers where you can.

  • Mixing thresholds between detectors. CharSoup and OpenNLP scores are not comparable.

  • Depending on setPriors. It threw UnsupportedOperationException for CharSoup in 4.1.0.

  • Exposing /language to untrusted callers in a shared server. It is not isolated in a fork. Keep it off or behind an authenticated proxy, in line with the guidance in the Security page.

  • Building on the translator module without checking it. See the maintenance section above.

  • Assuming the 3.x documentation is current. The 3.3.2 examples page still shows a removed class.

What this page did not verify

  • The behaviour of setMixedLanguages, setShortText and loadModels(Set) on CharSoup was not exercised.

  • setPriors was exercised only on CharSoupLanguageDetector (it throws); OpenNLPDetector.setPriors was not.

  • The Optimaize, Lingo24 and MIT-LL language-detection modules and every translator implementation’s remote or local back end were not called. Their existence at 4.1.0 and the endpoints and property names were read from Maven Central metadata and the compiled classes.

  • The 4.x JSON selection of a detector by friendly name (charsoup-language-detector) in the parse-context section was not exercised; only the metadata filter was configured and run.

  • Timings are single unwarmed measurements.

Checklist

  • Add tika-langdetect-charsoup (and nothing else) to start; call loadModels() once per detector.

  • Detect on body text of a reasonable length; split mixed documents and detect per chunk.

  • Gate on getConfidence() or isReasonablyCertain(), never on the raw score, and keep an explicit allow-list of languages the service handles.

  • Prefer the charsoup-metadata-filter (tk:detected-language*) over LanguageHandler when you also need the text.

  • Store the declared and the detected language separately and normalise their codes before comparing.

  • Reset detectors between documents, and do not share one between threads.

  • Keep /language off public servers, or behind authentication.

  • Treat tika-translate as undocumented and untested in 4.x, and do not depend on it: call your translation service directly, or wrap it in your own Translator, and keep credentials in the environment.

  • Label a sample of your own documents and choose thresholds from it.

References