RAG & LLM Ingestion

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

Baseline date: 2026-10-13. This page was written against Apache Tika 4.1.0 (the latest release on Maven Central on that date) with the 3.3.x differences noted, Java 21 and later for the examples (Tika 4.x needs Java 17, 3.x needs Java 11), Spring Boot 4.x on Spring Framework 7 for the framework pointers and the Docker image apache/tika:4.1.0-1 (or apache/tika:4.1.0-1-full when OCR is needed). The framework readers discussed near the end were checked at Spring AI 2.0.1 (spring-ai-tika-document-reader) and LangChain4j 1.22.0 (langchain4j-document-parser-apache-tika 1.22.0-beta32), the latest releases on Maven Central on the same date.

A retrieval-augmented generation (RAG) system answers a question by first retrieving passages from your own documents and then asking a large language model (LLM) to answer from those passages. Everything it can retrieve was put there by an ingestion pipeline: fetch the files, extract the text, cut it into chunks, attach metadata, embed each chunk and store it. Tika covers the second step for more than a thousand formats, and in 4.x it can also do parts of the third and fifth. This page is about using it well in that position: what to ask Tika for, which of its outputs feed which RAG feature, and where its job ends. The Bookshelf intake service is the running example: publishers and readers send manuscripts, catalogues, scanned loan forms, book covers, e-mails with attachments and reviews in several languages, and the assistant built on top must answer questions about all of them, with citations.

Related pages: Content Handlers & Output Formats (the Markdown handler), Embedded Documents & Recursive Parsing (the RMETA list), Metadata (key names and filters), Language Detection & Translation, OCR with Tesseract, PDF Parsing, VLM Parsers & Inference (engines, bindings and tk:chunks in depth), Tika Pipes and Search Indexing with Lucene, Solr & Elasticsearch. The RAG side of the pipeline, which this page does not repeat, is covered by Document Parsing, Ingestion Pipelines & Freshness and Document Ingestion & Chunking.

How this page was checked: the Java program, the two Python clients, the tika-server configuration and the reader compatibility test were compiled and run against Tika 4.1.0 on Java 25 (the examples target Java 21), on small original Bookshelf sample files written for this section. The embedding calls went to a local mock of the OpenAI embeddings API on 127.0.0.1 written for this page, never to a paid service, so the vectors shown are canned four-number values; every credential was a dummy value read from an environment variable. Tesseract was not installed on the test machine, which is why the scanned page in the runs below has no OCR text. The Tika Pipes emitter configuration with chunkStrategy was read from the 4.1.x documentation and not run. What was verified and what was not lists the details.

Where Tika sits in a RAG pipeline

The diagram shows a Bookshelf ingestion pipeline. Tika owns the boxes in the middle; everything to the left (where files come from) and to the right (vector store, retriever, prompt, LLM) belongs to other components.

flowchart LR src["Sources
(upload inbox, S3, mail)"] --> fetch["Fetch and hash
(stable source id)"] fetch --> parse["Tika: detect and parse
recursive list, Markdown"] parse --> check{"Text layer usable?"} check -->|"no text, OCR flagged"| ocr["Text recognizer
(Tesseract or VLM)"] ocr --> parse check -->|"columns, complex tables"| layout["Layout model
(outside Tika)"] check -->|"yes"| meta["Metadata filters
(language, allow-list)"] layout --> chunk meta --> chunk["Chunk
(MarkdownChunker or yours)"] chunk --> embed["Embed
(Tika inference or your embedder)"] embed --> store[("Vector store
and keyword index")] store --> rag["Retriever, prompt, LLM"]

Each Tika output has a job in the RAG system. The rest of the page goes through them in this order:

Tika output RAG use Section

tk:content in Markdown

Chunk boundaries at headings; tables and lists stay readable to the LLM

Markdown output for chunking

The recursive metadata list (one object per embedded document)

Each attachment becomes its own source, with its own chunks and citations

Embedded documents as separate chunks

dc:title, dc:creator, dcterms:created, Content-Type, tk:embedded-resource-path

Metadata filters at query time, citation labels, deduplication

Metadata as filters

tk:detected-language (and per-chunk detection)

Filter or route by language; choose a multilingual embedding model

Language per chunk

pdf:ocr-page-count, pdf:chars-per-page, OCR or VLM text

Find and fix pages that would otherwise be silently empty

Scanned content: OCR and VLM text

<div class="page"> in XHTML output

Page numbers for citations

Page numbers for citations

tk:chunks (4.1 inference)

Chunks and vectors produced by Tika itself

Inference: letting Tika chunk and embed

What Tika does not give you is just as important, and What Tika does not recover, and when to escalate lists it.

Three ways to put Tika in front

Shape How the pipeline calls Tika Good for

Library, in process

RecursiveParserWrapper (or PipesForkParser for isolation) inside the ingestion service, Java only

A JVM ingestion service that wants the chunks before anything leaves the process. See Parser API & AutoDetectParser.

tika-server

PUT /rmeta/md per file from any language; the server forks its own parsing JVMs

Python or TypeScript RAG stacks, and any service that should not run parsers in its own heap. See tika-server REST API.

Tika Pipes

A pipes iterator lists the corpus, fetchers read it, emitters write to Elasticsearch, OpenSearch, Solr, Kafka or files

Bulk and scheduled re-ingestion of a whole corpus. See Tika Pipes.

The Tika project recommends running parsers out of process whenever the input is untrusted, and RAG corpora usually are (uploads, e-mail, web pages). The in-process example below is the shortest way to show the outputs; for production use the same logic behind PipesForkParser, tika-server or Pipes. tika-python is a third-party client (chrismattmann/tika-python), not an Apache release; the Python examples on this page use only the standard library against tika-server’s REST API.

Source: Apache Tika 4.1.x, Java API and Apache Tika 4.1.x, tika-server.

Markdown output for chunking

Since 4.0 the default content handler is Markdown in tika-app, tika-server and the Pipes CLI, and the reason the project gives is chunking. A chunker that splits at headings first and at blank lines second produces chunks that follow the document’s own sections; it can only do that if the headings survive extraction. In plain text they do not.

Where the Markdown structure comes from:

Source What survives as Markdown

DOCX, ODT, RTF

Headings from the paragraph styles (Heading 1 to #), lists, bold and italic, tables as pipe tables, links.

HTML, EPUB

h1 to h6, lists, tables, links. Navigation and footers survive too, unless you use main-content extraction (-T / boilerpipe).

PDF, born digital

Headings only when the PDF is tagged (marked content); the rest is paragraphs in the order PDFBox reads them. Table cells come out as separate lines or paragraphs, not as a table.

E-mail

The body; attachments are separate documents in the recursive list.

Images, scanned pages

Nothing, unless a text recognizer runs (see Scanned content: OCR and VLM text). A VLM recognizer answers in Markdown, so its tables survive.

The Bookshelf catalogue.pdf is a tagged PDF. Through tika-server’s Markdown endpoint (output from the 4.1.0 run, first page):

$ curl -s -T catalogue.pdf http://localhost:9998/tika/md
Bookshelf Autumn Catalogue

# Autumn Catalogue 2026

## New fiction

The Silent Shelf by A. Moreno: a librarian finds a book that rewrites itself.

Paper Lanterns by K. Ito: twelve stories set in one Kyoto bookshop.

Bookshelf - page 1

Check the price of Paper Lanterns before printing.

# Order form
...

Source: Apache Tika 4.1.x, tika-server (the /tika/md endpoint). Run on 4.1.0 against a local server on another port; output abridged.

Four details of this output matter for RAG, and none of them is an error:

  • The first line is the document title, taken from the metadata, before the first heading. A chunker that splits at headings turns it into a tiny chunk of its own (see the run in Embedded documents as separate chunks).

  • Running headers and footers stay in the text: Bookshelf - page 1 is a footer. In XHTML output Tika marks it as <div class="artifact"> when the PDF tags it, which Page numbers for citations uses to drop it; Markdown loses the marker.

  • Annotations are inline: the sticky note Check the price…​ appears where the page ends. Decide whether review comments belong in the index.

  • There are no page markers. Markdown has no page element, so a chunk cut from this text cannot say which page it came from. If citations need page numbers, read Page numbers for citations.

Embedded pictures leave a placeholder such as ![](embedded:figure1.png), and in the recursive list the container’s Markdown may also end with a heading naming the picture (# figure1.png, observed on the DOCX inside publisher-email.eml). Neither is useful to an embedding model; strip them, or replace them with a caption if a VLM produced one.

Keep the handler at MARKDOWN whenever Tika’s own chunker runs: the inference code logs a warning when tk:content-handler-type is anything else, and chunks only at blank lines. The handler choices, write limits and the /rmeta/md family of endpoints are on Content Handlers & Output Formats.

Source: Apache Tika 4.1.x, Content Handler Requirements for Inference and Apache Tika 4.1.x, FAQ.

Chunking: Tika’s MarkdownChunker or your own

Two routes exist, and both start from the same Markdown:

  • Tika chunks. org.apache.tika.inference.MarkdownChunker (module tika-inference, marked experimental) splits at headings, then at blank lines when a section is longer than maxChunkChars, then hard at maxChunkChars, and overlaps consecutive chunks by overlapChars. It is what a TEXT inference binding uses, and it is a plain class you can call yourself, as the program below does.

  • Your framework chunks. Hand the Markdown to LangChain’s MarkdownHeaderTextSplitter, a Spring AI TokenTextSplitter, LangChain4j’s DocumentSplitters or your own code. Token-based splitting, parent-child chunks, semantic chunking and contextual retrieval are not in Tika.

Choosing sizes, overlap and strategies is a RAG question, answered in Document Ingestion & Chunking. Two Tika-specific observations from the runs below:

  • maxChunkChars and overlapChars count characters, not tokens. A 1,500-character chunk is roughly 300 to 400 English tokens; check your embedding model’s limit in tokens.

  • The overlap is applied even when the previous chunk is shorter than overlapChars. With 200 characters of overlap a 20-character title chunk is followed by a chunk that starts at offset 0 again, so the title appears twice. Merge tiny chunks into their neighbour, or drop chunks below a minimum length, before embedding.

Source: Apache Tika 4.1.0 Javadoc, MarkdownChunker and the class source at tag 4.1.0 of apache/tika.

Embedded documents as separate chunks

A RAG corpus is full of containers: e-mails with attachments, ZIP files of samples, DOCX files with pictures, PDFs with attached errata. If the whole container becomes one text, a chunk can mix the e-mail’s greeting with page three of an attached contract, the citation can only name the e-mail, and the retriever cannot filter by attachment type. Parse recursively instead: RecursiveParserWrapper (Java), /rmeta (tika-server), -J (tika-app) or the RMETA parse mode (Pipes) return one metadata object per document in the tree, each with its own tk:content. Chunk each one separately and keep its tk:embedded-resource-path on every chunk.

The program below is the whole Tika side of a Bookshelf ingestion job, in process. For each file it computes a SHA-256 source id, parses recursively to Markdown, skips archive listings, reports documents without text, flags PDFs with pages that need OCR, chunks each document with MarkdownChunker, detects the language per chunk and writes one JSON line per chunk, ready for an embedding step.

<dependencies>
  <dependency>
    <groupId>org.apache.tika</groupId>
    <artifactId>tika-core</artifactId>
    <version>4.1.0</version>
  </dependency>
  <dependency>
    <groupId>org.apache.tika</groupId>
    <artifactId>tika-parsers-standard-package</artifactId>
    <version>4.1.0</version>
    <type>pom</type>
  </dependency>
  <dependency>
    <groupId>org.apache.tika</groupId>
    <artifactId>tika-langdetect-charsoup</artifactId>
    <version>4.1.0</version>
  </dependency>
  <dependency>
    <groupId>org.apache.tika</groupId>
    <artifactId>tika-inference</artifactId>
    <version>4.1.0</version>
  </dependency>
  <dependency>
    <groupId>org.apache.tika</groupId>
    <artifactId>tika-serialization</artifactId>
    <version>4.1.0</version>
  </dependency>
</dependencies>
dependencies {
    implementation("org.apache.tika:tika-core:4.1.0")
    implementation("org.apache.tika:tika-parsers-standard-package:4.1.0")
    implementation("org.apache.tika:tika-langdetect-charsoup:4.1.0")
    implementation("org.apache.tika:tika-inference:4.1.0")
    implementation("org.apache.tika:tika-serialization:4.1.0")
}

tika-serialization is listed because the program writes JSON with Jackson, which it brings in; any JSON library will do. Gradle resolves the pom-packaged parser package through its module metadata, so no <type> equivalent is needed; see Setup & Installation. Only the Maven build was run.

import java.io.BufferedWriter;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.MessageDigest;
import java.util.HexFormat;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

import com.fasterxml.jackson.databind.ObjectMapper;

import org.apache.tika.config.loader.TikaLoader;
import org.apache.tika.inference.Chunk;
import org.apache.tika.inference.MarkdownChunker;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.language.detect.LanguageDetector;
import org.apache.tika.language.detect.LanguageResult;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.metadata.TikaCoreProperties;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.parser.RecursiveParserWrapper;
import org.apache.tika.sax.BasicContentHandlerFactory;
import org.apache.tika.sax.RecursiveParserWrapperHandler;

/** Turns each Bookshelf file into chunk records (JSON lines) ready for an embedding step. */
public class BookshelfRagIngest {

    private static final int MIN_CHARS_FOR_LANGUAGE = 200;
    private static final List<String> ARCHIVES = List.of("application/zip", "application/x-tar",
            "application/x-7z-compressed");

    public static void main(String[] args) throws Exception {
        TikaLoader loader = TikaLoader.loadDefault();
        RecursiveParserWrapper parser = new RecursiveParserWrapper(loader.loadAutoDetectParser());
        MarkdownChunker chunker = new MarkdownChunker(1500, 200);
        LanguageDetector detector = LanguageDetector.getDefaultLanguageDetector().loadModels();
        ObjectMapper json = new ObjectMapper();

        try (BufferedWriter out = Files.newBufferedWriter(Path.of("chunks.jsonl"), StandardCharsets.UTF_8)) {
            for (String arg : args) {
                Path file = Path.of(arg);
                String sourceId = sha256(file);                     // stable id: same bytes, same id
                RecursiveParserWrapperHandler handler = new RecursiveParserWrapperHandler(
                        new BasicContentHandlerFactory(BasicContentHandlerFactory.HANDLER_TYPE.MARKDOWN, -1));
                Metadata metadata = new Metadata();
                try (TikaInputStream tis = TikaInputStream.get(file, metadata)) {     // records tk:resource-name
                    parser.parse(tis, handler, metadata, new ParseContext());
                }
                int written = 0;
                for (Metadata doc : handler.getMetadataList()) {      // container first, then each embedded document
                    String path = doc.get("tk:embedded-resource-path");
                    String type = doc.get("Content-Type");
                    String content = doc.get("tk:content");
                    if (ARCHIVES.stream().anyMatch(type::startsWith)) {
                        continue;                                     // an archive's text is only its entry names
                    }
                    if (content == null || content.isBlank()) {
                        System.err.printf("%s %s: no text (%s)%n", file.getFileName(),
                                path == null ? "/" : path, type);
                        continue;                                     // a scanned page or a picture: needs OCR
                    }
                    String ocrPages = doc.get("pdf:ocr-page-count");
                    if (ocrPages != null && !"0".equals(ocrPages)) {
                        System.err.printf("%s %s: %s page(s) flagged for OCR%n", file.getFileName(),
                                path == null ? "/" : path, ocrPages);  // check the recognizer actually ran
                    }
                    String docLanguage = language(detector, content);
                    List<Chunk> chunks = chunker.chunk(content);
                    for (int i = 0; i < chunks.size(); i++) {
                        Chunk chunk = chunks.get(i);
                        String chunkLanguage = chunk.getText().length() >= MIN_CHARS_FOR_LANGUAGE
                                ? language(detector, chunk.getText()) : docLanguage;
                        Map<String, Object> record = new LinkedHashMap<>();
                        record.put("id", sourceId.substring(0, 16) + ":" + (path == null ? "/" : path) + "#" + i);
                        record.put("source_sha256", sourceId);
                        record.put("file", file.getFileName().toString());
                        record.put("embedded_path", path == null ? "/" : path);
                        record.put("depth", doc.getInt(TikaCoreProperties.EMBEDDED_DEPTH));
                        record.put("content_type", type);
                        record.put("title", doc.get("dc:title"));
                        record.put("language", chunkLanguage);
                        record.put("start", chunk.getStartOffset());
                        record.put("end", chunk.getEndOffset());
                        record.put("text", chunk.getText());
                        out.write(json.writeValueAsString(record));
                        out.newLine();
                        written++;
                    }
                }
                System.err.printf("%s: %d documents, %d chunks%n", file.getFileName(),
                        handler.getMetadataList().size(), written);
            }
        }
    }

    private static String language(LanguageDetector detector, String text) {
        LanguageResult result = detector.detect(text);
        return result.isReasonablyCertain() ? result.getLanguage() : "und";
    }

    private static String sha256(Path file) throws Exception {
        MessageDigest digest = MessageDigest.getInstance("SHA-256");
        return HexFormat.of().formatHex(digest.digest(Files.readAllBytes(file)));
    }
}

Run on the sample files (log lines from Tika itself removed):

$ java -cp target/classes:$(cat cp.txt) BookshelfRagIngest manuscript.docx catalogue.pdf publisher-email.eml \
      reviews-es.html reseñas-fr.txt scanned-loan-form.pdf book-cover.jpg
manuscript.docx: 1 documents, 1 chunks
catalogue.pdf: 1 documents, 1 chunks
publisher-email.eml /samples.zip/chapter-one.docx/figure1.png: no text (image/png)
publisher-email.eml: 6 documents, 5 chunks
reviews-es.html: 1 documents, 2 chunks
reseñas-fr.txt: 1 documents, 1 chunks
scanned-loan-form.pdf /: 1 page(s) flagged for OCR
scanned-loan-form.pdf: 1 documents, 1 chunks
book-cover.jpg /: no text (image/jpeg)
book-cover.jpg: 1 documents, 0 chunks

$ head -n 2 chunks.jsonl     # wrapped here
{"id":"a8438d3d7ac90c0c:/#0","source_sha256":"a8438d3d7ac90c0c7901f2c88fee4e30dc30412803f81327bfe1019f767d6f48",
 "file":"manuscript.docx","embedded_path":"/","depth":0,
 "content_type":"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
 "title":"The Lighthouse Keeper","language":"eng","start":0,"end":75,
 "text":"The Lighthouse Keeper  \n\nThe Lighthouse Keeper\n\nChapter one begins at dusk."}
{"id":"775dabd42ebe9c7c:/#0","source_sha256":"775dabd42ebe9c7c1ea7cc6c3164a41fcafc51d3541db835f34d1e9a21135385",
 "file":"catalogue.pdf","embedded_path":"/","depth":0,"content_type":"application/pdf",
 "title":"Spring catalogue","language":"eng","start":0,"end":88,
 "text":"Spring catalogue  \n\nBookshelf spring catalogue Forty titles from independent publishers."}

Source: Apache Tika 4.1.0 Javadoc, RecursiveParserWrapper, MarkdownChunker, Apache Tika 4.1.x, Language Detection and the RecursiveParserWrapper example in tika-example. Compiled and run on Tika 4.1.0. The catalogue.pdf here is the one-page stand-in used by the metadata pages; the two-page tagged catalogue appears in Markdown output for chunking and Page numbers for citations.

What the run showed:

  • publisher-email.eml became four sources (five chunks), not one. The e-mail body, chapter-one.docx, catalogue.pdf and cover-note.txt from the attached samples.zip each got their own chunks and their own embedded_path, such as /samples.zip/chapter-one.docx. The ZIP itself was skipped: its Markdown is only the list of entry names (# chapter-one.docx, # catalogue.pdf, …​), which would otherwise be embedded as three meaningless chunks.

  • Pictures have no text. The PNG inside the DOCX and book-cover.jpg produced no chunks. Without a text recognizer they are invisible to a text-only RAG system; see Scanned content: OCR and VLM text.

  • A scanned page can hide behind a text layer. scanned-loan-form.pdf produced one chunk, from the typed cover note on its second page, and pdf:ocr-page-count said one page needed OCR. Without that check the loan form looks ingested while its scanned page is missing.

  • Short sections produce tiny or repeated chunks. reviews-es.html gave a 19-character chunk (Reseñas de lectores, the title line) and then a chunk from offset 0 to 419 that repeats it, because of the overlap rule described in Chunking: Tika’s MarkdownChunker or your own.

  • Writing a reserved key throws. An earlier version of the program called metadata.set("tk:resource-name", ...) and failed with IllegalArgumentException: Writing reserved key 'tk:resource-name' via the String API is not allowed. TikaInputStream.get(Path, Metadata) already records the file name; see Metadata for the write guard.

The chunk id combines the source hash, the embedded path and the chunk number, so re-ingesting an unchanged file produces the same ids (an idempotent upsert) and a changed file produces new ones. Two attachments with the same name at the same path in one container would collide; add tk:embedded-id (or the embedded id path) if your corpus has them.

For a large tree, keep memory in check: the recursive list holds every document’s text at once. The streaming handler and the embedded limits (maxDepth, maxCount) are on Embedded Documents & Recursive Parsing.

Metadata as filters

Retrieval quality depends as much on filtering as on vectors: only Spanish reviews, only catalogues from 2026, only documents attached to this e-mail, only what this user may see. Tika supplies most of the descriptive fields; the access-control fields come from your system, never from the document.

Tika key Chunk field Use in RAG

tk:embedded-resource-path

embedded_path

Citation label (catalogue.pdf in samples.zip in the e-mail); filter or group by attachment.

tk:embedded-depth

depth

Down-rank deep attachments, or exclude them for some questions.

Content-Type

content_type

Filter by format (application/pdf), route to a format-specific prompt. Strip the charset parameter first.

dc:title

title

Citation label; prepend to the chunk text before embedding so short chunks carry their context.

dc:creator, dcterms:created, dcterms:modified

author, created

Filter by author or date. Treat as hints: they come from the document and can be wrong or forged.

tk:detected-language, tk:detected-language-confidence

language

Filter or route by language (next section).

tk:resource-name

file

Display only. For e-mail the parser may replace it with the subject plus .eml.

pdf:ocr-page-count, tk:exception:write-limit-reached, tk:exception:*

quality flags

Keep partial or failed extractions out of the index, or mark them.

tk:parsed-by

parser

Provenance; re-parse a subset when a parser is upgraded.

Two rules keep this manageable. First, emit only the keys you index: a DOCX alone carries dozens of keys, and every unused one costs bandwidth and storage. Second, rename at the edge, so the index schema does not use Tika’s names. Both are metadata filters in tika-config.json, and they apply in tika-app, inside tika-server’s forked JVMs and at the Pipes emitter. The configuration used for the tika-server run in Inference: letting Tika chunk and embed keeps eleven keys:

{
  "metadata-filters": [
    "charsoup-metadata-filter",
    {
      "include-field-metadata-filter": {
        "include": ["tk:chunks", "tk:embedded-resource-path", "tk:embedded-depth", "Content-Type",
                    "dc:title", "dc:creator", "dcterms:created", "tk:detected-language",
                    "tk:detected-language-confidence", "pdf:ocr-page-count", "tk:parsed-by"]
      }
    }
  ]
}

Source: Apache Tika 4.1.x, Metadata Filters. Run on 4.1.0 as part of the server configuration below. Tika added Content-Length back to the response of the root document, so the allow-list governs the parse output, not every byte of the HTTP response.

The order is literal: the language filter runs before the allow-list, or the allow-list removes the text it needs. The include list deliberately leaves out tk:content: when Tika’s inference has already chunked the text, each chunk carries its own text, and the full text would only double the payload. Keep tk:content if you chunk outside Tika. To rename keys, add a field-name-mapping-filter last; its syntax is on Metadata.

At query time the vector store applies these fields as filters. The filter syntax of each store is on Metadata Filtering, and per-user access filters are on Access Control & Privacy.

Language per chunk

Language matters to a RAG system in three places: the embedding model (a monolingual model embeds other languages poorly), the filter (answer from Spanish reviews), and the prompt (answer in the language of the question). Tika detects it with CharSoup, the recommended detector in 4.x:

  • Per document, with the charsoup-metadata-filter in the filter chain. It reads tk:content and writes tk:detected-language, tk:detected-language-confidence and tk:detected-language-confidence-raw. In the server run below every document with text in publisher-email.eml got eng with confidence HIGH, except the ZIP, whose listing of file names got LOW.

  • Per chunk, in code, as BookshelfRagIngest does: detect on each chunk of at least 200 characters and let shorter chunks inherit the document’s language. In the run reviews-es.html gave spa and reseñas-fr.txt gave fra.

Per chunk is the unit that matters, because the chunk is what is retrieved; a bilingual catalogue would otherwise label its French section as English. Short chunks are where detectors fail, which is why the program uses a threshold and reports und (undetermined) when isReasonablyCertain() is false. The detector, its confidence levels, its thread-safety (one detector per thread) and the rationale are on Language Detection & Translation. Translating before embedding is rarely worth it: a multilingual embedding model is cheaper and keeps the original wording for the citation.

Source: Apache Tika 4.1.x, Language Detection.

Scanned content: OCR and VLM text

A scanned page, a photographed form or a book cover has no text layer. Tika 4.x does not fail on them; it returns empty or partial text, and the RAG system silently never retrieves them. The fix has two parts: detect, then recognize.

Detect. For PDFs, Tika measures every page and records the result:

Key Meaning for ingestion

pdf:chars-per-page

One value per page. A 0 is a page with no text layer.

pdf:unmapped-unicode-chars-per-page

Glyphs PDFBox could not map to Unicode; a high count means garbled text that should be OCR’d too.

pdf:ocr-page-count

How many pages tripped the OCR verdict, whether or not an engine ran.

tk:content empty on an image/* document

A picture, inline or attached, that nothing transcribed.

For scanned-loan-form.pdf the 4.1.0 run reported pdf:chars-per-page = 0, 100 and pdf:ocr-page-count = 1: the first page is a scan, the second a typed note. With no recognizer installed the scan contributed nothing, and only the second page reached the index. That is the situation the flagged for OCR line in BookshelfRagIngest exists to catch. When a recognizer does run, the same key is still 1, so compare it with the text length or the recognizer’s own keys before deciding the page is missing.

Recognize. Tika 4.1 has one switch for this, the text-recognizers list in tika-config.json, and two kinds of engine:

Engine Strengths Costs

Tesseract (tesseract-ocr-parser, or in-process tess4j-parser)

Local, free, predictable; good on clean typed scans with the right language pack.

Plain text only (no tables), weak on handwriting and photographs; CPU heavy; needs the binary or the -full image.

VLM (openai-vlm-parser, claude-vlm-parser, gemini-vlm-parser)

Answers in Markdown, so tables and headings survive; reads photographs and covers; a local model on vLLM or Ollama keeps the data in house.

A model call per page or picture: latency, cost, rate limits, and for hosted models the document leaves your network.

A local Tesseract setup for the Bookshelf scans, with the PDF parser deciding per page:

{
  "engines": {
    "tesseract": { "tesseract-ocr-parser": { "language": "eng" } }
  },
  "text-recognizers": [ { "engine": "tesseract" } ]
}

A local VLM through an OpenAI-compatible server, with the key from the environment:

{
  "engines": {
    "local-vlm": { "openai-vlm-parser": {
      "baseUrl": "${env:BOOKSHELF_VLM_URL}",
      "model": "${env:BOOKSHELF_VLM_MODEL}",
      "apiKey": "${env:BOOKSHELF_VLM_KEY}" } }
  },
  "text-recognizers": [ { "engine": "local-vlm" } ]
}

Source: Apache Tika 4.1.x, Configuration (the text-recognizers list and ${env:...} interpolation) and Apache Tika 4.1.x, VLM Parsers. Neither configuration was run for this page; both are reduced from the configurations that were run on OCR with Tesseract and VLM Parsers & Inference, which cover the options, the per-page policy (pages.text = AUTO, OCR, EXTRACT), rendering resolution, cost and data residency.

Two RAG-specific cautions:

  • Mark recognized text. OCR and VLM text have errors a born-digital text does not. Keep a flag on the chunk (for example from pdf:ocr-page-count or the vlm:* keys the VLM parsers write) so the prompt or the UI can say from a scanned page.

  • Do not caption everything. Running a VLM on every inline picture of a large corpus is expensive and fills the index with descriptions of logos. Restrict it by media type, by size, or to the document types that need it.

Pictures can also be retrieved without any text, by embedding them (an IMAGES binding) or embedding whole rendered pages (PAGES), which is the visual-retrieval approach; see Inference: letting Tika chunk and embed and Multimodal RAG.

Page numbers for citations

A good RAG answer cites catalogue.pdf, page 2, not just catalogue.pdf. Tika’s Markdown has no page markers, and a TEXT chunk carries only character offsets (a text locator), so page numbers have to come from elsewhere. Three options:

Option How

XHTML output

Ask for /rmeta/xml (or the XML handler): every PDF page is a <div class="page">, and tagged headers and footers are <div class="artifact">. Split the text per page and chunk each page separately. You lose Markdown headings in exchange for page boundaries, unless you convert the HTML of each page yourself.

Page vectors

A PAGES inference binding embeds each rendered page and stores a paginated locator with the page number. Good for visual retrieval; the page is the unit, so the answer cites a page by construction.

Per-page parsing upstream

Split the PDF into pages before Tika (for example with PDFBox) and send each page as its own document. Simple, but embedded documents and document-level metadata are then repeated or lost.

The first option, as a small standard-library Python client of tika-server:

"""Page-numbered text for every PDF in a file, from tika-server's /rmeta/xml (stdlib only)."""
import json
import os
import sys
import urllib.request
from html.parser import HTMLParser

TIKA_URL = os.environ.get("TIKA_URL", "http://localhost:9998")
SKIP_CLASSES = {"artifact"}                     # running headers and footers of tagged PDFs


class Pages(HTMLParser):
    def __init__(self):
        super().__init__()
        self.pages = []
        self.stack = []                         # the class of every open <div>

    def handle_starttag(self, tag, attrs):
        if tag == "div":
            css = dict(attrs).get("class", "")
            self.stack.append(css)
            if css == "page":
                self.pages.append([])

    def handle_startendtag(self, tag, attrs):
        pass                                    # <div ... /> placeholders open nothing

    def handle_endtag(self, tag):
        if tag == "div" and self.stack:
            self.stack.pop()

    def handle_data(self, data):
        if "page" in self.stack and not SKIP_CLASSES & set(self.stack) and data.strip():
            self.pages[-1].append(data.strip())


def pages_per_document(path):
    with open(path, "rb") as f:
        request = urllib.request.Request(TIKA_URL + "/rmeta/xml", data=f.read(), method="PUT")
    with urllib.request.urlopen(request, timeout=120) as response:
        documents = json.load(response)
    for doc in documents:
        if doc.get("Content-Type", "").startswith("application/pdf"):
            parser = Pages()
            parser.feed(doc.get("tk:content", ""))
            yield doc.get("tk:embedded-resource-path", "/"), [" ".join(p) for p in parser.pages]


if __name__ == "__main__":
    for doc_path, pages in pages_per_document(sys.argv[1]):
        for number, text in enumerate(pages, start=1):
            print(doc_path, "page", number, "|", text[:72])
$ TIKA_URL=http://localhost:9998 python3 tika_pages.py catalogue.pdf
/ page 1 | Autumn Catalogue 2026 New fiction The Silent Shelf by A. Moreno: a libra
/ page 2 | Order form Non-fiction Ink and Iron: a history of the printing press, 48
/errata.pdf page 1 | Errata: The Silent Shelf, 2nd printing, page 112.

Source: Apache Tika 4.1.x, tika-server (the /rmeta endpoints and their content types) and Apache Tika 4.1.x, PDF Parser. Run against a local tika-server 4.1.0 on Python 3.9; the output above uses the two-page tagged catalogue.pdf from the PDF Parsing page, which carries an attached errata.pdf.

What the run showed:

  • The footers Bookshelf - page 1 and Bookshelf - page 2 were dropped (they were artifact divs); the annotation on page 1 was kept, because it sits in its own annotation div inside the page. Add annotation to SKIP_CLASSES to drop review notes too.

  • Through /tika/xml (not /rmeta/xml) the attached errata.pdf appeared as a third page of the catalogue, because the non-recursive endpoint inlines embedded documents. The recursive endpoint gives it its own page 1.

  • The PDF’s outline (bookmarks) and its AcroForm field values sit after the last page div, so a per-page split loses them. Index them separately if they matter; the form values of the catalogue held a member number, which an index should probably not contain anyway.

Store the page number with each chunk as metadata, and keep the character offsets too: together they let a UI highlight the passage, as described on Citations & Grounding.

Inference: letting Tika chunk and embed

Tika 4.x can do the chunking and the embedding itself: 4.0 added the first embedding filters, and 4.1 replaced them with engines and inference bindings. An engines entry names an OpenAI-shaped embeddings endpoint (openai-embedding-engine), and an inference binding says what to embed: TEXT (the tk:content of every document, cut by markdown-chunker), IMAGES, PAGES or MEDIA. The result lands in the tk:chunks metadata key of each document. The module is experimental: its classes, keys and the tk:chunks layout may change in a minor release. Everything about engines, bindings, vendor parameters, concurrency and the Java-side InferenceDispatcher is on VLM Parsers & Inference; this section shows the RAG view: a tika-server that returns chunks with vectors, and a client that turns them into vector-store records.

When to let Tika embed, and when not:

Let Tika embed Embed in your pipeline

One embeddings endpoint, OpenAI-shaped (or behind an OpenAI-shaped gateway such as LiteLLM)

A framework already manages models, batching and retries (Spring AI EmbeddingModel, LangChain4j EmbeddingModel)

Pages and pictures must be embedded too: Tika renders and batches them per document tree

You need token-based chunk sizes, parent-child chunks, contextual retrieval or late chunking

Pipes writes straight to Elasticsearch or OpenSearch

The same chunks go to several stores, or you re-embed often with new models without re-parsing

The tika-server configuration of the run: a server block, one embedding engine whose URL and key come from the environment, a TEXT binding, and the metadata filters from Metadata as filters.

{
  "server": { "host": "127.0.0.1", "port": 9998 },
  "engines": {
    "embedder": {
      "openai-embedding-engine": {
        "baseUrl": "${env:BOOKSHELF_EMBED_URL}",
        "model": "mock-embed",
        "apiKey": "${env:BOOKSHELF_EMBED_KEY}",
        "maxBatchSize": 16
      }
    }
  },
  "inference": [
    { "id": "text-vectors", "engine": "embedder", "input": "TEXT", "tasks": ["embed"],
      "chunker": { "markdown-chunker": { "maxChunkChars": 1500, "overlapChars": 200 } },
      "maxChunks": 500 }
  ],
  "metadata-filters": [
    "charsoup-metadata-filter",
    {
      "include-field-metadata-filter": {
        "include": ["tk:chunks", "tk:embedded-resource-path", "tk:embedded-depth", "Content-Type",
                    "dc:title", "dc:creator", "dcterms:created", "tk:detected-language",
                    "tk:detected-language-confidence", "pdf:ocr-page-count", "tk:parsed-by"]
      }
    }
  ]
}
$ export BOOKSHELF_EMBED_URL=http://127.0.0.1:8000   # a local embeddings server (a mock in this run)
$ export BOOKSHELF_EMBED_KEY=...                     # never in the file
$ java -jar tika-server-standard-4.1.0.jar -c tika-config.json

Source: Apache Tika 4.1.x, Inference: engines and bindings and Apache Tika 4.1.x, tika-server. Run on 4.1.0 (on another port) against the local mock. A tika-config.json without a server block failed at startup with TikaConfigException: Couldn't find 'server' element, so keep one even if it only sets the host and port. The model name is the mock’s; use your model’s.

The client: send a file to /rmeta/md, skip archive listings, parse tk:chunks and decode the vectors.

"""Send one file to tika-server, turn its tk:chunks into vector-store records (stdlib only)."""
import base64
import hashlib
import json
import os
import struct
import sys
import urllib.request

TIKA_URL = os.environ.get("TIKA_URL", "http://localhost:9998")
SKIP_TYPES = ("application/zip", "application/x-tar", "application/x-7z-compressed")


def decode_vector(b64):
    raw = base64.b64decode(b64)
    return list(struct.unpack(">%df" % (len(raw) // 4), raw))  # big-endian float32


def records(path):
    data = open(path, "rb").read()
    request = urllib.request.Request(TIKA_URL + "/rmeta/md", data=data, method="PUT",
                                     headers={"Content-Disposition": "attachment; filename=" + os.path.basename(path)})
    with urllib.request.urlopen(request, timeout=120) as response:
        documents = json.load(response)
    source = hashlib.sha256(data).hexdigest()
    for doc in documents:
        content_type = doc.get("Content-Type", "")
        if content_type.startswith(SKIP_TYPES):
            continue  # an archive's text is only its entry names
        chunks = json.loads(doc.get("tk:chunks", "[]"))  # one string holding a JSON array
        doc_path = doc.get("tk:embedded-resource-path", "/")
        for i, chunk in enumerate(chunks):
            span = chunk["locators"]["text"][0]
            yield {
                "id": "%s:%s#%d" % (source[:16], doc_path, i),
                "file": os.path.basename(path),
                "embedded_path": doc_path,
                "content_type": content_type,
                "title": doc.get("dc:title"),
                "language": doc.get("tk:detected-language"),
                "start": span["start_offset"],
                "end": span["end_offset"],
                "text": chunk.get("text"),
                "vector": decode_vector(chunk["vector"]),
            }


if __name__ == "__main__":
    for name in sys.argv[1:]:
        for record in records(name):
            print(json.dumps(record, ensure_ascii=False))
$ TIKA_URL=http://localhost:9998 python3 tika_rag_client.py publisher-email.eml     # first record, wrapped
{"id": "5de2ff9dc8c4fe21:/#0", "file": "publisher-email.eml", "embedded_path": "/", "content_type": "message/rfc822",
 "title": "Spring submissions: samples attached", "language": "eng", "start": 0, "end": 110,
 "text": "Spring submissions: samples attached  \n\nHello,  Please find the spring samples in the attached archive.  Marta",
 "vector": [0.10000000149011612, 0.20000000298023224, 0.30000001192092896, 0.4000000059604645]}

Source: Apache Tika 4.1.x, Chunk Strategies (the tk:chunks layout and the vector encoding) and Inference: engines and bindings. Run on Python 3.9 against tika-server 4.1.0 and the mock; the vectors are the mock’s canned values.

What the run showed:

  • One embeddings request per file, not per chunk. The mock logged one POST /v1/embeddings with eight inputs for the whole publisher-email.eml tree (the ZIP listing included, since Tika embeds it too; the client only skips it when storing). Exclude archives from the binding with _mime-exclude if those calls cost money.

  • tk:chunks is a string. In the /rmeta JSON the value is one string containing a JSON array, so the client parses it a second time. Each chunk has text, vector, producer (the binding id), modality and locators.

  • Vectors are base64 of big-endian float32. struct.unpack(">%df") in Python, ByteBuffer.wrap(bytes).asFloatBuffer() in Java. The mock’s [0.1, 0.2, 0.3, 0.4] round-tripped exactly (to float32 precision).

  • Text chunks have only a text locator. No page numbers, as discussed in Page numbers for citations; a PAGES binding adds paginated locators on its own page chunks.

  • The inference stage runs before the metadata filters, so an allow-list must include tk:chunks, or the vectors are computed, paid for and then dropped.

With Tika Pipes, the Elasticsearch and OpenSearch emitters can write the chunks for you. chunkStrategy is INLINE (the default: tk:chunks stays a nested array in the file’s document, vectors still base64) or DOCUMENTS (one index document per chunk next to the file’s, vectors decoded to float arrays under v.<producer>, locators flattened to fields such as start_offset, end_offset, page, text). DOCUMENTS requires attachmentStrategy: SEPARATE_DOCUMENTS; the emitter refuses any other pairing at load.

{
  "emitters": {
    "bookshelf-rag": {
      "es-emitter": {
        "esUrl": "https://search.bookshelf.internal:9200/bookshelf-chunks",
        "idField": "doc_id",
        "attachmentStrategy": "SEPARATE_DOCUMENTS",
        "chunkStrategy": "DOCUMENTS",
        "updateStrategy": "OVERWRITE",
        "apiKey": "${env:BOOKSHELF_ES_API_KEY}"
      }
    }
  },
  "pipes": { "parseMode": "RMETA" }
}

Source: Apache Tika 4.1.x, Chunk Strategies and Elasticsearch Plugin. Not run. Use RMETA or CONCATENATE; CONTENT_ONLY emits only the text and drops the chunks. The documentation warns that re-indexing replaces chunk documents by id (<id>-chunk-0, …​) but does not delete the extra chunks of an earlier, longer version: delete by container_id before re-indexing a changed file. The index mapping and the search side are on Search Indexing with Lucene, Solr & Elasticsearch and Tika Pipes.

What Tika does not recover, and when to escalate

Tika reads what a format stores. For DOCX, HTML and EPUB that includes the structure, because the format stores it. For PDF it is mostly glyphs at positions, and Tika (through PDFBox) recovers text in a reasonable order, but it is not a layout model. The official documentation does not offer, and this page did not find, any Tika feature for:

  • Reading order across columns and boxes. A two-column article or a page with side boxes may interleave lines from different columns. PDFBox’s sortByPosition helps some files and hurts others; see PDF Parsing.

  • Table structure in PDFs. Cells come out as lines or paragraphs; row and column relationships are lost. A question such as what is the price of Paper Lanterns? fails when the title and the price ended up in different chunks.

  • Figures, charts and diagrams. No description of what they show, unless a VLM recognizer transcribes them.

  • Headings in untagged PDFs. Without marked content there are no # headings, so heading-based chunking falls back to paragraphs.

  • Bounding boxes. The tk:chunks format defines a spatial locator, but the 4.1.x documentation states that no component produces it yet, so you cannot highlight a passage on the page image from Tika output alone.

When the corpus needs those, escalate per document, not per corpus. The site’s Document Parsing page describes the parser ladder (text-layer parsers, then layout models such as Docling and Unstructured, then cloud document AI, then a vision model) and a quality check that decides when to climb. Tika supplies the signals for that check:

Signal from Tika Escalate to

pdf:ocr-page-count > 0 and little text

OCR (Tesseract) or a VLM recognizer, still inside Tika (Scanned content: OCR and VLM text)

High pdf:overall-percentage-unmapped-unicode-chars

OCR of the rendered page: the text layer is garbled

Many short lines, numbers separated from labels, a document type known to be tabular (price lists, statements)

A layout model with table extraction, or a VLM that answers in Markdown tables

Multi-column scientific or newspaper layouts

A layout model that orders reading by region

Everything else (DOCX, HTML, EPUB, e-mail, simple PDFs)

Stay with Tika: it is faster and cheaper by orders of magnitude

A common hybrid: Tika for detection, metadata, embedded documents and every non-PDF format; a layout model for the PDFs the check routes to it; one chunk-record format for both, so the index cannot tell where a chunk came from except by a parser field you set.

Source: Apache Tika 4.1.x, Chunk Strategies (the spatial locator) and Apache Tika 4.1.x, PDF Parser.

Spring AI and LangChain4j readers

Both Java AI frameworks ship a Tika-based reader. The site’s RAG pages already show them in use, so this page only records what they do with Tika, checked from their source and run on the sample files:

Spring AI 2.0.1 TikaDocumentReader LangChain4j 1.22.0 ApacheTikaDocumentParser

Tika version it depends on

tika-core and tika-parsers-standard-package 3.3.1 (2.1.0-M1 not checked)

tika-core and tika-parsers-standard-package 3.2.3

Parser and handler

new AutoDetectParser(), new BodyContentHandler(-1): default configuration, plain text, no write limit

AutoDetectParser, BodyContentHandler(-1) by default; parser, handler, metadata and context can be supplied

Documents per file

One Document; attachment text is concatenated into it (the e-mail run included the DOCX text)

One Document; same concatenation

Metadata

Only source (the file name)

With new ApacheTikaDocumentParser(true), every Tika key, multi-values joined with ;, under 3.x names (X-TIKA:Parsed-By, resourceName, pdf:charsPerPage)

Image or scanned file

A Document with empty text (book-cover.jpg)

BlankDocumentException

Markdown, recursive list, tk:chunks, language

No

No

Source: the TikaDocumentReader source in the spring-ai-tika-document-reader 2.0.1 sources jar and the ApacheTikaDocumentParser source in the langchain4j-document-parser-apache-tika 1.22.0-beta32 sources jar, both from Maven Central; Spring AI, ETL Pipeline and LangChain4j, Apache Tika document parser. Run on the sample files with the readers' own Tika 3.x dependencies.

The important finding is the version. Neither reader works with Tika 4.x on the class path. Importing the Tika 4.1.0 BOM into a project that uses both readers, so that tika-core resolved to 4.1.0, made every call fail:

spring-ai   manuscript.docx FAILED java.lang.NoSuchMethodError: 'void org.apache.tika.parser.AutoDetectParser.parse(
    java.io.InputStream, org.xml.sax.ContentHandler, org.apache.tika.metadata.Metadata, org.apache.tika.parser.ParseContext)'
langchain4j manuscript.docx FAILED java.lang.NoSuchMethodError: 'void org.apache.tika.parser.Parser.parse(
    java.io.InputStream, org.xml.sax.ContentHandler, org.apache.tika.metadata.Metadata, org.apache.tika.parser.ParseContext)'

Tika 4.x changed Parser.parse to take a TikaInputStream instead of an InputStream, and the readers are compiled against the 3.x signature. The same project also ended up with a 3.2.3 tika-parsers-standard-package next to 4.1.0 parser modules, a mix nobody should ship. The consequences for a Bookshelf service on Tika 4.x:

  • Do not mix. A service that embeds Tika 4.x cannot also use these readers in the same class path. Check with mvn dependency:tree -Dincludes=org.apache.tika.

  • Keep the reader, keep Tika 3.x. An application that only needs the reader can stay on the reader’s Tika 3.x line, which is supported until March 2027 per the 4.1.x roadmap page (the wiki roadmap gives an earlier date). Track the framework’s upgrade to 4.x.

  • Or write your own reader. A Spring AI DocumentReader or a LangChain4j DocumentParser is a small interface. Wrap a tika-server client (the Python client above, in Java with RestClient), or PipesForkParser, and return one framework Document per embedded document with the metadata of Metadata as filters. Then the framework’s splitters, embedding models and vector stores take over unchanged. The Spring wiring for calling tika-server is on Spring Boot Integration.

Even on Tika 3.x, the readers' defaults are the plain-text, one-document-per-file shape this page argues against for RAG: no headings for the splitter, attachments merged, and (for Spring AI) no metadata to filter on. Both readers accept other handlers or parsers through their constructors, but they still return a single document per file. For the framework sides themselves see Spring AI: Retrieval & RAG Bridge, Integrating with Spring AI and LangChain4j.

Re-ingestion and freshness

A RAG index goes stale in two ways: the documents change, and the extractor changes. Tika is involved in both.

  • Content changes. Hash the bytes (the source_sha256 above) and skip unchanged files. Tika can compute digests itself through the digesters configuration, which is useful in Pipes where your code never sees the bytes; see Metadata.

  • Extractor changes. A Tika upgrade can change the text of the same file: 4.0 switched the default handler to Markdown, renamed most metadata keys to tk: names, and later releases improve parsers. Store the Tika version and tk:parsed-by with each chunk, and plan a re-parse of the corpus after an upgrade, not only a re-embed. Comparing old and new extraction on a sample is what tika-eval is for; see tika-eval & Extraction Quality.

  • Deletes. When a file is removed or shrinks, its old chunks must go. Chunk ids derived from the source id make that a delete by prefix or by a source_sha256 field; with the Pipes DOCUMENTS strategy, delete by container_id.

Batch versus streaming ingestion, change data capture, tombstones and provenance hashes are covered on Ingestion Pipelines & Freshness.

Security and privacy

Documents fed to an LLM are untrusted input twice: once to the parser, once to the model.

  • Parser safety. Run Tika out of process for untrusted uploads (tika-server or PipesForkParser), set timeouts and limits, and keep it patched (the PDF XFA XXE, CVE-2025-54988 and its wider re-issue CVE-2025-66516, affected Tika up to 3.2.1). See Security and Robustness, Limits & Timeouts.

  • Indirect prompt injection. Tika extracts text a human reader never sees: white-on-white text, text behind images, hidden spreadsheet cells, PDF annotations, document properties, HTML comments and alt text. An attacker can place instructions there, and once chunked and retrieved they reach the model’s context. Tika does not filter them and should not; apply the guardrails on Guardrails & Prompt Injection, keep retrieved text clearly delimited in the prompt, and consider dropping annotations and metadata fields from the chunk text.

  • Personal data. Extraction surfaces form field values (the catalogue’s member number above), e-mail headers and author names. Redact before embedding, since a vector store is hard to clean afterwards; see Access Control & Privacy.

  • Credentials and data residency. Keep every engine key in the environment (${env:NAME}), never in tika-config.json, and remember that a hosted VLM or embedding engine receives the document’s pages and text. Use a local model for documents that must not leave your network.

Memory, performance and robustness

  • Parsing is rarely the bottleneck in RAG; embedding and OCR are. In the runs here parsing a small file took tens of milliseconds (tk:parse-time-millis), while an embedding or VLM call takes hundreds of milliseconds to seconds. Batch the embeddings (Tika’s TEXT binding sends one request per document tree in batches of maxBatchSize) and keep OCR to the pages that need it.

  • The recursive list holds all text in memory. For mailbox exports and large archives set the embedded limits and a write limit, and check tk:exception:* keys for truncation before you store partial text as complete.

  • Inference runs inside the parse budget. IMAGES and PAGES embeddings are flushed inside parse(), under its timeout; a slow embedding endpoint can turn into parse timeouts. Size numClients (server, Pipes) to the endpoint’s concurrency limit, as described on VLM Parsers & Inference.

  • Cost caps. maxChunks and maxBytes on a binding cap what one document can send to a paid endpoint. Set them before pointing Tika at a hosted model.

  • Throughput, deployment shapes and sizing are on Performance & Scaling.

What changed: 1.x to 4.x

Version RAG-relevant state

1.x (the book’s era)

Plain text or XHTML; RecursiveParserWrapper arrived during 1.x; language detection with the old LanguageIdentifier; no notion of chunks or embeddings. Tika in Action (2011) predates RAG; its search-indexing chapter (chapter 10) is the closest ancestor of this page.

2.x

Pipes and the forked-process model; X-TIKA:* metadata keys; plain text remained the usual input to early RAG frameworks, which is the shape the Spring AI and LangChain4j readers still use.

3.x

Java 11; the readers above are built on it. No VLM parsers and no inference section (see VLM Parsers & Inference); RAG pipelines chunked and embedded outside Tika.

4.0

Markdown becomes the default handler; tk: metadata keys; tk:content in recursive output; VLM parsers; the first embedding filters writing tk:chunks; Parser.parse takes a TikaInputStream (the change that breaks the 3.x-based readers).

4.1

engines, text-recognizers and inference bindings (TEXT, IMAGES, PAGES, MEDIA) replace the 4.0 filters (deprecated); chunkStrategy on the Elasticsearch and OpenSearch emitters; ${env:...} interpolation for keys.

Source: Apache Tika 4.1.x, Migrating to Tika 4.x, Content Handler Requirements for Inference and the Apache Tika news page (4.0.0 and 4.1.0 release notes). The 2.x and 3.x rows are summarised from the version notes on the other pages of this section, not re-verified for this page.

Pitfalls

  • Plain-text extraction for RAG. Tika.parseToString and BodyContentHandler still return plain text in 4.x. Ask for Markdown explicitly in Java, or headings are lost before the splitter sees them.

  • One text per container. Concatenated attachments make chunks that span documents and citations that name only the outer file. Use the recursive list.

  • Embedding archive listings and image placeholders. A ZIP’s Markdown is its file names; ![](embedded:...) lines and picture headings add noise. Skip or strip them.

  • Trusting that every page was read. Check pdf:ocr-page-count and empty image/* documents; an unrecognized scan is silently absent.

  • Dropping tk:chunks with an allow-list. Inference runs before metadata filters; include tk:chunks (and the keys your records need) in the list.

  • Assuming tk:chunks is an array. Over /rmeta it is a string holding a JSON array. Parse it twice.

  • Using CONTENT_ONLY with inference in Pipes. It drops the chunks at emit time.

  • Expecting page numbers from Markdown chunks. Text chunks carry character offsets only.

  • Mixing Tika 3.x readers with Tika 4.x. NoSuchMethodError at the first parse. Pick one line per class path.

  • Treating document metadata as truth. dc:creator and dates come from the file; do not use them for access control.

  • Sending everything to a hosted model. Cost and data residency; set maxChunks, maxBytes and media-type filters.

What was verified and what was not

Verified by running on Tika 4.1.0 (Java 25 for the Java program, Python 3.9 for the clients):

  • BookshelfRagIngest: the recursive Markdown parse, MarkdownChunker offsets and overlap behaviour, CharSoup per chunk, the archive and empty-document handling, pdf:ocr-page-count on the scanned form, and the reserved-key exception.

  • tika-server 4.1.0 with the shown tika-config.json: /rmeta/md with the TEXT binding against a local mock embeddings server (one request per tree), the charsoup-metadata-filter and allow-list output, tk:chunks as a string, the vector decoding, and the startup failure without a server block.

  • /tika/md, /tika/xml and /rmeta/xml on the tagged catalogue.pdf, and the per-page Python client (artifact divs, the embedded errata.pdf, the outline and form values after the last page).

  • Spring AI 2.0.1 and LangChain4j 1.22.0-beta32 readers: their Tika dependency versions (from the POMs on Maven Central), their source, their output on the sample files with Tika 3.x, and the NoSuchMethodError with Tika 4.1.0.

Not verified:

  • The Tesseract and VLM configurations in Scanned content: OCR and VLM text (no Tesseract on the test machine, no VLM called); they are reduced from configurations run on the OCR with Tesseract and VLM Parsers & Inference pages.

  • The Pipes es-emitter with chunkStrategy: DOCUMENTS; it follows the 4.1.x Chunk Strategies page.

  • Real embedding models, real vector stores and retrieval quality: the mock returns fixed vectors, so nothing on this page measures retrieval.

  • The Gradle build file; only Maven was run.

  • Whether a later Spring AI or LangChain4j release moves to Tika 4.x; check their POMs when you upgrade.

Checklist

  • Tika runs out of process for untrusted documents (tika-server, PipesForkParser or Pipes).

  • Output is Markdown, parsed recursively; each embedded document is its own source with tk:embedded-resource-path.

  • Archive listings, picture placeholders and tiny chunks are skipped or merged.

  • Pages flagged for OCR and empty images are counted; a recognizer runs where it pays off, and recognized text is marked.

  • Every chunk carries a stable id, the source hash, title, type, language and (for PDFs, if citations need it) the page number.

  • Metadata filters keep only indexed keys, with tk:chunks included when Tika embeds.

  • Engine keys come from the environment; maxChunks, maxBytes and media-type filters cap paid calls.

  • Layout-heavy PDFs are routed to a layout model by a measured signal, not by guesswork.

  • The Tika version and tk:parsed-by are stored, and a re-parse is planned for upgrades.

  • No Tika 3.x-based framework reader shares a class path with Tika 4.x.

References