Extending Tika

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

Baseline date: 2026-10-12. This page was written against Apache Tika 4.1.0 (the latest release on Maven Central on that date) with the 3.3.x differences noted, Java 21 and later for the examples (Tika 4.x needs Java 17), Spring Boot 4.x on Spring Framework 7 and the Docker image apache/tika:4.1.0. Tika is built to be extended: every parser, detector, encoding detector, metadata filter and OCR engine it ships is found through the same service and component mechanisms that your own classes can use. This page writes the pieces the Bookshelf intake service needs that Tika does not have: a parser for its own purchase-order format, a detector for its batch archives, a decorator that stamps every document with its intake channel, a command-line tool wired in as a parser, and a remote OCR service wired in as a text recognizer. It then shows how each piece is registered, named in tika-config.json and kept safe.

The Parser contract itself (the four arguments, stream ownership, exceptions, CompositeParser and DefaultParser) is on Parser API & AutoDetectParser, the media-type registry and the detector pipeline are on Media Types & Detection, the XHTML that a parser must emit is on Content Handlers & Output Formats, and the key rules are on Metadata. The JSON file format is on Configuration, the built-in OCR engine is on OCR with Tesseract and the VLM engines are on VLM Parsers & Inference.

All Java examples on this page were compiled with javac from JDK 25 (release 21) and run against Tika 4.1.0 in a test Maven project: tika-core, tika-parsers-standard-package as a pom dependency, tika-serialization for TikaLoader, and tika-annotation-processor for @TikaComponent. The sample files are small stand-ins: a two-line purchase order order-0042.bso, a batch.zip that carries a bookshelf-batch.json entry and a plain.zip that does not, the book-cover.jpg (800 x 1200 pixels), catalogue.pdf and legacy-catalogue.txt. The outputs shown are the real outputs of those runs, trimmed of log lines. What was not run is said where it matters, and summarised at the end.

What you can extend

Tika has one extension idea, applied to several interfaces: write a class that implements the interface, register it so that Tika can find it, and, in 4.x, give it a component name so that tika-config.json can configure it.

Extension point Interface (4.1.0) Typical reason

Parser

org.apache.tika.parser.Parser

A format Tika does not read, or reads in a way you need to change.

Detector

org.apache.tika.detect.Detector

A format that magic bytes and file names cannot recognise, such as a ZIP with a known entry.

Media types

custom-mimetypes.xml (data, not code)

A format with a recognisable signature or extension that only needs a name and a parent.

Encoding detector

org.apache.tika.detect.EncodingDetector

A charset heuristic for your corpus; see Character Encoding Detection.

Metadata filter

org.apache.tika.metadata.filter.MetadataFilter

Rename, drop or normalise keys after the parse; see Metadata.

Language detector, translator

org.apache.tika.language.detect.LanguageDetector, org.apache.tika.language.translate.Translator

See Language Detection & Translation.

OCR or other enrichment engine

Parser plus org.apache.tika.parser.enricher.TextRecognizer (or ContentEnricher)

A recognizer other than Tesseract or the built-in VLM engines.

External program

external-parser configuration, no code

A command-line tool (exiftool, ffmpeg, sox) whose output you want as metadata or text.

Fetcher, emitter, iterator, reporter

org.apache.tika.pipes.api.* factories in a PF4J plugin

A source or a sink that Tika Pipes does not ship.

Each one costs you the same three things: correctness on hostile input (every file the extension sees may be malformed or malicious), a bounded use of memory and time, and a registration that does not surprise anyone who later reads the configuration. The sections below show how to pay each of them.

classDiagram direction TB class Serializable { <<interface>> } class SelfConfiguring { <<interface>> } class Parser { <<interface>> +getSupportedTypes(ParseContext) Set~MediaType~ +parse(TikaInputStream, ContentHandler, Metadata, ParseContext) void } class AbstractParser { <<abstract>> +parse(TikaInputStream, ContentHandler, Metadata) void } class ParserDecorator { -Parser parser +withTypes(Parser, Set~MediaType~)$ Parser +withoutTypes(Parser, Set~MediaType~)$ Parser +withMimeFilters(Parser, Set~MediaType~, Set~MediaType~)$ Parser +getWrappedParser() Parser +getDecorationName() String } class MimeFilteringDecorator { +getIncludeTypes() Set~MediaType~ +getExcludeTypes() Set~MediaType~ } class CompositeParser class DefaultParser class AutoDetectParser class TextRecognizer { <<interface>> +recognizesText(ParseContext) boolean } class OrderParser { bookshelf-order-parser } class ChannelStampingParser class ShelfOcrParser { shelf-ocr-parser } Serializable <|-- Parser SelfConfiguring <|-- Parser Parser <|.. AbstractParser Parser <|.. ParserDecorator Parser <|.. CompositeParser CompositeParser <|-- DefaultParser CompositeParser <|-- AutoDetectParser ParserDecorator <|-- MimeFilteringDecorator ParserDecorator o-- Parser : wraps one CompositeParser o-- Parser : maps types to Parser <|.. OrderParser ParserDecorator <|-- ChannelStampingParser Parser <|.. ShelfOcrParser TextRecognizer <|.. ShelfOcrParser

The diagram shows the parser side of the hierarchy in 4.1.0. Parser extends Serializable and the marker interface SelfConfiguring, which tells the configuration loader to leave a parser’s JSON in the ParseContext for the parser to read. AbstractParser is now only a deprecated adapter. ParserDecorator wraps exactly one parser and CompositeParser maps media types to many. The three Bookshelf classes (OrderParser, ChannelStampingParser and ShelfOcrParser) are the ones written on this page. TextRecognizer extends ContentEnricher, which extends org.apache.tika.parser.inference.Engine, a Closeable; that chain is left out of the diagram.

Source: Parser, Javadoc 4.1.0, ParserDecorator, Javadoc 4.1.0 and TextRecognizer, Javadoc 4.1.0.

A custom Parser

The Bookshelf service receives purchase orders from its suppliers as small text files with a fixed first line, a header block, a --- separator and one line per title:

order-0042.bso
BSHELF-ORDER/1
Order: PO-2026-0042
Supplier: Harbour Lane Books
Date: 2026-10-01
X-Courier: Northwind Parcels
X-Priority: rush
---
9780000000017 | The Lighthouse Keeper | 3
9780000000024 | Tides of Paper | 1

Without help, Tika types the file as text/plain and indexes it as one flat string. The Media Types & Detection page already declared application/vnd.bookshelf.order in a custom-mimetypes.xml, with a glob, a magic string and text/plain as its parent (that file is repeated in Custom MIME types below). What is missing is a parser that turns the header into metadata and the order lines into a table.

Parser or AbstractParser

The book and the 3.x guides tell you to extend AbstractParser. In 4.1.0 that class is marked @Deprecated with the note "for removal in 4.x": it only implements a three-argument parse method left over from Tika 0.x by calling the four-argument one with a new ParseContext. A new parser implements Parser directly, which has exactly two methods to write:

Method What your implementation must do

Set<MediaType> getSupportedTypes(ParseContext context)

Return the media types this parser handles. CompositeParser (and therefore AutoDetectParser) calls it to build its type-to-parser map. Return a constant set; compute it from the context only when the answer really depends on the request, and return an empty set when the parser cannot work (for example, a missing native tool), so that the type falls back to another parser instead of failing.

void parse(TikaInputStream tis, ContentHandler handler, Metadata metadata, ParseContext context)

Read the stream, write XHTML SAX events to handler (through an XHTMLContentHandler) and set keys in metadata. Do not close the stream; the caller owns it. Throw TikaException for a document that is not what it claims to be, IOException for a read failure, and SAXException only when the handler throws one (for example, a write limit).

The parser

The keys first. The header fields are known in advance, so each gets a Property constant in the parser’s own bso: namespace; the X- headers are named by the supplier, so they go under a declared KeyPrefix instead (the reason is explained in Adding a metadata key):

package demo;

import org.apache.tika.metadata.KeyPrefix;
import org.apache.tika.metadata.Property;

/** Metadata keys coined by the Bookshelf order parser. */
public final class OrderProperties {

    /** Fixed header fields: one constant each. */
    public static final Property ORDER_NUMBER = Property.internalText("bso:order-number");
    public static final Property SUPPLIER = Property.internalText("bso:supplier");
    public static final Property ORDER_DATE = Property.internalDate("bso:order-date");
    public static final Property LINE_COUNT = Property.internalInteger("bso:line-count");
    public static final Property TOTAL_COPIES = Property.internalInteger("bso:total-copies");

    /** X- headers: the names come from the document, so they go under a declared prefix. */
    public static final KeyPrefix HEADER =
            KeyPrefix.file("bso:header:", "X- header fields of a Bookshelf purchase order");

    private OrderProperties() {
    }
}

Then a small options bean, whose property names become JSON keys:

package demo;

import java.io.Serializable;

/** Options of the order parser; the field names are the JSON keys. */
public class OrderParserConfig implements Serializable {

    private static final long serialVersionUID = 1L;

    private int maxLines = 10_000;
    private boolean emitTable = true;

    public int getMaxLines() {
        return maxLines;
    }

    public void setMaxLines(int maxLines) {
        this.maxLines = maxLines;
    }

    public boolean isEmitTable() {
        return emitTable;
    }

    public void setEmitTable(boolean emitTable) {
        this.emitTable = emitTable;
    }
}

And the parser:

package demo;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.charset.StandardCharsets;
import java.util.Set;

import org.xml.sax.ContentHandler;
import org.xml.sax.SAXException;

import org.apache.tika.annotation.TikaComponent;
import org.apache.tika.config.ConfigDeserializer;
import org.apache.tika.config.JsonConfig;
import org.apache.tika.config.ParseContextConfig;
import org.apache.tika.exception.TikaConfigException;
import org.apache.tika.exception.TikaException;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.HttpHeaders;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.metadata.TikaCoreProperties;
import org.apache.tika.mime.MediaType;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.parser.Parser;
import org.apache.tika.sax.XHTMLContentHandler;

/** Parses Bookshelf purchase orders (application/vnd.bookshelf.order). */
@TikaComponent(name = "bookshelf-order-parser")
public class OrderParser implements Parser {

    private static final long serialVersionUID = 1L;

    public static final MediaType ORDER = MediaType.application("vnd.bookshelf.order");
    private static final Set<MediaType> TYPES = Set.of(ORDER);
    private static final String MAGIC = "BSHELF-ORDER/1";

    private final OrderParserConfig defaultConfig;

    public OrderParser() {
        this(new OrderParserConfig());
    }

    public OrderParser(OrderParserConfig config) {
        this.defaultConfig = config;
    }

    /** Called by TikaLoader with the JSON body of {"bookshelf-order-parser": {...}}. */
    public OrderParser(JsonConfig json) {
        this(ConfigDeserializer.buildConfig(json, OrderParserConfig.class));
    }

    @Override
    public Set<MediaType> getSupportedTypes(ParseContext context) {
        return TYPES;
    }

    @Override
    public void parse(TikaInputStream tis, ContentHandler handler, Metadata metadata,
                      ParseContext context) throws IOException, SAXException, TikaException {
        OrderParserConfig config = config(context);
        // The stream belongs to the caller: read it, never close it.
        BufferedReader reader = new BufferedReader(
                new InputStreamReader(tis, StandardCharsets.UTF_8));
        String first = reader.readLine();
        if (first == null || !first.equals(MAGIC)) {
            throw new TikaException("Not a Bookshelf order: missing " + MAGIC + " line");
        }
        metadata.set(HttpHeaders.CONTENT_TYPE, ORDER.toString());

        XHTMLContentHandler xhtml = new XHTMLContentHandler(handler, metadata);
        xhtml.startDocument();

        String line;
        while ((line = reader.readLine()) != null && !line.equals("---")) {
            int colon = line.indexOf(':');
            if (colon <= 0) {
                continue;
            }
            String name = line.substring(0, colon).trim();
            String value = line.substring(colon + 1).trim();
            switch (name) {
                case "Order" -> {
                    metadata.set(OrderProperties.ORDER_NUMBER, value);
                    metadata.set(TikaCoreProperties.TITLE, "Purchase order " + value);
                }
                case "Supplier" -> metadata.set(OrderProperties.SUPPLIER, value);
                case "Date" -> metadata.set(OrderProperties.ORDER_DATE, value + "T00:00:00Z");
                default -> {
                    if (name.startsWith("X-")) {
                        metadata.add(OrderProperties.HEADER, name.substring(2), value);
                    }
                }
            }
        }

        xhtml.element("h1", metadata.get(TikaCoreProperties.TITLE));
        if (config.isEmitTable()) {
            xhtml.startElement("table");
        }
        int lines = 0;
        int copies = 0;
        while ((line = reader.readLine()) != null && lines < config.getMaxLines()) {
            String[] cells = line.split("\\|");
            if (cells.length != 3) {
                continue;
            }
            lines++;
            copies += Integer.parseInt(cells[2].trim());
            if (config.isEmitTable()) {
                xhtml.startElement("tr");
                for (String cell : cells) {
                    xhtml.element("td", cell.trim());
                }
                xhtml.endElement("tr");
            } else {
                xhtml.element("p", cells[1].trim() + " x" + cells[2].trim());
            }
        }
        if (config.isEmitTable()) {
            xhtml.endElement("table");
        }
        metadata.set(OrderProperties.LINE_COUNT, lines);
        metadata.set(OrderProperties.TOTAL_COPIES, copies);
        xhtml.endDocument();
    }

    /** Per-request overrides arrive in the ParseContext under the component name. */
    private OrderParserConfig config(ParseContext context) throws IOException {
        try {
            return ParseContextConfig.getConfig(context, "bookshelf-order-parser",
                    OrderParserConfig.class, defaultConfig);
        } catch (TikaConfigException e) {
            throw new IOException(e);
        }
    }
}

Source: Parser, Javadoc 4.1.0, XHTMLContentHandler, Javadoc 4.1.0, Apache Tika 4.1.x, Serialization and Configuration (the JsonConfig constructor and ParseContextConfig.getConfig pattern) and Apache Tika 3.3.2, Parser guide.

What each part is for:

XHTMLContentHandler

Wraps the caller’s handler and guarantees well-formed XHTML: startDocument() writes <html>, a <head> with one <meta> per metadata value and the <title>, and opens <body>; endDocument() closes it all. Use element(name, value), startElement/endElement and characters for the body. Keep to the elements a downstream handler understands (p, h1 to h6, table/tr/td, ul/li, a, div); the Markdown and text handlers map those.

Metadata before startDocument()

The <head> is written when startDocument() runs, so only the keys set before it appear there. In the run below, bso:line-count and bso:total-copies are in the Metadata object but not in the XHTML head, because they are only known at the end. That is normal; just do not expect a consumer that reads only the XHTML to see them.

Content-Type

AutoDetectParser has already set it from detection. Set it yourself only to refine it (for example, a version parameter); writing a different family confuses everyone downstream.

Configuration

The JsonConfig constructor is what the loader calls for {"bookshelf-order-parser": {...}} in tika-config.json. ConfigDeserializer.buildConfig binds the JSON to the bean with Jackson, which needs tika-serialization on the classpath. ParseContextConfig.getConfig then merges a per-request override (if the caller put one in the ParseContext under the component name) with that default.

Limits

maxLines caps the work for a hostile file with millions of lines. The XHTML size is already capped by the caller’s write limit; a parser’s own loops are not, so bound them.

Running it

With the class compiled and registered (next section), a plain AutoDetectParser finds it:

package demo;

import java.nio.file.Path;

import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.parser.AutoDetectParser;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.parser.Parser;
import org.apache.tika.sax.ToXMLContentHandler;

public class ParseOrder {
    public static void main(String[] args) throws Exception {
        Parser parser = new AutoDetectParser();
        Metadata metadata = new Metadata();
        ToXMLContentHandler handler = new ToXMLContentHandler();
        try (TikaInputStream tis = TikaInputStream.get(Path.of(args[0]), metadata)) {
            parser.parse(tis, handler, metadata, new ParseContext());
        }
        System.out.println(handler);
        for (String name : metadata.names()) {
            System.out.println(name + " = " + String.join(", ", metadata.getValues(name)));
        }
    }
}

Output for order-0042.bso (the XHTML, then the metadata):

<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta name="bso:order-date" content="2026-10-01T00:00:00Z" />
<meta name="tk:content-type-magic-detected" content="application/vnd.bookshelf.order" />
<meta name="bso:order-number" content="PO-2026-0042" />
<meta name="bso:header:Priority" content="rush" />
<meta name="dc:title" content="Purchase order PO-2026-0042" />
<meta name="tk:resource-name" content="order-0042.bso" />
<meta name="bso:supplier" content="Harbour Lane Books" />
<meta name="bso:header:Courier" content="Northwind Parcels" />
<meta name="Content-Length" content="208" />
<meta name="tk:parsed-by" content="org.apache.tika.parser.DefaultParser" />
<meta name="tk:parsed-by" content="demo.OrderParser" />
<meta name="Content-Type" content="application/vnd.bookshelf.order" />
<title>Purchase order PO-2026-0042</title>
</head>
<body><h1>Purchase order PO-2026-0042</h1>
<table><tr>	<td>9780000000017</td>	<td>The Lighthouse Keeper</td>	<td>3</td></tr>
<tr>	<td>9780000000024</td>	<td>Tides of Paper</td>	<td>1</td></tr>
</table>
</body></html>
bso:order-date = 2026-10-01T00:00:00Z
bso:order-number = PO-2026-0042
bso:header:Priority = rush
tk:resource-name = order-0042.bso
bso:header:Courier = Northwind Parcels
tk:parsed-by = org.apache.tika.parser.DefaultParser, demo.OrderParser
bso:total-copies = 4
tk:content-type-magic-detected = application/vnd.bookshelf.order
dc:title = Purchase order PO-2026-0042
bso:supplier = Harbour Lane Books
tk:parsed-by-full-set = org.apache.tika.parser.DefaultParser, demo.OrderParser
Content-Length = 208
bso:line-count = 2
Content-Type = application/vnd.bookshelf.order

tk:parsed-by is the proof that your parser ran: DefaultParser dispatched to demo.OrderParser. If it shows TXTParser or TextAndCSVParser instead, either the type was not detected as application/vnd.bookshelf.order (check tk:content-type-magic-detected and the custom-mimetypes.xml) or the parser was not registered.

A per-request override goes through the ParseContext under the component name. With context.setJsonConfig("bookshelf-order-parser", "{\"emitTable\": false}") the same parser, unchanged, writes paragraphs instead of a table:

<body><h1>Purchase order PO-2026-0042</h1>
<p>The Lighthouse Keeper x3</p>
<p>Tides of Paper x1</p>
</body></html>

On tika-server and in pipes, per-request configuration is off by default and arrives over the wire in restricted mode; treat any parser option you expose as something a caller may set. The trust model is explained on the 4.1.x serialization page and in Configuration.

Source: ParseContext, Javadoc 4.1.0 (setJsonConfig) and ParseContextConfig, Javadoc 4.1.0.

Registering components

Tika finds a component in one of two ways, and 4.x adds a third piece that is easy to miss.

Mechanism What it is What it gives you

Java service file

META-INF/services/org.apache.tika.parser.Parser (or ...detect.Detector, and so on) listing class names, one per line

DefaultParser and DefaultDetector load the class automatically: drop the jar on the classpath and it runs.

Component index

META-INF/tika/parsers.idx (and detectors.idx, encoding-detectors.idx, metadata-filters.idx, translators.idx, renderers.idx, language-detectors.idx, parse-context.idx) mapping a component name to a class

tika-config.json can name the component ("bookshelf-order-parser"). Without an index entry, a name fails with "Unknown component name"; class names are not accepted in its place.

@TikaComponent

A compile-time annotation in org.apache.tika.annotation, processed by TikaComponentProcessor

Generates both of the above for you.

The @TikaComponent annotation

The annotation has four attributes. Everything else is derived:

Attribute Default Effect

name

derived

The component name used in JSON. Empty means the class name in kebab case: OrderParser would become order-parser, PDFParser becomes pdf-parser. Give your own components an explicit, prefixed name (bookshelf-order-parser) so they never clash with a future Tika name.

spi

true

Whether the class is also written to META-INF/services. true means default-parser loads it automatically; false means it exists only when a configuration names it (Tika uses this for opt-in components such as regex-capture-parser, magic-detector and its own text recognizers' alternatives).

contextKey

auto

The class used as the key when the component is put in a ParseContext. Detected from the known interfaces; set it only when a class implements more than one.

defaultFor

none

Marks the class as the default implementation of an interface when a ParseContext is loaded with defaults.

After a build with the processor, the test project contained:

target/classes/META-INF/services/org.apache.tika.parser.Parser
    demo.OrderParser

target/classes/META-INF/tika/parsers.idx
    bookshelf-order-parser=demo.OrderParser:key=org.apache.tika.parser.Parser
    shelf-ocr-parser=demo.ShelfOcrParser:key=org.apache.tika.parser.Parser

target/classes/META-INF/tika/detectors.idx
    bookshelf-batch-detector=demo.BatchZipDetector:key=org.apache.tika.detect.Detector

ShelfOcrParser and BatchZipDetector (both spi = false, later on this page) are in the index but not in a service file, so they run only when tika-config.json names them.

Source: TikaComponent.java, Tika 4.1.0 source and Apache Tika 4.1.x, Serialization and Configuration.

Build setup: the processor must actually run

The annotation and its processor ship in tika-annotation-processor, a compile-time-only dependency (the annotation has CLASS retention, so nothing of it is needed at run time). The 4.1.x OCR-engine guide says that javac picks the processor up from the classpath. That was true until JDK 22: since JDK 23, javac no longer runs processors found on the class path unless asked to. In the test project, compiling with JDK 25 and the dependency only in provided scope produced no META-INF files at all, silently. Declare the processor explicitly:

<dependencies>
    <dependency>
        <groupId>org.apache.tika</groupId>
        <artifactId>tika-core</artifactId>
        <version>4.1.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.tika</groupId>
        <artifactId>tika-annotation-processor</artifactId>
        <version>4.1.0</version>
        <scope>provided</scope>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <annotationProcessorPaths>
                    <path>
                        <groupId>org.apache.tika</groupId>
                        <artifactId>tika-annotation-processor</artifactId>
                        <version>4.1.0</version>
                    </path>
                </annotationProcessorPaths>
            </configuration>
        </plugin>
    </plugins>
</build>
dependencies {
    implementation("org.apache.tika:tika-core:4.1.0")
    compileOnly("org.apache.tika:tika-annotation-processor:4.1.0")
    annotationProcessor("org.apache.tika:tika-annotation-processor:4.1.0")
}

With plain javac, -proc:full (or -processorpath) has the same effect; the run printed Generated SPI file: META-INF/services/org.apache.tika.parser.Parser with 1 implementations and Generated index file: META-INF/tika/parsers.idx with 1 components. The Maven form was run; the Gradle form was not. Add a build check that the jar contains META-INF/tika/parsers.idx, because a missing index only shows up at run time, as "Unknown component name" for configured components or as a parser that never runs.

Source: Apache Tika 4.1.x, Wiring in Your Own OCR Engine (Build and package) and Inside Java, JDK 23 changes the default annotation processing policy.

Without the annotation: a service file by hand

You can still register a parser the 1.x to 3.x way, with a hand-written service file:

src/main/resources/META-INF/services/org.apache.tika.parser.Parser
demo.OrderParser

DefaultParser loads it, so new AutoDetectParser() and a default-parser entry use it. What you lose is the name: a tika-config.json cannot configure or reference the class. For a parser that needs no options that is enough; for anything configurable, use the annotation.

Who wins for the same type

DefaultParser sorts the service-loaded parsers so that the Tika ones come first, and CompositeParser keeps the last parser registered for a type, so a parser of your own on the classpath beats a Tika parser that claims the same type. The same "last one wins" applies to the order of entries in a parsers list in tika-config.json, which is easy to get backwards (see Overriding and decorating parsers). Detectors work the other way round: DefaultDetector runs the service-loaded ones in class-name order after magic detection and takes the most specific answer.

Where the jar goes

  • Embedded Tika: on the application classpath, next to tika-core.

  • tika-app and tika-server: in a directory named by the tika.extras.dir system property (java -Dtika.extras.dir=/opt/bookshelf/tika-extras -jar tika-server-standard.jar). Every jar there joins the class path that service loading scans and is forwarded to the forked parse JVMs. Nothing is loaded unless the property is set.

  • Docker: mount the directory at /tika-extras.

The extras directory is a trusted code location: everything in it runs with the full privileges of the Tika process. Keep it read-only for the service account and out of reach of anything that handles requests. The tika.extras.dir mechanism was taken from the 4.1.x configuration page and not run here.

Source: Apache Tika 4.1.x, Configuration (Adding extra jars) and DefaultParser, Javadoc 4.1.0.

Custom MIME types

A parser is chosen by media type, so a new format needs a type before it needs a parser. The Media Types & Detection page covers custom-mimetypes.xml in full (globs, magic, root XML, priorities, the tika.custom-mimetypes system property); this is the file the Bookshelf project keeps at src/main/resources/custom-mimetypes.xml:

<?xml version="1.0" encoding="UTF-8"?>
<mime-info>

  <!-- A Bookshelf purchase order: a text file whose first bytes identify it. -->
  <mime-type type="application/vnd.bookshelf.order">
    <_comment>Bookshelf purchase order</_comment>
    <glob pattern="*.bso"/>
    <magic priority="60">
      <match value="BSHELF-ORDER/1" type="string" offset="0"/>
    </magic>
    <sub-class-of type="text/plain"/>
  </mime-type>

  <!-- An XML manifest recognised by its root element and namespace. -->
  <mime-type type="application/vnd.bookshelf.manifest+xml">
    <root-XML localName="manifest" namespaceURI="urn:bookshelf:manifest:1"/>
    <glob pattern="*.bsm"/>
    <sub-class-of type="application/xml"/>
  </mime-type>

  <!-- A batch of manuscripts: a ZIP that a custom Detector will recognise. -->
  <mime-type type="application/vnd.bookshelf.batch+zip">
    <glob pattern="*.bsb"/>
    <sub-class-of type="application/zip"/>
  </mime-type>

</mime-info>

Three points matter when a type is meant for a custom parser:

  • Declare the parent. sub-class-of decides what happens when your parser is absent: the order falls back to the text parser, the manifest to the XML parser, the batch to the ZIP parser. That is a safe degradation, so always set it.

  • The type string must match exactly. OrderParser.getSupportedTypes returns MediaType.application("vnd.bookshelf.order"); one character of difference and the parent’s parser wins silently.

  • Ship the XML with the parser. Put custom-mimetypes.xml in the same jar as the parser, so the type and its parser are deployed (and removed) together.

Source: MimeTypesFactory, Javadoc 4.1.0 and Media Types & Detection.

A custom Detector

Magic bytes cannot tell a Bookshelf batch from any other ZIP: the batch is a ZIP with a bookshelf-batch.json entry. The detector written on the media types page does that check; here it gets a component name and spi = false, so it runs only where a configuration asks for it:

package demo;

import java.io.IOException;
import java.util.zip.ZipFile;

import org.apache.tika.annotation.TikaComponent;
import org.apache.tika.detect.Detector;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.mime.MediaType;
import org.apache.tika.parser.ParseContext;

/** Recognises a ZIP that carries a bookshelf-batch.json entry. */
@TikaComponent(name = "bookshelf-batch-detector", spi = false)
public class BatchZipDetector implements Detector {

    private static final long serialVersionUID = 1L;

    public static final MediaType BATCH = MediaType.application("vnd.bookshelf.batch+zip");

    @Override
    public MediaType detect(TikaInputStream tis, Metadata metadata, ParseContext context)
            throws IOException {
        if (tis == null) {
            return MediaType.OCTET_STREAM;
        }
        tis.mark(4);
        byte[] head = new byte[4];
        int n = tis.read(head);
        tis.reset();
        if (n < 4 || head[0] != 'P' || head[1] != 'K' || head[2] != 3 || head[3] != 4) {
            return MediaType.OCTET_STREAM;
        }
        try (ZipFile zip = new ZipFile(tis.getPath().toFile())) {
            if (zip.getEntry("bookshelf-batch.json") != null) {
                return BATCH;
            }
        } catch (IOException e) {
            // not a readable ZIP: say nothing
        }
        return MediaType.OCTET_STREAM;
    }
}

The 4.x contract is one method, detect(TikaInputStream, Metadata, ParseContext). The stream may be null (detection by name only); a detector must mark/reset around anything it reads and must not close the stream; it returns application/octet-stream for "I do not know" and never throws for foreign data. tis.getPath() spools the stream to a temporary file once if it is not already a file, which is why the cheap four-byte check comes first.

To use it, list it in the detectors array before a default-detector entry (a detectors list loads only what it names, so leaving out default-detector would drop every built-in detector):

{
  "detectors": [
    "bookshelf-batch-detector",
    "default-detector"
  ]
}

The whole configuration is in Putting it together: the Bookshelf tika-config.json. With it, the test project printed:

files/batch.zip detected as application/vnd.bookshelf.batch+zip
files/plain.zip detected as application/zip

Two behaviours worth knowing:

  • The batch type has no parser of its own, so AutoDetectParser falls back to the parent’s ZipParser, which then sets Content-Type to application/zip in the output metadata. The detected type is what loader.loadDetectors() (or the AutoDetectParser detector) returns; if your index needs it, record it yourself or write a parser for the type.

  • With spi = true (or a hand-written META-INF/services/org.apache.tika.detect.Detector), DefaultDetector would load the detector everywhere, including in new Tika(). For a detector that opens archives, opt-in is the cheaper choice.

Source: Detector, Javadoc 4.1.0 and Apache Tika 4.1.x, Detector Configuration.

Overriding and decorating parsers

Often the parser you need already exists and only its reach is wrong: it claims a type you want somewhere else, or it should also handle a type it does not know. ParserDecorator is the tool for that in code, and the _mime-include/_mime-exclude directives and default-parser exclusions are the same tools in tika-config.json.

In code: withTypes, withoutTypes and a decorator subclass

Factory (4.1.0) Effect

ParserDecorator.withTypes(parser, types)

The result claims exactly types, whatever the wrapped parser claims. That can restrict a parser or bind it to a type it does not know.

ParserDecorator.withoutTypes(parser, types)

The result claims what the wrapped parser claims minus types.

ParserDecorator.withMimeFilters(parser, include, exclude)

Both at once; returns the parser unchanged when both sets are empty. This is what the JSON directives build (a ParserDecorator.MimeFilteringDecorator).

ParserDecorator.withFallbacks(parsers, types)

Deprecated; use FallbackParser from org.apache.tika.parser.multiple.

new ParserDecorator(parser) subclass

Override parse (and possibly getSupportedTypes) and call super to add behaviour around another parser.

The example binds the plain TXTParser to the order type (for a raw-text debugging view), puts it after DefaultParser in a CompositeParser so that it wins for that type, wraps the whole thing in a decorator that stamps the intake channel on every document, and finally removes PDF support from a parser with withoutTypes:

package demo;

import java.io.IOException;
import java.nio.file.Path;
import java.util.List;
import java.util.Set;

import org.xml.sax.ContentHandler;
import org.xml.sax.SAXException;

import org.apache.tika.exception.TikaException;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.metadata.Property;
import org.apache.tika.metadata.TikaCoreProperties;
import org.apache.tika.mime.MediaType;
import org.apache.tika.mime.MediaTypeRegistry;
import org.apache.tika.parser.AutoDetectParser;
import org.apache.tika.parser.CompositeParser;
import org.apache.tika.parser.DefaultParser;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.parser.Parser;
import org.apache.tika.parser.ParserDecorator;
import org.apache.tika.parser.txt.TXTParser;
import org.apache.tika.sax.BodyContentHandler;

public class Decorators {

    static final Property INTAKE_CHANNEL = Property.internalText("bso:intake-channel");

    /** A decorator subclass: same types, one extra metadata value after every parse. */
    static class ChannelStampingParser extends ParserDecorator {
        private static final long serialVersionUID = 1L;
        private final String channel;

        ChannelStampingParser(Parser parser, String channel) {
            super(parser);
            this.channel = channel;
        }

        @Override
        public void parse(TikaInputStream tis, ContentHandler handler, Metadata metadata,
                          ParseContext context) throws IOException, SAXException, TikaException {
            try {
                super.parse(tis, handler, metadata, context);
            } finally {
                metadata.set(INTAKE_CHANNEL, channel);
            }
        }

        @Override
        public String getDecorationName() {
            return "Intake channel " + channel;
        }
    }

    public static void main(String[] args) throws Exception {
        MediaTypeRegistry registry = MediaTypeRegistry.getDefaultRegistry();

        // 1. withTypes: TXTParser now claims exactly the order type.
        Parser rawOrders = ParserDecorator.withTypes(new TXTParser(), Set.of(OrderParser.ORDER));
        System.out.println("withTypes claims: " + rawOrders.getSupportedTypes(new ParseContext()));

        // The last parser for a type wins inside a CompositeParser.
        Parser composite = new CompositeParser(registry, List.of(new DefaultParser(), rawOrders));
        Parser stamped = new ChannelStampingParser(new AutoDetectParser(composite), "web-upload");

        for (String file : args) {
            Metadata metadata = new Metadata();
            BodyContentHandler handler = new BodyContentHandler(-1);
            try (TikaInputStream tis = TikaInputStream.get(Path.of(file), metadata)) {
                stamped.parse(tis, handler, metadata, new ParseContext());
            }
            List<String> parsedBy = List.of(metadata.getValues(TikaCoreProperties.TIKA_PARSED_BY));
            System.out.println(file + " -> " + parsedBy.get(parsedBy.size() - 1)
                    + ", channel=" + metadata.get(INTAKE_CHANNEL));
        }

        // 2. withoutTypes: the PDF parser stops claiming PDFs.
        Parser noPdf = ParserDecorator.withoutTypes(new DefaultParser(),
                Set.of(MediaType.application("pdf")));
        System.out.println("without PDF supports application/pdf: "
                + noPdf.getSupportedTypes(new ParseContext()).contains(MediaType.application("pdf")));
    }
}

Output for order-0042.bso, legacy-catalogue.txt and catalogue.pdf:

withTypes claims: [application/vnd.bookshelf.order]
files/order-0042.bso -> org.apache.tika.parser.txt.TXTParser, channel=web-upload
files/legacy-catalogue.txt -> org.apache.tika.parser.csv.TextAndCSVParser, channel=web-upload
files/catalogue.pdf -> org.apache.tika.parser.pdf.PDFParser, channel=web-upload
without PDF supports application/pdf: false

The order went to the decorated TXTParser instead of OrderParser, because it came later in the composite; the text file kept its usual 4.x parser (TextAndCSVParser), and every result carries bso:intake-channel. Setting the key in a finally block means it is present even when the inner parse throws, which is what an intake log needs.

Source: ParserDecorator, Javadoc 4.1.0 and CompositeParser, Javadoc 4.1.0.

In tika-config.json: exclusions and MIME filters

The same moves in configuration:

{
  "parsers": [
    {
      "default-parser": {
        "exclude": ["executable-parser", "class-parser"]
      }
    },
    {
      "txt-parser": {
        "_mime-include": ["application/vnd.bookshelf.order"]
      }
    }
  ]
}

  • exclude on default-parser removes whole parsers by component name. The names are the ones in the parsers.idx files (tika-app --list-parser-names prints them, as shown on the Tika App page). Excluding parsers for formats you never accept (here, executables and Java class files for a book intake) shrinks the attack surface.

  • _mime-include and _mime-exclude are framework directives: the loader strips them from the component’s own options and wraps the component in a MIME-filtering ParserDecorator. _mime-include behaves like withTypes: the parser claims exactly the listed types.

  • Configuring a parser explicitly (here txt-parser) removes its copy from default-parser. In this file that means txt-parser handles only orders, and plain text goes to the parser that also claims it (TextAndCSVParser in the run). Check what else you lose when you restrict a parser that default-parser would otherwise have loaded.

Order matters. The test project loaded the file above and the same file with the two entries swapped. With default-parser first, the order went to TXTParser; with txt-parser first, it went to OrderParser from the classpath, because the later default-parser entry registered it after the restricted TXTParser. Put the entries that must win for a type after default-parser.

default-parser, then txt-parser:  order-0042.bso -> org.apache.tika.parser.txt.TXTParser
txt-parser, then default-parser:  order-0042.bso -> demo.OrderParser

The 3.x XML equivalent of an exclusion and a filter, for comparison:

<properties>
  <parsers>
    <parser class="org.apache.tika.parser.DefaultParser">
      <parser-exclude class="org.apache.tika.parser.executable.ExecutableParser"/>
    </parser>
    <parser class="org.apache.tika.parser.txt.TXTParser">
      <mime>application/vnd.bookshelf.order</mime>
    </parser>
  </parsers>
</properties>

Source: Apache Tika 4.1.x, Configuration (the parsers list and default-parser), Apache Tika 4.1.x, Serialization and Configuration (Framework Directives) and Apache Tika 3.3.2, Configuring Tika. The 3.x XML was not run here.

ExternalParser: wiring in a command-line tool

external-parser runs a program for each document and turns its output into metadata, text or both. It is the 4.x replacement for the 3.x tika-external-parsers.xml auto-discovery (with CompositeExternalParser and ExternalParsersFactory), which is gone: in 4.x every external parser is a JSON entry, and nothing is enabled unless you configure it.

How it works

  1. The parser spools the document to a temporary file (tis.getPath()) and substitutes ${INPUT_FILE} in commandLine with its path. If the command contains ${OUTPUT_FILE}, a temporary output file is created and substituted too.

  2. It starts the process with ProcessBuilder, from an argument array, not through a shell, so the file name is never interpreted by a shell.

  3. It reads stdout and stderr up to maxStdOut and maxStdErr bytes and kills the process after timeoutMillis.

  4. It hands each stream to its handler parser (stdoutHandler, stderrHandler, outputFileHandler); the usual handler is regex-capture-parser, which maps regular-expression groups to metadata keys.

  5. contentSource picks which stream becomes the XHTML text: stdout, stderr, outputFile or none (metadata only).

  6. It records external-process:exit-value, external-process:timeout, the stream lengths and whether they were truncated.

Key Default Meaning

commandLine

none

The program and its arguments, one array element each, with ${INPUT_FILE} and optionally ${OUTPUT_FILE}.

supportedTypes

empty

The media types this parser claims (see the warning below about _mime-include).

checkCommandLine

none

A command that proves the tool is installed (["exiftool", "-ver"]). Run lazily, on the first getSupportedTypes call; on failure the parser claims no types.

checkErrorCodes

[127]

Exit codes of the check that mean "not available".

stdoutHandler, stderrHandler, outputFileHandler

none

A parser configuration (usually regex-capture-parser with a captureMap) applied to that stream.

contentSource

stdout, or outputFile when the command uses ${OUTPUT_FILE}

Which stream becomes the text; none for metadata only.

returnStdout, returnStderr

false

Also store the raw stream in metadata.

timeoutMillis

60000

Kill the process after this time.

maxStdOut, maxStdErr

10000

Bytes of each stream to keep.

A runnable example with file(1)

Neither exiftool, ffmpeg nor sox was installed on the host used for this page, but the POSIX file command was, so the run uses it to add a short description and the pixel size of every JPEG or PNG:

{
  "parsers": [
    { "default-parser": {} },
    {
      "external-parser": {
        "supportedTypes": ["image/jpeg", "image/png"],
        "commandLine": ["file", "--brief", "${INPUT_FILE}"],
        "checkCommandLine": ["file", "--version"],
        "checkErrorCodes": [126, 127],
        "timeoutMillis": 10000,
        "maxStdOut": 4096,
        "contentSource": "none",
        "stdoutHandler": {
          "regex-capture-parser": {
            "captureMap": {
              "file:description": "^([^,\\r\\n]+)",
              "file:dimensions": ",\\s*(\\d+x\\d+),\\s*components"
            }
          }
        }
      }
    }
  ]
}

Loaded with TikaLoader.load(path).loadAutoDetectParser() and run on book-cover.jpg:

files/book-cover.jpg -> org.apache.tika.parser.external.ExternalParser
   external-process:exit-value = 0
   external-process:timeout = false
   external-process:stdout-length = 128
   external-process:stdout-truncated = false
   external-process:stderr-length = 0
   external-process:stderr-truncated = false
   file:dimensions = 800x1200
   file:description = JPEG image data
   Content-Type = image/jpeg

The external parser replaced JpegParser for JPEGs (it comes after default-parser, so it wins), which means the tiff: and img: keys of the image parser are gone. If you want both, keep the external parser on a type that no other parser claims, or merge the results yourself with a SupplementingParser; the latter was not tried here.

The official 4.1.x page gives the same structure for exiftool (stdoutHandler), ffmpeg and sox (both write their information to stderr, so they use stderrHandler, returnStderr and a larger maxStdErr). Abridged from those examples:

{
  "parsers": [
    { "default-parser": {} },
    {
      "external-parser": {
        "supportedTypes": ["video/mp4", "video/x-msvideo"],
        "commandLine": ["ffmpeg", "-i", "${INPUT_FILE}"],
        "checkCommandLine": ["ffmpeg", "-version"],
        "checkErrorCodes": [126, 127],
        "contentSource": "none",
        "returnStderr": true,
        "maxStdErr": 20000,
        "stderrHandler": {
          "regex-capture-parser": {
            "captureMap": {
              "xmpDM:duration": "\\s*Duration:\\s*([0-9:\\.]+),.*",
              "xmpDM:audioSampleRate": "\\s*Stream.*:.+Audio:.*,\\s+(\\d+)\\s+Hz,.*"
            }
          }
        }
      }
    }
  ]
}

Source: Apache Tika 4.1.x, External Parser Configuration and ExternalParser, Javadoc 4.1.0. The ffmpeg form was not run (ffmpeg was not installed).

The lazy check and _mime-include

The official examples scope an external parser with _mime-include. In the 4.1.0 test project that defeated the lazy check: with exiftool absent and "_mime-include": ["image/jpeg"], the parse of book-cover.jpg failed with TikaException: TIKA-198: Illegal IOException from ...MimeFilteringDecorator caused by Cannot run program "exiftool". The MIME-filtering decorator answers getSupportedTypes from its own include list, so the external parser’s "tool missing, claim nothing" answer is never consulted. With the same file using "supportedTypes": ["image/jpeg"] instead, the check ran, the external parser claimed nothing, and the JPEG went to JpegParser as intended. Use supportedTypes whenever you rely on checkCommandLine, or make sure the tool is installed everywhere the configuration is deployed.

This is an observed behaviour of 4.1.0 read against its source (ParserDecorator.MimeFilteringDecorator caches the include list), not a documented one; recheck it on later releases.

Security risks

An external parser runs a native program, written in an unsafe language, on input from strangers, with the privileges of the Tika process. That is the riskiest extension on this page:

  • The tool’s own vulnerabilities become yours. ExifTool, FFmpeg and libmagic have all had CVEs for crafted files. Pin and patch the tool like any dependency, and track its advisories, not only Tika’s.

  • Configuration is code execution. Whoever can change commandLine can run any program. Treat tika-config.json as code: read-only, reviewed, never assembled from request data. The JSON allowlist stops a request from instantiating a new parser, but it does not stop a trusted file from naming a dangerous command.

  • No shell, but arguments still matter. ProcessBuilder prevents shell injection through the file name, but a tool can still interpret its arguments: keep ${INPUT_FILE} as a separate array element and never put it inside an option string a tool parses.

  • Bound everything. Keep timeoutMillis, maxStdOut and maxStdErr small. In tika-server and pipes, the parse runs in a forked JVM with its own timeout; a tool that ignores signals can still outlive a parse, so run it in a container with CPU and memory limits.

  • Temporary files. The input is spooled to a temporary file that the tool reads; set java.io.tmpdir to a private directory on a size-limited volume.

  • Least privilege. Run Tika (and therefore the tool) as an unprivileged user without network access unless the tool needs it. The Docker & Kubernetes Deployment and Security pages of this section cover the rest.

Source: Apache Tika, Security model and Apache Tika 4.1.x, Security.

Wiring a custom OCR engine

OCR in 4.1 is not a parser that AutoDetectParser dispatches to. It is a text recognizer: a Parser that the image and PDF parsers invoke on an image they have already parsed, and whose output they place in the document’s text. Tesseract is the built-in one (see OCR with Tesseract) and the VLM engines are others (see VLM Parsers & Inference). If your engine speaks the OpenAI-compatible chat completions API, the existing openai-vlm-parser may already cover it.

The Bookshelf warehouse has a small internal service that reads shelf labels on photographs. This engine sends the image to it over HTTP:

package demo;

import java.io.Serializable;

/** Options of the shelf-label OCR engine; the field names are the JSON keys. */
public class ShelfOcrConfig implements Serializable {

    private static final long serialVersionUID = 1L;

    private String endpoint = "http://localhost:8099/ocr";
    private String apiKey = "";
    private long timeoutMillis = 10_000;
    private long maxBytes = 10_000_000;
    private boolean skipOcr = false;

    public String getEndpoint() { return endpoint; }
    public void setEndpoint(String endpoint) { this.endpoint = endpoint; }
    public String getApiKey() { return apiKey; }
    public void setApiKey(String apiKey) { this.apiKey = apiKey; }
    public long getTimeoutMillis() { return timeoutMillis; }
    public void setTimeoutMillis(long timeoutMillis) { this.timeoutMillis = timeoutMillis; }
    public long getMaxBytes() { return maxBytes; }
    public void setMaxBytes(long maxBytes) { this.maxBytes = maxBytes; }
    public boolean isSkipOcr() { return skipOcr; }
    public void setSkipOcr(boolean skipOcr) { this.skipOcr = skipOcr; }
}
package demo;

import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.Set;

import org.xml.sax.ContentHandler;
import org.xml.sax.SAXException;

import org.apache.tika.annotation.TikaComponent;
import org.apache.tika.config.ConfigDeserializer;
import org.apache.tika.config.JsonConfig;
import org.apache.tika.config.ParseContextConfig;
import org.apache.tika.exception.TikaConfigException;
import org.apache.tika.exception.TikaException;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.mime.MediaType;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.parser.Parser;
import org.apache.tika.parser.enricher.TextRecognizer;
import org.apache.tika.sax.XHTMLContentHandler;

/** Sends images to the Bookshelf shelf-label OCR service and writes its text. */
@TikaComponent(name = "shelf-ocr-parser", spi = false)
public class ShelfOcrParser implements Parser, TextRecognizer {

    private static final long serialVersionUID = 1L;

    private static final Set<MediaType> TYPES =
            Set.of(MediaType.image("jpeg"), MediaType.image("png"));

    private final ShelfOcrConfig defaultConfig;
    private transient HttpClient client;

    public ShelfOcrParser() {
        this(new ShelfOcrConfig());
    }

    public ShelfOcrParser(ShelfOcrConfig config) {
        this.defaultConfig = config;
    }

    public ShelfOcrParser(JsonConfig json) {
        this(ConfigDeserializer.buildConfig(json, ShelfOcrConfig.class));
    }

    @Override
    public Set<MediaType> getSupportedTypes(ParseContext context) {
        return TYPES;
    }

    @Override
    public boolean recognizesText(ParseContext context) {
        return !config(context).isSkipOcr();
    }

    @Override
    public void parse(TikaInputStream tis, ContentHandler handler, Metadata metadata,
                      ParseContext context) throws IOException, SAXException, TikaException {
        ShelfOcrConfig config = config(context);
        if (config.isSkipOcr()) {
            return;
        }
        Path image = tis.getPath();                       // spooled at most once
        if (Files.size(image) > config.getMaxBytes()) {
            return;                                       // hostile input: cap what you send
        }
        HttpRequest request = HttpRequest.newBuilder(URI.create(config.getEndpoint()))
                .timeout(Duration.ofMillis(config.getTimeoutMillis()))
                .header("Authorization", "Bearer " + config.getApiKey())
                .header("Content-Type", "application/octet-stream")
                .POST(HttpRequest.BodyPublishers.ofFile(image))
                .build();
        String text;
        try {
            HttpResponse<String> response =
                    client().send(request, HttpResponse.BodyHandlers.ofString());
            if (response.statusCode() != 200) {
                throw new TikaException("OCR service returned " + response.statusCode());
            }
            text = response.body();
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new TikaException("OCR call interrupted", e);
        }
        XHTMLContentHandler xhtml = new XHTMLContentHandler(handler, metadata);
        xhtml.startDocument();
        xhtml.element("p", text);
        xhtml.endDocument();
    }

    private synchronized HttpClient client() {
        if (client == null) {
            client = HttpClient.newBuilder()
                    .connectTimeout(Duration.ofMillis(defaultConfig.getTimeoutMillis()))
                    .build();
        }
        return client;
    }

    private ShelfOcrConfig config(ParseContext context) {
        try {
            return ParseContextConfig.getConfig(context, "shelf-ocr-parser",
                    ShelfOcrConfig.class, defaultConfig);
        } catch (TikaConfigException | IOException e) {
            return defaultConfig;
        }
    }
}

What makes it an OCR engine rather than a parser:

TextRecognizer

Declares that the engine’s output is the document’s text, so the PDF parser’s automatic OCR strategy may use it in place of a page’s extracted text. An engine whose output only annotates (a captioner, a tagger) implements ContentEnricher instead, and its output never replaces text. Both are new in 4.1 and marked experimental in the 4.1.x guide: 4.2 is expected to batch recognition per document, with the stated intent that a one-image-at-a-time engine like this one keeps working.

recognizesText

Answered per parse; return false when this request asked to skip OCR.

Real media types

Advertise the image types you read (image/png), not the pre-4.1 image/ocr-* routing pseudo-types, which are retired.

The handler

The caller wraps your handler so that your startDocument/endDocument are swallowed and only the body lands, inline with the image’s own output. Write plain text, not headings or tables, and do not set Content-Type.

spi = false

Keeps the engine out of classpath discovery, so it runs only when named. With spi = true and Tesseract also present, the winner would depend on registration order (reported as a WARN at startup).

Bound every call

A timeout on the HTTP call and a size cap on what you send. An engine that runs a local process should extend org.apache.tika.parser.AbstractExternalProcessParser, so that the pipes deadline reaches the process.

Use the caller’s ParseContext

It carries the recursion guard that stops an enricher from enriching its own output. Never create a new one inside parse.

Availability

The official guide suggests implementing org.apache.tika.config.Initializable and returning an empty getSupportedTypes when the service is unreachable at startup, so that a named but dead engine fails the configuration load instead of doing nothing. This example skipped that to stay short; production code should not.

The engine is configured once under engines and named in text-recognizers (the engines + text-recognizers form is new in 4.1.0; naming an engine under parsers is deprecated since 4.1.0 and unsupported in 4.2.0). The API key comes from an environment variable through the 4.1 ${env:NAME} interpolation:

{
  "engines": {
    "shelf-ocr": {
      "shelf-ocr-parser": {
        "endpoint": "http://localhost:8099/ocr",
        "apiKey": "${env:SHELF_OCR_API_KEY}",
        "timeoutMillis": 5000
      }
    }
  },
  "text-recognizers": [
    { "engine": "shelf-ocr" }
  ]
}

A text-recognizers list is authoritative: with it, Tesseract on the classpath is ignored. To keep Tesseract for the types your engine does not read, add it as a second engine with _mime-exclude for the types the first one covers, as the 4.1.x guide shows. In the run of Putting it together: the Bookshelf tika-config.json, with a local stand-in for the service that answered SHELF B-12 Maritime Fiction, the cover photograph came out as:

files/book-cover.jpg detected as image/jpeg
  Content-Type: image/jpeg
  parsed-by:    org.apache.tika.parser.CompositeParser, org.apache.tika.parser.DefaultParser, org.apache.tika.parser.image.JpegParser, demo.ShelfOcrParser
  text:         SHELF B-12 Maritime Fiction

JpegParser did the parse and invoked the engine, and tk:parsed-by records both. With SHELF_OCR_API_KEY unset, the load failed before any document was read, naming the variable and the JSON path but not a value:

org.apache.tika.exception.TikaConfigException: environment variable SHELF_OCR_API_KEY is not set; referenced at /engines/shelf-ocr/shelf-ocr-parser/apiKey

The scanned-PDF path (an engine called for PDF page renders) and the startup log line the guide describes were not checked here; the PDF side is on PDF Parsing.

Source: Apache Tika 4.1.x, Wiring in Your Own OCR Engine, Apache Tika 4.1.x, Configuration (text-recognizers, secrets from the environment) and TextRecognizer, Javadoc 4.1.0.

Adding a metadata key

Every key a 4.x parser writes is either a Property constant or lives under a declared KeyPrefix. The rules, as OrderParser applies them:

  • A fixed key is a Property in your own namespace. Property.internalText("bso:order-number"), Property.internalDate(...), Property.internalInteger(...) and the other public factories create typed keys; declare them once as static final fields. Use kebab case after the prefix, like Tika’s own keys.

  • Never coin tk: or X-TIKA: keys. Those namespaces are reserved for Tika: the public Property factories throw IllegalArgumentException for such a name, and the reserved-key write guard rejects ad-hoc writes to them. Only classes inside org.apache.tika.metadata can mint them, through package-private factories.

  • Reuse a standard key when one fits. The order’s title went to dc:title (TikaCoreProperties.TITLE), so every consumer that reads titles finds it. Coin a key only for what no standard covers.

  • Names that come from the document go under a KeyPrefix. The supplier decides which X- headers exist, so they cannot be constants. KeyPrefix.file("bso:header:", "...") declares the prefix once (for names coined by a tool, use KeyPrefix.tool(...)), and metadata.add(OrderProperties.HEADER, name, value) writes bso:header:Courier. The prefix must end in : or -, must not be reserved, and must never be built from document text or per parse: the registry is static and grows forever. The write route skips blank, over-long or flooding names with a WARN instead of throwing.

  • Inside Tika itself a new key also needs the generated registry under tika-metadata-schema to be regenerated (tika-metadata-schema/regen.sh) and committed, because CI fails on a stale registry. That step is for contributions to Tika, not for keys in your own parser.

Source: Apache Tika 4.1.x, Adding a Metadata Key, Property, Javadoc 4.1.0 and KeyPrefix, Javadoc 4.1.0.

Putting it together: the Bookshelf tika-config.json

One configuration file wires everything on this page except the external parser and the decorator:

{
  "parsers": [
    { "bookshelf-order-parser": { "maxLines": 500, "emitTable": false } },
    {
      "default-parser": {
        "exclude": ["executable-parser", "class-parser"]
      }
    }
  ],
  "detectors": [
    "bookshelf-batch-detector",
    "default-detector"
  ],
  "engines": {
    "shelf-ocr": {
      "shelf-ocr-parser": {
        "endpoint": "http://localhost:8099/ocr",
        "apiKey": "${env:SHELF_OCR_API_KEY}",
        "timeoutMillis": 5000
      }
    }
  },
  "text-recognizers": [
    { "engine": "shelf-ocr" }
  ]
}

Here bookshelf-order-parser comes first, but nothing else claims the order type, so the order does not matter for it; it is configured explicitly only to pass options. The program that loaded it (the stand-in OCR service runs in the same JVM so that the example works offline):

package demo;

import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;

import com.sun.net.httpserver.HttpServer;

import org.apache.tika.config.loader.TikaLoader;
import org.apache.tika.detect.Detector;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.HttpHeaders;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.metadata.TikaCoreProperties;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.parser.Parser;
import org.apache.tika.sax.BodyContentHandler;

public class ConfiguredIntake {
    public static void main(String[] args) throws Exception {
        // Stand-in for the OCR service, so the example runs offline.
        HttpServer fake = HttpServer.create(new InetSocketAddress("localhost", 8099), 0);
        fake.createContext("/ocr", exchange -> {
            exchange.getRequestBody().readAllBytes();
            byte[] body = "SHELF B-12 Maritime Fiction".getBytes(StandardCharsets.UTF_8);
            exchange.sendResponseHeaders(200, body.length);
            try (OutputStream out = exchange.getResponseBody()) {
                out.write(body);
            }
        });
        fake.start();
        try {
            TikaLoader loader = TikaLoader.load(Path.of("tika-config.json"));
            Detector detector = loader.loadDetectors();
            Parser parser = loader.loadAutoDetectParser();
            for (String file : args) {
                Metadata metadata = new Metadata();
                BodyContentHandler handler = new BodyContentHandler(-1);
                try (TikaInputStream tis = TikaInputStream.get(Path.of(file), metadata)) {
                    System.out.println(file + " detected as "
                            + detector.detect(tis, new Metadata(), new ParseContext()));
                    parser.parse(tis, handler, metadata, loader.loadParseContext());
                }
                System.out.println("  Content-Type: " + metadata.get(HttpHeaders.CONTENT_TYPE));
                System.out.println("  parsed-by:    "
                        + String.join(", ", metadata.getValues(TikaCoreProperties.TIKA_PARSED_BY)));
                System.out.println("  text:         " + handler.toString().strip().replace('\n', '|'));
            }
        } finally {
            fake.stop(0);
        }
    }
}

Run with SHELF_OCR_API_KEY=test-only-key (a dummy value; the stand-in ignores it):

files/order-0042.bso detected as application/vnd.bookshelf.order
  Content-Type: application/vnd.bookshelf.order
  parsed-by:    org.apache.tika.parser.CompositeParser, demo.OrderParser
  text:         Purchase order PO-2026-0042|The Lighthouse Keeper x3|Tides of Paper x1
files/batch.zip detected as application/vnd.bookshelf.batch+zip
  Content-Type: application/zip
  parsed-by:    org.apache.tika.parser.CompositeParser, org.apache.tika.parser.DefaultParser, org.apache.tika.parser.pkg.ZipParser
  text:         bookshelf-batch.json|{"titles":2}|||||a.txt|x
files/plain.zip detected as application/zip
  Content-Type: application/zip
  parsed-by:    org.apache.tika.parser.CompositeParser, org.apache.tika.parser.DefaultParser, org.apache.tika.parser.pkg.ZipParser
  text:         a.txt|x
files/book-cover.jpg detected as image/jpeg
  Content-Type: image/jpeg
  parsed-by:    org.apache.tika.parser.CompositeParser, org.apache.tika.parser.DefaultParser, org.apache.tika.parser.image.JpegParser, demo.ShelfOcrParser
  text:         SHELF B-12 Maritime Fiction
files/catalogue.pdf detected as application/pdf
  Content-Type: application/pdf
  parsed-by:    org.apache.tika.parser.CompositeParser, org.apache.tika.parser.DefaultParser, org.apache.tika.parser.pdf.PDFParser
  text:         Bookshelf spring catalogue|Forty titles from independent publishers.

The order shows the JSON options at work (emitTable: false gave paragraphs); because it was configured explicitly, OrderParser hangs directly under the top-level composite instead of under DefaultParser. The batch was detected by the custom detector and parsed by the parent’s ZipParser, as described above.

Source: Apache Tika 4.1.x, Configuration and Apache Tika 4.1.x, Getting Started with the Java API.

Writing a pipes plugin

Fetchers (where documents come from), emitters (where results go), pipes iterators (which documents to process) and reporters (what happened) are not loaded through META-INF/services or @TikaComponent alone. In 4.x they are PF4J plugins: a zip with a plugin.properties descriptor, a org.pf4j.Plugin subclass, one factory per extension type marked with @org.pf4j.Extension and implementing an org.apache.tika.pipes.api factory interface (such as FetcherFactory), and the extension itself. The plugin gets its own class loader, so it must not bundle the Tika pipes API, pf4j or the logging API, which the host provides. A Bookshelf fetcher for its supplier portal would be such a plugin. The Tika Pipes page of this section covers pipes itself; the build, packaging and installation of a plugin are in the official guide linked below. This page did not build one.

Source: Apache Tika 4.1.x, Writing a Pipes Plugin.

From the book to 4.x

Tika in Action (chapter 11, "Extending Tika") extends Tika for an invented prescription format: it adds a media type in custom-mimetypes.xml and writes a parser that pulls the doctor and the patient into metadata. That example lives on, updated, as PrescriptionParser in the tika-example module, a subclass of XMLParser that claims application/x-prescription+xml. The ideas carry over unchanged; the mechanics do not:

Topic 1.x (book) to 3.x 4.x

Base class

AbstractParser, parse(InputStream, ContentHandler, Metadata, ParseContext)

Implement Parser; parse(TikaInputStream, ...). AbstractParser is deprecated for removal.

Registration

Hand-written META-INF/services/org.apache.tika.parser.Parser

@TikaComponent generates the service file and the component index; hand-written files still work for SPI.

Configuration

tika-config.xml, <parser class="..."> with <mime> and <parser-exclude>

tika-config.json, component names, _mime-include/_mime-exclude, exclude on default-parser. Class names are not accepted.

Options

Setter injection from <params> (2.x)

A config bean bound by Jackson through a JsonConfig constructor, per-request overrides through ParseContextConfig.

Detector

detect(InputStream, Metadata)

detect(TikaInputStream, Metadata, ParseContext).

Metadata keys

Any string

Property constants in your namespace or a declared KeyPrefix; tk:/X-TIKA: reserved.

External tools

tika-external-parsers.xml auto-discovery

An explicit external-parser entry in JSON per tool.

OCR engine

A parser for image/ocr-* types (2.x and 3.x)

A TextRecognizer under engines, named in text-recognizers (4.1).

Extra jars for the server

On the class path

tika.extras.dir, or /tika-extras in Docker; PF4J plugins for pipes.

Source: PrescriptionParser, tika-example 4.1.0 and Apache Tika 4.1.x, Migrating to 4.x.

The chapter 11 parser in 4.x form

The book’s parser reads an XML prescription and copies the doctor and patient elements into metadata. In 4.x the same idea is still a subclass of XMLParser that tees the XHTML handler with two ElementMetadataHandler instances. What changes is that the keys are declared Property constants, the supported type is returned for the ParseContext, and @TikaComponent registers the class. For Bookshelf it reads the "reading prescriptions" a librarian attaches to a loan:

package demo;

import java.util.Set;

import org.xml.sax.ContentHandler;

import org.apache.tika.annotation.TikaComponent;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.metadata.Property;
import org.apache.tika.mime.MediaType;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.parser.xml.ElementMetadataHandler;
import org.apache.tika.parser.xml.XMLParser;
import org.apache.tika.sax.TeeContentHandler;

/** Tika in Action, chapter 11, in 4.x form: doctor and patient of an XML prescription as metadata. */
@TikaComponent(name = "bookshelf-prescription-parser")
public class PrescriptionParser extends XMLParser {

    private static final long serialVersionUID = 1L;

    static final String XPD = "http://example.com/2011/xpd";
    public static final MediaType TYPE = MediaType.application("x-prescription+xml");
    public static final Property DOCTOR = Property.internalText("xpd:doctor");
    public static final Property PATIENT = Property.internalText("xpd:patient");

    @Override
    protected ContentHandler getContentHandler(ContentHandler handler, Metadata metadata,
                                               ParseContext context) {
        return new TeeContentHandler(super.getContentHandler(handler, metadata, context),
                new ElementMetadataHandler(XPD, "doctor", metadata, DOCTOR),
                new ElementMetadataHandler(XPD, "patient", metadata, PATIENT));
    }

    @Override
    public Set<MediaType> getSupportedTypes(ParseContext context) {
        return Set.of(TYPE);
    }
}

A sample file and a call to the parser directly (through AutoDetectParser it also needs a custom-mimetypes.xml entry with <root-XML localName="prescription" namespaceURI="http://example.com/2011/xpd"/>, as in Custom MIME types):

<?xml version="1.0" encoding="UTF-8"?>
<xpd:prescription xmlns:xpd="http://example.com/2011/xpd">
  <xpd:doctor>Dr. Elena Marsh</xpd:doctor>
  <xpd:patient>Bookshelf reading room</xpd:patient>
  <xpd:drug>Two chapters before bed</xpd:drug>
</xpd:prescription>
PrescriptionParser parser = new PrescriptionParser();
Metadata metadata = new Metadata();
BodyContentHandler handler = new BodyContentHandler(-1);
try (TikaInputStream tis = TikaInputStream.get(Path.of("loan-prescription.xpd"), metadata)) {
    parser.parse(tis, handler, metadata, new ParseContext());
}

Output, compiled with -proc:full and run against Tika 4.1.0 (the processor also wrote bookshelf-prescription-parser=demo.PrescriptionParser to META-INF/tika/parsers.idx):

xpd:doctor  = Dr. Elena Marsh
xpd:patient = Bookshelf reading room
text        = Dr. Elena Marsh Bookshelf reading room Two chapters before bed

The official tika-example version still passes the key names as plain strings; declaring them as Property constants follows the 4.x key rules in Adding a metadata key.

Source: PrescriptionParser, tika-example on GitHub and Tika in Action, chapter 11.

Memory, performance and robustness

  • Your parser runs on every file of its type, and some of those files are hostile. Bound loops, counts and buffers (maxLines above); prefer streaming over reading the whole stream into memory.

  • Parsers are shared between threads. AutoDetectParser and the loader hand one instance to every request. Keep per-parse state in local variables, keep the options bean read-only after construction, and create lazily built resources (the HttpClient above) in a thread-safe way.

  • Prefer tis.getPath() once to reading twice. It spools to disk at most once; a detector or parser that needs random access should use it instead of buffering.

  • Detectors are on the hot path. A detector runs for every document of every type: do the cheapest possible test first and open the file only when it matters, or make the detector opt-in (spi = false) and name it only where it is needed.

  • External processes and remote services are the slow part. Their timeouts decide your throughput when a tool hangs; in pipes and tika-server, align them with the forked-parse timeout.

  • Isolate risky extensions. A native tool or a parser of your own that may crash or loop belongs in a forked parse (PipesForkParser, tika-server or pipes), not in your application JVM.

Pitfalls

  • The annotation processor did not run. On JDK 23 and later, a provided dependency is not enough; declare annotationProcessorPaths (or -proc:full). Symptom: no META-INF/tika in the jar, "Unknown component name".

  • Type string mismatch. The type in custom-mimetypes.xml, the one returned by getSupportedTypes and the one in _mime-include must be identical, or the parent’s parser wins without an error.

  • Entry order in parsers. The last entry for a type wins; an override placed before default-parser loses to the classpath.

  • Restricting a parser removes it from the defaults. Naming txt-parser with _mime-include means it no longer handles the types it used to.

  • _mime-include hides the external parser lazy check (observed in 4.1.0, see above); use supportedTypes.

  • An external parser replaces the built-in parser for its types. The image keys of JpegParser disappeared when file took over JPEGs.

  • Metadata set after startDocument() is not in the XHTML head.

  • Closing the stream inside parse. The caller owns it; wrapping it in a reader is fine, closing the reader is not.

  • Coining tk: keys. They throw. Use your own prefix.

  • A new ParseContext inside an engine or a decorator. Pass the caller’s one; it carries the recursion guard and the per-request configuration.

  • Secrets in the configuration file. Use ${env:NAME}; the load fails loudly when the variable is missing.

What was verified and what was not

Run against Tika 4.1.0 in the test project, with the outputs shown: the OrderParser (registration through @TikaComponent, dispatch by AutoDetectParser, JSON options and a per-request override), the processor behaviour on JDK 25 with and without annotationProcessorPaths or -proc:full, the generated META-INF/services and META-INF/tika/*.idx files, the named bookshelf-batch-detector in a detectors list, ParserDecorator.withTypes, withoutTypes and a decorator subclass, the exclude and _mime-include JSON forms and the effect of entry order, external-parser with file(1) and regex-capture-parser, the lazy check with supportedTypes versus _mime-include (exiftool absent), the ShelfOcrParser as a text recognizer with a local stand-in service, and the missing-variable failure of ${env:NAME}.

Taken from the 4.1.x documentation or the 4.1.0 source and not run: the exiftool, ffmpeg and sox configurations (the tools were not installed), the Gradle build, tika.extras.dir and the Docker /tika-extras mount, the text recognizer on PDF page renders and its startup log line, the Initializable availability check, the tika-metadata-schema registry step, the 3.x XML forms, and the pipes plugin build.

References