Robustness, Limits & Timeouts

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

Baseline date: 2026-10-12. This page was written against Apache Tika 4.1.0 (the latest release on Maven Central on that date) with the 3.3.x differences noted, Java 21 and later for the examples (Tika 4.x needs Java 17), Spring Boot 4.x on Spring Framework 7 and the Docker image apache/tika:4.1.0. Tika parses formats it did not design, written by programs it does not control, often sent by people it should not trust. Most documents parse in milliseconds. A few, in any large corpus, make a parser allocate gigabytes, loop forever, or take the JVM down with a native crash. This page is about those few: what can go wrong, why a library cannot defend against all of it in-process, how Tika 4.x isolates parsing in forked JVMs, which limits bound the damage a document can do, how the 4.x timeout model decides between "slow" and "hung", how to size the forks, and how to prove your integration survives each failure with MockParser.

Related pages: Parser API & AutoDetectParser (the exceptions a parse throws), Embedded Documents & Recursive Parsing (embedded limits in context), Configuration (where the limits live in tika-config.json), OCR with Tesseract (per-parser timeouts), Tika Pipes (the forking runtime), Tika Server (REST API) (429 and 503), Tika App (Command Line & GUI) (--fork), Docker & Kubernetes Deployment (container limits) and Security (why none of this is a security boundary on its own).

Every Java example that prints output on this page was compiled and run against Tika 4.1.0 on Java 21 on an 8-core macOS machine (tika-core, tika-parsers-standard-package as a pom dependency, tika-pipes-fork-parser, and the tika-core tests jar for MockParser). The failure modes were produced with small MockParser scripts written for this page (a healthy file, a hang, an out-of-memory error, a System.exit, a container with four embedded documents, a container with four slow embedded pages and a deliberately failing embedded document), a 19,576-byte zip-bomb.zip holding 20,000,000 identical characters, and the Bookshelf publisher-email.eml (which carries samples.zip with a manuscript, a catalogue and a cover note). Defaults and key names were read from the 4.1.0 sources of OutputLimits, EmbeddedLimits, TimeoutLimits, ExceptionReporting, PipesConfig, PipesForkParserConfig and PipesResult. The section What was verified and what was not lists what could not be checked.

<dependencies>
    <dependency>
        <groupId>org.apache.tika</groupId>
        <artifactId>tika-core</artifactId>
        <version>4.1.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.tika</groupId>
        <artifactId>tika-parsers-standard-package</artifactId>
        <version>4.1.0</version>
        <type>pom</type>
    </dependency>
    <dependency>
        <groupId>org.apache.tika</groupId>
        <artifactId>tika-pipes-fork-parser</artifactId>
        <version>4.1.0</version>
    </dependency>
    <!-- MockParser, for tests only -->
    <dependency>
        <groupId>org.apache.tika</groupId>
        <artifactId>tika-core</artifactId>
        <version>4.1.0</version>
        <classifier>tests</classifier>
        <scope>test</scope>
    </dependency>
</dependencies>
dependencies {
    implementation(platform("org.apache.tika:tika-bom:4.1.0"))
    implementation("org.apache.tika:tika-core")
    implementation("org.apache.tika:tika-parsers-standard-package")
    implementation("org.apache.tika:tika-pipes-fork-parser")
    testImplementation("org.apache.tika:tika-core:4.1.0:tests")
}

Source: Apache Tika 4.1.x, Pipes from Java and the artifact listing at Maven Central, tika-core 4.1.0 (which publishes tika-core-4.1.0-tests.jar). The Maven form was run; the Gradle form was not.

What can go wrong in-process

A parser is ordinary Java (plus, for some formats, native code or an external program) running inside your JVM. When it misbehaves, it misbehaves with your heap, your threads and your process. The official robustness page puts it bluntly: do not run Tika in the same process as anything whose availability matters, such as an indexer or a search system.

Failure What it looks like What contains it

OutOfMemoryError

A crafted or merely huge file makes a parser build a giant object graph (a PDF image stream, a spreadsheet with millions of cells, a deeply nested structure). The error is not a TikaException; it can strike any thread, and after it the JVM state is suspect.

Process isolation. Limits reduce the likelihood but cannot prevent it.

Infinite loop or hang

A corrupt file sends a parser into a loop that never reads more bytes and never writes output. The calling thread never returns. Thread.interrupt does nothing to a CPU-bound loop.

Process isolation with a stall detector (progressTimeoutMillis). In-process, nothing can stop it (verified below).

Native crash or System.exit

Native libraries (image codecs, some OCR paths) can segfault; buggy code can call System.exit. The whole JVM is gone.

Process isolation only.

Zip bomb

A few kilobytes of archive expand to gigabytes of text or millions of entries.

output-limits (zipBombThreshold, zipBombRatio, maxPackageEntryDepth) and embedded-limits.

XML bomb, deep nesting

Entity expansion or tens of thousands of nested elements.

Hardened XML readers (XXE and entity expansion off), output-limits.maxXmlDepth.

Huge output

A legitimate 2,000-page catalogue produces 30 million characters that your index cannot take.

output-limits.writeLimit or a handler write limit.

Embedded explosion

An e-mail with an archive with thousands of attachments, each with images.

embedded-limits (maxDepth, maxCount).

Metadata flood

A file with thousands of custom properties, or one property with megabytes of value.

standard-metadata-limiter-factory.

Slow external process

Tesseract, readpst or GDAL take minutes, or never return.

Per-parser timeouts (timeoutMillis) inside the total task budget.

Information leak in errors

Exception messages carry file paths, host names or fragments of the document to the caller.

exception-reporting (MESSAGE_REDACTED, REDACTED).

Resource leaks

A parser leaks file handles, temp files or memory over thousands of documents.

Fork recycling (maxFilesProcessedPerProcess), closing every TikaInputStream.

The defences fall into three layers, and you need all three for untrusted input:

  • Limits bound how much a document may produce: characters, embedded documents, nesting, metadata, bytes unpacked. They work in-process and cost almost nothing.

  • Timeouts bound how long a document may take. The total budget is cooperative and partly works in-process; the stall detector needs something outside the parsing thread that can kill it.

  • Process isolation is the only defence against out-of-memory errors, true hangs and native crashes. The parse runs in a forked JVM (a fork); when the fork dies, your application gets a status code and a fresh fork.

Source: Apache Tika 4.1.x, The Robustness of Apache Tika, Apache Tika 4.1.x, Setting Limits and Apache Tika, Security model.

Process isolation options

In 4.x there is exactly one forking mechanism, Tika Pipes, and four ways in. 3.x had several independent ones.

Entry point How it forks Version

PipesForkParser (tika-pipes-fork-parser)

Programmatic: a Closeable you construct once and call parse(...) on. It starts a pool of forked JVMs lazily (one by default), restarts a fork after a crash, and returns a PipesForkResult with a status instead of throwing. Thread-safe.

4.x

Tika Pipes (tika-app -i/-o, the async processor)

Batch: an iterator feeds fetch/emit tuples to numClients forks.

2.x+; reworked in 4.x

tika-server

Every parsing endpoint (/tika, /rmeta, /unpack, /meta, /detect, and /pipes, /async with allowPipes) is served by a fork. Not optional. A dead fork becomes a 503; a full pool a 429 with Retry-After.

4.x (always on)

tika-app --fork

One file in one fork, with --task-timeout and --progress-timeout in milliseconds.

4.x

tika-grpc

A gRPC interface over the same Pipes backend.

4.x

ForkParser (org.apache.tika.fork)

A Parser that serialised the parser and its classloader into a pooled child JVM (setPoolSize, setServerParseTimeoutMillis, setServerPulseMillis, setMaxFilesProcessedPerServer, setJavaCommand). The book’s chapter 9 approach.

1.x to 3.x; removed in 4.x

tika-server watchdog child

1.x opt-in -spawnChild; from 2.x the server forked a child by default and restarted it after an OOM or a timeout (taskTimeoutMillis, default 5 minutes), with --noFork (or <noFork>) to return to the 1.x in-process behaviour. /pipes and /async already used Tika Pipes.

1.x to 3.x; replaced in 4.x by the pipes-backed server, and noFork, maxFiles, maxRestarts, taskPulseMillis and related keys now fail startup

A 3.x ForkParser program, for recognition only (3.x form, not run for this page):

// Tika 3.x only: ForkParser was removed in 4.x
try (ForkParser parser = new ForkParser(ForkParserExample.class.getClassLoader(), new AutoDetectParser())) {
    parser.setPoolSize(4);
    parser.setServerParseTimeoutMillis(60_000);
    parser.setJavaCommand(List.of("java", "-Xmx512m"));
    try (InputStream in = Files.newInputStream(Path.of("catalogue.pdf"))) {
        parser.parse(in, new BodyContentHandler(-1), new Metadata(), new ParseContext());
    }
}

In 4.x, migrate ForkParser to PipesForkParser (below), and the 3.x server flags to the pipes and parse-context sections of tika-config.json. The difference is more than a rename: ForkParser threw exceptions to the caller and streamed SAX events back, while PipesForkParser always returns a result object whose status says which of the outcomes in the next section happened.

Source: ForkParser, Javadoc 3.3.2, Apache Tika wiki, TikaServer (2.x and 3.x era, for -spawnChild and --noFork) and Apache Tika 4.1.x, Migrating tika-server to 4.x.

The life of a forked parse

Whatever the entry point, a forked parse goes through the same states. The client (the parent JVM: your application, tika-server or tika-app) owns a pool of numClients clients; each client owns at most one fork at a time.

stateDiagram-v2 [*] --> NotStarted NotStarted --> Starting: first parse request (lazy start) Starting --> Ready: server successfully started Ready --> Running: parse request sent Running --> Ready: PARSE_SUCCESS, PARSE_SUCCESS_WITH_EXCEPTION or PARTIAL_TIMEOUT Running --> Timeout: no progress for progressTimeoutMillis, or total plus progress elapsed Running --> OutOfMemory: OutOfMemoryError in the fork Running --> Crashed: fork died (System.exit, native crash, OS OOM-killer) Timeout --> Restarting: status TIMEOUT, fork exits with code 17 OutOfMemory --> Restarting: status OOM, fork exits with code 18 Crashed --> Restarting: status UNSPECIFIED_CRASH Ready --> Restarting: maxFilesProcessedPerProcess reached Ready --> NotStarted: idle for socketTimeoutMillis Restarting --> Starting: next parse request Ready --> [*]: close()

The important properties:

  • A crash costs one document, not the application. The client marks the fork for restart, returns a result with a PROCESS_CRASH status (OOM, TIMEOUT or UNSPECIFIED_CRASH) and starts a new fork on the next request.

  • The restart is not free. In the run below a fork with the standard parser package took about 4.5 to 6 seconds to start on an 8-core laptop. The document after a crash pays that time.

  • Content of a crashed document is lost (except an intermediate result, if emitIntermediateResults is on in a pipes batch). A PARTIAL_TIMEOUT, in contrast, keeps everything extracted up to the deadline and keeps the fork.

  • Retry policy depends on the status. OOM and TIMEOUT on a healthy host usually reproduce on the same bytes: quarantine the file. A fork killed from outside (for example by the OS OOM-killer, see Reading a timeout) may well succeed on retry.

The PipesResult.RESULT_STATUS values group into five categories:

Category Statuses Meaning for the caller

SUCCESS

PARSE_SUCCESS, PARSE_SUCCESS_WITH_EXCEPTION, PARSE_EXCEPTION_NO_EMIT, EMIT_SUCCESS, EMIT_SUCCESS_PARSE_EXCEPTION, EMIT_SUCCESS_PASSBACK, EMPTY_OUTPUT, PARTIAL_TIMEOUT

The fork is healthy. Read the metadata list; check the truncation flags and the embedded exceptions.

PROCESS_CRASH

OOM, TIMEOUT, UNSPECIFIED_CRASH

The fork died on this document and is restarted automatically. Record and quarantine.

TASK_EXCEPTION

FETCH_EXCEPTION, EMIT_EXCEPTION, FETCHER_NOT_FOUND, EMITTER_NOT_FOUND, PRESET_NOT_FOUND, PAYLOAD_LIMIT_EXCEEDED

This task failed for a reason outside the parser (input missing, emitter down, result too large for the IPC channel). Continue with the next.

INITIALIZATION_FAILURE

FETCHER_INITIALIZATION_EXCEPTION, EMITTER_INITIALIZATION_EXCEPTION, CLIENT_UNAVAILABLE_WITHIN_MS

A component could not start, or no fork was free in time (tika-server’s 429). Possibly transient.

FATAL

FAILED_TO_INITIALIZE

The configuration is broken. Stop.

PipesForkResult has isSuccess(), isProcessCrash(), isTaskException(), isInitializationFailure() and isFatal() for these categories. PipesForkParser throws PipesForkParserException (with getStatus()) for initialisation and configuration failures, so a try/catch around parse separates "Tika is misconfigured" from "this document is bad".

Source: PipesResult.RESULT_STATUS, Javadoc 4.1.0, PipesForkResult, Javadoc 4.1.0 and Apache Tika 4.1.x, Timeouts. The exit codes 17 and 18 are those logged by the 4.1.0 fork in the run below (exiting: 17, exiting: 18); they are not documented on the 4.1.x site.

PipesForkParser: the Bookshelf intake survives every failure

The Bookshelf intake worker runs every uploaded file through one shared PipesForkParser. The program below feeds it six MockParser scripts in a row: a healthy file, a hang, a healthy file, an out-of-memory error, a System.exit, and a healthy file again. Each script is a small XML document that MockParser executes (see Testing with MockParser).

package bookshelf;

import java.nio.file.Path;
import java.util.List;

import org.apache.tika.config.TimeoutLimits;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.metadata.TikaCoreProperties;
import org.apache.tika.pipes.api.PipesResult;
import org.apache.tika.pipes.fork.PipesForkParser;
import org.apache.tika.pipes.fork.PipesForkParserConfig;
import org.apache.tika.pipes.fork.PipesForkParserException;
import org.apache.tika.pipes.fork.PipesForkResult;

public class ForkIntake {

    public static void main(String[] args) throws Exception {
        Path inbox = Path.of(args[0]);
        List<String> files = List.of("ok.xml", "hang.xml", "ok.xml", "oom.xml", "crash.xml", "ok.xml");

        PipesForkParserConfig config = new PipesForkParserConfig()
                .setTimeoutLimits(new TimeoutLimits(600_000, 5_000)) // 10 min per document, 5 s stall
                .addJvmArg("-Xmx256m")
                .setMaxFilesPerProcess(1_000);

        try (PipesForkParser parser = new PipesForkParser(config)) {          // starts no JVM yet
            for (String name : files) {
                long start = System.nanoTime();
                try (TikaInputStream tis = TikaInputStream.get(inbox.resolve(name))) {
                    PipesForkResult result = parser.parse(tis);
                    long ms = (System.nanoTime() - start) / 1_000_000;
                    PipesResult.RESULT_STATUS status = result.getStatus();
                    System.out.printf("%-10s %-18s %-14s %6d ms  %s%n", name, status, status.getCategory(), ms,
                            outcome(result));
                } catch (PipesForkParserException e) {
                    // configuration or initialisation problem: not the document's fault
                    System.out.println(name + " application error " + e.getStatus() + ": " + e.getMessage());
                }
            }
        }
    }

    static String outcome(PipesForkResult result) {
        if (result.isSuccess()) {
            Metadata m = result.getMetadata();
            return "text=" + result.getContent().strip().replaceAll("\\s+", " ")
                    + (m.get(TikaCoreProperties.TASK_DEADLINE_REACHED) != null ? " (truncated by deadline)" : "");
        }
        if (result.isProcessCrash()) {
            return "fork lost, quarantine: " + result.getMessage().lines().findFirst().orElse("");
        }
        return "failed: " + result.getMessage();
    }
}

Output (log lines removed, long lines wrapped by hand):

ok.xml     PARSE_SUCCESS      SUCCESS          4990 ms  text=A healthy manuscript Chapter one. It was a dark and stormy night.
hang.xml   TIMEOUT            PROCESS_CRASH    5106 ms  fork lost, quarantine: java.lang.RuntimeException:
                                                        Server-side progress timeout: no progress for 5087ms (limit: 5000ms)
ok.xml     PARSE_SUCCESS      SUCCESS          4500 ms  text=A healthy manuscript Chapter one. It was a dark and stormy night.
oom.xml    OOM                PROCESS_CRASH      89 ms  fork lost, quarantine: java.lang.OutOfMemoryError: Java heap space
crash.xml  UNSPECIFIED_CRASH  PROCESS_CRASH    4523 ms  fork lost, quarantine: java.io.EOFException: Stream closed before magic byte
ok.xml     PARSE_SUCCESS      SUCCESS          4425 ms  text=A healthy manuscript Chapter one. It was a dark and stormy night.

The timings tell the story. The first ok.xml pays the lazy start of the first fork. The hang is killed after 5 seconds of silence. Each document after a crash pays a new fork start (about 4.5 seconds here); the oom.xml result arrives in 89 ms because the fork that served the previous ok.xml was warm. The parent log shows the transitions of the state diagram:

INFO  PerClientServerManager pipes-cpu-sizing: hostCores=8, numClients=1, parentReserved=2, autoCap=slice=6,
      heap=user-set in forkedJvmArgs
ERROR [fork] PipesServer exiting: 17
INFO  PerClientServerManager clientId=0: marking server for restart (TIMEOUT)
ERROR [fork] PipesServer exiting: 18
INFO  PerClientServerManager clientId=0: marking server for restart (OOM)
WARN  PipesClient clientId=0 - crash while waiting for server

A first run with -Xmx128m also logged forked JVM max heap is 128 MB, below the 256 MB needed to parse reliably: the fork checks its own heap at startup.

Things to know about PipesForkParser:

  • Build it once and share it; it is thread-safe and each instance owns its forks. close() stops them and deletes its temporary config file.

  • Pass a TikaInputStream.get(Path) when you have a file; a stream without a file is spooled to a temp file first, because the fork reads from the file system (through an internal fetcher with a generated __tika-internal-fetcher-<uuid> id).

  • The default result mode is RMETA (one Metadata per document, content under tk:content); getContent() is the container’s text only. Iterate getMetadataList() for embedded text.

  • PipesForkParserConfig setters: setTimeoutLimits, setWriteLimit, setMaxEmbeddedCount, setEmbeddedLimits, setHandlerType (the default is TEXT), setParseMode, addJvmArg/setJvmArgs, setJavaPath, setMaxFilesPerProcess, setNumClients (expert; default 1), setPluginsDir and setUserConfigPath for a tika-config.json. getPipesConfig() exposes the underlying PipesConfig for the rest.

Source: PipesForkParserExample in tika-example (4.1.0), PipesForkParser, Javadoc 4.1.0 and PipesForkParserConfig, Javadoc 4.1.0.

Limits

Every limit group is an entry in the parse-context section of tika-config.json, loaded into the ParseContext by TikaLoader.loadParseContext(), and read back by each class’s static get(ParseContext), which returns the configured instance or one with the defaults.

parse-context key Class Bounds

embedded-limits

EmbeddedLimits

Depth and count of embedded documents. Unlimited by default.

output-limits

OutputLimits

Extracted characters, XML nesting, package nesting, zip-bomb ratio.

timeout-limits

TimeoutLimits

Total wall-clock budget per task and the stall detector.

unpack-config

UnpackConfig

Total bytes written by ParseMode.UNPACK. Pipes only.

standard-metadata-limiter-factory

StandardMetadataLimiterFactory

Metadata size, field size, key size, values per field, field allow and deny lists.

exception-reporting

ExceptionReporting

How much of an exception is reported, and its length.

The Bookshelf worker configuration with every group set:

{
  "parse-context": {
    "embedded-limits": { "maxDepth": 5, "throwOnMaxDepth": false, "maxCount": 200, "throwOnMaxCount": false },
    "output-limits": {
      "writeLimit": 500000, "throwOnWriteLimit": false,
      "maxXmlDepth": 100, "maxPackageEntryDepth": 10,
      "zipBombThreshold": 1000000, "zipBombRatio": 100
    },
    "timeout-limits": { "totalTaskTimeoutMillis": 300000, "progressTimeoutMillis": 120000, "throwOnDeadline": false },
    "standard-metadata-limiter-factory": {
      "maxTotalBytes": 1048576, "maxFieldSize": 102400, "maxKeySize": 1024, "maxValuesPerField": 100
    },
    "exception-reporting": { "level": "MESSAGE_REDACTED", "maxLength": 10000 }
  },
  "pipes": {
    "numClients": 3,
    "forkedJvmArgs": ["-Xmx2g"]
  }
}

In code, construct the classes and put them on the context; the constructors take the fields in the order shown:

ParseContext context = new ParseContext();
context.set(EmbeddedLimits.class, new EmbeddedLimits(5, false, 200, false));      // maxDepth, throw, maxCount, throw
context.set(OutputLimits.class, new OutputLimits(500_000, false, 100, 10, 1_000_000, 100));
context.set(TimeoutLimits.class, new TimeoutLimits(300_000, 120_000));          // total, progress
context.set(ExceptionReporting.class, new ExceptionReporting(ExceptionReporting.Level.MESSAGE_REDACTED, 10_000));

Source: Apache Tika 4.1.x, Setting Limits and the Javadoc of OutputLimits, EmbeddedLimits and TimeoutLimits (4.1.0). The JSON was checked for key names against the 4.1.0 classes; the full worker file was not loaded as a whole.

output-limits

Setting Default Description

writeLimit

-1 (unlimited)

Maximum characters of text to extract. Extraction stops when reached.

throwOnWriteLimit

false

Throw WriteLimitReachedException at the limit instead of stopping and setting tk:exception:write-limit-reached=true.

maxXmlDepth

100

Maximum XML element nesting depth in the output. Guards against nesting bombs.

maxPackageEntryDepth

10

Maximum depth of nested package entries (a zip in a zip in a zip).

zipBombThreshold

1,000,000

Extracted characters before the zip-bomb ratio check activates.

zipBombRatio

100

Maximum ratio of extracted characters to input bytes read before the parse is aborted as a suspected zip bomb.

Who honours writeLimit. In 4.1.0 the writeLimit and throwOnWriteLimit of OutputLimits are read by handlers built from the context: BasicContentHandlerFactory.newInstance(type, context), WriteOutContentHandler.newInstance(handler, context) and tika-server, which builds its handler factory from the configured output-limits. They are not applied to a handler you construct yourself: in a test on this page, AutoDetectParser with new BodyContentHandler(-1) and OutputLimits with a write limit of 100 returned all 108,000 characters of the mock file and set no flag, both with throwOnWriteLimit true and false. RecursiveParserWrapper takes its limit from the handler factory, not from the context. With PipesForkParser, output-limits.writeLimit in a setUserConfigPath file had no effect either (108,012 characters came back) because the fork parser supplies its own TEXT handler factory with no limit; use PipesForkParserConfig.setWriteLimit(int). The zip-bomb, XML depth and package depth settings, in contrast, are read by AutoDetectParser's SecureContentHandler from the context in every case.

The zip-bomb guard is per stream. The check compares characters written with bytes read from the stream being parsed. The test archive zip-bomb.zip is 19,576 bytes and holds one 20,000,000-character text file:

  • parsed into one BodyContentHandler, the archive aborted at 1,957,915 characters with Suspected zip bomb: 19576 input bytes produced 1957915 output characters (wrapped in a TikaException);

  • parsed with RecursiveParserWrapper (and therefore with /rmeta, Pipes and PipesForkParser, which work in RMETA mode by default), the entry is parsed as its own embedded document from its own 20,000,000-byte stream, the ratio is 1, and all 20,000,028 characters came back without an error.

So in RMETA mode the ratio guard does not catch one highly compressible entry; writeLimit does. Always set a write limit for untrusted input, and treat the zip-bomb ratio as a second line.

embedded-limits

Setting Default Description

maxDepth

-1 (unlimited)

Maximum nesting depth. Recursion stops at the limit; siblings at the current level still parse.

throwOnMaxDepth

false

Throw EmbeddedLimitReachedException at maxDepth instead of continuing and setting tk:exception:embedded-depth-limit-reached=true.

maxCount

-1 (unlimited)

Maximum total embedded documents. Processing stops when reached.

throwOnMaxCount

false

Throw EmbeddedLimitReachedException at maxCount instead of continuing and setting tk:exception:embedded-resource-limit-reached=true.

With maxCount=2 on a container with four embedded documents, the run below returned the container and the first two; the flag is on the container’s metadata. With throwOnMaxCount=true the same two documents were returned, the container carried the flag and tk:exception:container-exception with EmbeddedLimitReachedException: Max embedded count reached: 2, and RecursiveParserWrapper.parse threw it to the caller. The embedded documents page covers the depth limit with a tree diagram: Embedded Documents & Recursive Parsing.

timeout-limits

Setting Default Description

totalTaskTimeoutMillis

3,600,000 (1 hour)

Wall-clock budget for the whole task, embedded documents included. Every per-parser timeout is clipped to what remains of it.

progressTimeoutMillis

120,000 (2 minutes)

How long the task may go silent before it counts as hung. Enforced (the fork killed) only when the parse runs in a forked JVM.

throwOnDeadline

false

Throw EmbeddedLimitReachedException when the total budget is exhausted instead of skipping the remaining embedded documents and reporting PARTIAL_TIMEOUT.

The model behind these three numbers has its own section: The timeout model.

unpack-config

ParseMode.UNPACK (Pipes, tika-app -z / -Z) writes embedded bytes out. maxUnpackBytes (default 10 GiB; -1 unlimited, 0 means zero bytes, not unlimited) caps the total per file. At the limit, extraction stops and the bytes already written are kept; the parse still reports success and the truncation appears only as a WARN in the fork’s log, not in the metadata or the status. Lower it for untrusted input and watch the logs:

{
  "pipes": { "parseMode": "UNPACK" },
  "parse-context": { "unpack-config": { "maxUnpackBytes": 104857600 } }
}

The rest of UnpackConfig (suffix and key-base strategies, Frictionless output) is covered on Embedded Documents & Recursive Parsing. Not run for this page.

standard-metadata-limiter-factory

Configuring a MetadataWriteLimiterFactory makes Metadata.newInstance(parseContext) return a Metadata that filters every write:

Setting Default Description

maxTotalBytes

10 MB

Total estimated size of all metadata in UTF-16 bytes. Further metadata is dropped and tk:warn:truncated-metadata is set.

maxFieldSize

100 KB

Maximum size of one field’s values. Longer values are truncated.

maxKeySize

1024

Maximum key length in UTF-16 bytes.

maxValuesPerField

10

Maximum values on a multi-valued field. Extra values are dropped.

includeFields / excludeFields

empty

Allow and deny lists.

includeEmpty

false

Whether to store empty values.

A fixed set of fields (Content-Type, tk:resource-name, tk:content, tk:parsed-by, tk:exception:container-exception, tk:exception:embedded-exception and a few more) bypasses the allow and deny lists and the total budget, because parser dispatch and error reporting depend on them. The metadata page explains the limiter with the rest of the key model: Metadata. Not run for this page.

exception-reporting

Exception text travels to the caller in tk:exception:container-exception, tk:exception:embedded-exception, tk:exception:warn, tk:exception:embedded-stream-exception, in pipes result messages and in tika-server 422 and 500 bodies. Stack traces and messages can carry file paths, host names and pieces of the document.

Setting Default Description

level

FULL

FULL: complete stack trace. MESSAGE_REDACTED: the trace with every exception message removed. REDACTED: exception class names only, no frames.

maxLength

-1 (unlimited)

Cut the formatted exception at this many characters and append ...[truncated].

An embedded document whose parser throws NullPointerException("cannot read /srv/bookshelf/inbox/loan-42.pdf") reported, in tk:exception:embedded-exception of that document:

FULL (2,104 characters):
org.apache.tika.exception.TikaException: Unexpected RuntimeException from org.apache.tika.parser.mock.MockParser@38d5b107
	at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:378)
	...
Caused by: java.lang.NullPointerException: cannot read /srv/bookshelf/inbox/loan-42.pdf

MESSAGE_REDACTED, maxLength 200 (214 characters):
org.apache.tika.exception.TikaException
	at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:378)
	at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:359)
	at org.ap...[truncated]

REDACTED (82 characters):
org.apache.tika.exception.TikaException
Caused by: java.lang.NullPointerException

The policy is loaded from the configuration only and is rejected in a per-request parse-context (tika-server answers 400, tika-grpc INVALID_ARGUMENT), so a caller cannot switch redaction off. A third-party parser that still calls the deprecated two-argument EmbeddedDocumentUtil.recordException reports FULL whatever the policy says. In 4.x this replaces the 3.x server option returnStackTrace.

Source: Apache Tika 4.1.x, Setting Limits (exception reporting, metadata limits, unpack limits) and ExceptionReporting, Javadoc 4.1.0.

The limits in-process, run

The program below exercises the limits that work without a fork, with a shared AutoDetectParser and RecursiveParserWrapper.

package bookshelf;

import java.nio.file.Path;

import org.apache.tika.config.EmbeddedLimits;
import org.apache.tika.config.ExceptionReporting;
import org.apache.tika.config.OutputLimits;
import org.apache.tika.config.TimeoutLimits;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.metadata.TikaCoreProperties;
import org.apache.tika.parser.AutoDetectParser;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.parser.RecursiveParserWrapper;
import org.apache.tika.sax.BasicContentHandlerFactory;
import org.apache.tika.sax.BasicContentHandlerFactory.HANDLER_TYPE;
import org.apache.tika.sax.BodyContentHandler;
import org.apache.tika.sax.RecursiveParserWrapperHandler;

public class InProcessLimits {

    static final AutoDetectParser PARSER = new AutoDetectParser();
    static final RecursiveParserWrapper WRAPPER = new RecursiveParserWrapper(PARSER);

    public static void main(String[] args) throws Exception {
        Path inbox = Path.of(args[0]);

        // A. OutputLimits.writeLimit is honoured by handlers built from the context
        ParseContext context = new ParseContext();
        context.set(OutputLimits.class, new OutputLimits(60, false, 100, 10, 1_000_000, 100));
        rmeta("A writeLimit=60 throw=false", inbox.resolve("bundle.xml"), context,
                BasicContentHandlerFactory.newInstance(HANDLER_TYPE.TEXT, context));

        // B. the same limit, throwing
        context = new ParseContext();
        rmeta("B writeLimit=60 throw=true", inbox.resolve("bundle.xml"), context,
                new BasicContentHandlerFactory(HANDLER_TYPE.TEXT, 60, true, context));

        // C. at most two embedded documents
        context = new ParseContext();
        context.set(EmbeddedLimits.class, new EmbeddedLimits(5, false, 2, false));
        rmeta("C maxCount=2", inbox.resolve("bundle.xml"), context,
                new BasicContentHandlerFactory(HANDLER_TYPE.TEXT, -1));

        // D. the zip-bomb ratio: 19,576 compressed bytes, 20,000,000 characters inside
        try (TikaInputStream tis = TikaInputStream.get(inbox.resolve("zip-bomb.zip"))) {
            PARSER.parse(tis, new BodyContentHandler(-1), new Metadata(), new ParseContext());
        } catch (Exception e) {
            System.out.println("D zip " + e.getClass().getSimpleName() + ": " + e.getCause().getMessage());
        }

        // E. redacted exception reporting
        context = new ParseContext();
        context.set(ExceptionReporting.class, new ExceptionReporting(ExceptionReporting.Level.REDACTED, -1));
        rmeta("E REDACTED", inbox.resolve("npe-embedded.xml"), context,
                new BasicContentHandlerFactory(HANDLER_TYPE.TEXT, -1));

        // F. a 3-second total budget, in-process, against four 2-second embedded pages
        context = new ParseContext();
        context.set(TimeoutLimits.class, new TimeoutLimits(3_000, 3_000));
        long start = System.nanoTime();
        Metadata metadata = new Metadata();
        BodyContentHandler text = new BodyContentHandler(-1);
        try (TikaInputStream tis = TikaInputStream.get(inbox.resolve("slow-bundle.xml"))) {
            PARSER.parse(tis, text, metadata, context);
        }
        System.out.println("F deadline after " + (System.nanoTime() - start) / 1_000_000 + " ms, "
                + TikaCoreProperties.TASK_DEADLINE_REACHED.getName() + "="
                + metadata.get(TikaCoreProperties.TASK_DEADLINE_REACHED)
                + ", text=[" + text.toString().strip().replace('\n', ' ') + "]");

        // G. the same budget against a 5-second in-JVM hang: nothing stops it
        start = System.nanoTime();
        try (TikaInputStream tis = TikaInputStream.get(inbox.resolve("hang5.xml"))) {
            PARSER.parse(tis, new BodyContentHandler(-1), new Metadata(), context);
        }
        System.out.println("G hang returned after " + (System.nanoTime() - start) / 1_000_000 + " ms");
    }

    static void rmeta(String label, Path file, ParseContext context, BasicContentHandlerFactory factory)
            throws Exception {
        RecursiveParserWrapperHandler handler = new RecursiveParserWrapperHandler(factory);
        try (TikaInputStream tis = TikaInputStream.get(file)) {
            WRAPPER.parse(tis, handler, new Metadata(), context);
        } catch (Exception e) {
            System.out.println(label + " threw " + e.getClass().getSimpleName() + ": " + e.getMessage());
        }
        System.out.println(label + ": " + handler.getMetadataList().size() + " documents");
        for (Metadata m : handler.getMetadataList()) {
            String content = m.get(TikaCoreProperties.TIKA_CONTENT);
            String problem = m.get(TikaCoreProperties.EMBEDDED_EXCEPTION);
            System.out.printf("  %-8s write-limit=%-5s embedded-limit=%-5s text=%s%s%n",
                    m.get(TikaCoreProperties.RESOURCE_NAME_KEY) == null ? "(root)" : m.get(TikaCoreProperties.RESOURCE_NAME_KEY),
                    m.get(TikaCoreProperties.WRITE_LIMIT_REACHED),
                    m.get(TikaCoreProperties.EMBEDDED_RESOURCE_LIMIT_REACHED),
                    content == null ? "null" : "[" + content.strip().replace('\n', ' ') + "]",
                    problem == null ? "" : "\n           embedded-exception=" + problem.replace("\n", " | "));
        }
    }
}

Output:

A writeLimit=60 throw=false: 5 documents
  (root)   write-limit=true  embedded-limit=null  text=[Container text. e1.xml]
  e1.xml   write-limit=null  embedded-limit=null  text=[first embedded]
  e2.xml   write-limit=null  embedded-limit=null  text=null
  e3.xml   write-limit=null  embedded-limit=null  text=null
  e4.xml   write-limit=null  embedded-limit=null  text=null
B writeLimit=60 throw=true: 2 documents
  (root)   write-limit=true  embedded-limit=null  text=[Container text. e1.xml]
  e1.xml   write-limit=true  embedded-limit=null  text=[first embedded]
C maxCount=2: 3 documents
  (root)   write-limit=null  embedded-limit=true  text=[Container text. e1.xml  e2.xml]
  e1.xml   write-limit=null  embedded-limit=null  text=[first embedded document text]
  e2.xml   write-limit=null  embedded-limit=null  text=[second embedded document text]
D zip TikaException: Suspected zip bomb: 19576 input bytes produced 1957915 output characters
E REDACTED: 2 documents
  (root)   write-limit=null  embedded-limit=null  text=[Container text. broken.xml]
  broken.xml write-limit=null  embedded-limit=null  text=null
           embedded-exception=org.apache.tika.exception.TikaException | Caused by: java.lang.NullPointerException |
F deadline after 4011 ms, tk:exception:task-deadline-reached=true, text=[Scanned loan forms, one per embedded page. page1.xml page one   page2.xml page two]
G hang returned after 5005 ms

Read A and B side by side, because the write limit in RecursiveParserWrapper is one budget for the whole tree, not per document:

  • throwOnWriteLimit=false keeps walking: every embedded document still gets a metadata entry, but once the budget is spent their content is null, and only the container carries tk:exception:write-limit-reached=true.

  • throwOnWriteLimit=true stops the walk: documents after the one that hit the limit are missing from the list altogether (here e2 to e4), and the flag is set on the container and on the document that hit the limit. The exception does not reach the caller; RecursiveParserWrapper records it as the flag.

In both modes the list no longer tells you how many documents the file really contains, so a write-limited result is not an inventory. If you need the inventory, parse once with a write limit for the text and use maxCount rather than the write limit to bound the tree.

The tests for this page found no PropertyTypeException for tk:exception:write-limit-reached on embedded documents with Tika 4.1.0, although RecursiveParserWrapper runs with a small positive write limit could be expected to raise one. RecursiveParserWrapper does call metadata.add on that single-valued key for embedded documents, which would throw if the key were already set on the same Metadata, but in 90 combinations (write limits 5 to 400 characters, TEXT, MARKDOWN and XML handlers, the two-argument factory with its default throwOnWriteLimitReached, and throwOnWriteLimit true and false) over a nested three-level mock file, the four-document bundle and publisher-email.eml (e-mail, zip, DOCX with an embedded PNG, PDF and text file), every parse completed and set the flag as described above. If you meet that exception, the file shape that triggers it was not covered here; catch it around the parse and report it upstream.

Source: BasicContentHandlerFactory, Javadoc 4.1.0, SecureContentHandler, Javadoc 4.1.0 and Apache Tika wiki, RecursiveParserWrapper (3.x era, for the list-of-metadata design).

Detecting truncation

A limit that fires silently is worse than no limit, because the index then holds half a document that looks whole. Check these signals on every result and route flagged documents to a review queue or a slower lane:

Signal Where Set when

tk:exception:write-limit-reached=true

container (and, when throwing, the document that hit it)

writeLimit reached

tk:exception:embedded-resource-limit-reached=true

container

maxCount reached

tk:exception:embedded-depth-limit-reached=true

the document whose children were skipped

maxDepth reached

tk:exception:task-deadline-reached=true and status PARTIAL_TIMEOUT

container; pipes status

totalTaskTimeoutMillis exhausted, remaining embedded documents skipped

tk:warn:truncated-metadata=true

any document

the metadata limiter dropped or cut a value

tk:exception:embedded-exception with a TikaTimeoutException

the embedded document

a per-parser timeout fired; its message says requested and granted

external-process:timeout=true

the embedded document

an ExternalParser-style parser recorded a timed-out process (ExternalProcess.IS_TIMEOUT)

status TIMEOUT, OOM, UNSPECIFIED_CRASH

pipes status; tika-server 503

the fork was lost, no content

WARN in the fork log

log only

maxUnpackBytes reached; nothing in metadata or status

The Java constants are TikaCoreProperties.WRITE_LIMIT_REACHED, EMBEDDED_RESOURCE_LIMIT_REACHED, EMBEDDED_DEPTH_LIMIT_REACHED, TASK_DEADLINE_REACHED and TRUNCATED_METADATA. In 3.x the write-limit flag was X-TIKA:EXCEPTION:write_limit_reached; LegacyKeyMigrationFilter maps the 4.x names back if downstream code still expects them (see Metadata).

A compact check for a PipesForkResult:

static boolean isComplete(PipesForkResult result) {
    if (!result.isSuccess() || result.getStatus() == PipesResult.RESULT_STATUS.PARTIAL_TIMEOUT) {
        return false;
    }
    for (Metadata m : result.getMetadataList()) {
        for (Property flag : List.of(TikaCoreProperties.WRITE_LIMIT_REACHED,
                TikaCoreProperties.EMBEDDED_RESOURCE_LIMIT_REACHED, TikaCoreProperties.EMBEDDED_DEPTH_LIMIT_REACHED,
                TikaCoreProperties.TASK_DEADLINE_REACHED, TikaCoreProperties.TRUNCATED_METADATA)) {
            if ("true".equals(m.get(flag))) {
                return false;
            }
        }
    }
    return true;
}

This helper was compiled against 4.1.0 but not run; the flags it reads are the ones printed in the runs on this page.

Source: the TikaCoreProperties constants in the 4.1.0 tika-core source and Apache Tika 4.1.x, Setting Limits.

The timeout model

Tika 4.0 replaced the per-parser, ad hoc timeouts of 3.x with one model (TimeoutLimits plus the per-task ParseTimeout state) shared by library use, tika-app --fork and Tika Pipes. Three settings bound a parse, and each answers a different question:

Setting Question Default

totalTaskTimeoutMillis

How long may this document take, end to end, embedded documents included?

3,600,000 (1 hour)

progressTimeoutMillis

How long may the parse go silent before it counts as hung?

120,000 (2 minutes)

per-parser timeoutMillis (Tesseract, Tess4J, strings, LibPst, VLM, inference, ExternalParser, DWG)

How long may one call to an external process or service take?

parser-specific, for example 120,000 for Tesseract and 600,000 for LibPst

How they compose:

  • A per-parser timeout is always honoured, but clipped. The effective budget is the smaller of the configured value and the time left in the document’s total budget.

  • An external call in progress counts as progress. While Tika waits on a bounded external process, the wait reports progress periodically, so a legitimate 10-minute readpst run does not need progressTimeoutMillis raised to 10 minutes. The stall detector fires only on genuine silence: an in-JVM hang, or a wedged kill.

  • Budgets compose recursively. A PDF inside a zip inside an e-mail draws every operation from the one total budget of the top-level task.

  • Monotonic clock. All three use the JVM’s monotonic clock; NTP steps, DST changes and laptop sleep do not count.

Cooperative deadline and hard watchdog

There are two families of outcome:

  • The document survives. When the total budget runs out, the remaining embedded documents are skipped cleanly, the container gets tk:exception:task-deadline-reached=true, and the pipes status is PARTIAL_TIMEOUT. The content is complete up to the deadline and the fork stays up. An embedded document whose external call timed out carries a recorded TikaTimeoutException; the status is PARSE_SUCCESS_WITH_EXCEPTION or EMIT_SUCCESS_PARSE_EXCEPTION.

  • The fork is lost. A genuine hang forces the fork to exit; the client restarts it and the status is TIMEOUT.

The deadline is cooperative: Tika checks it at embedded-document boundaries and before external calls. A hard external watchdog in the client backs it up, but deliberately late: it does not fire until totalTaskTimeoutMillis plus progressTimeoutMillis has elapsed, so that the cooperative path has room to skip the rest, filter and emit its PARTIAL_TIMEOUT result. A wind-down that itself hangs is caught by the ordinary stall detector.

PipesForkParser with a 3-second budget against four embedded pages of about 2 seconds each, and then with a write limit of 60 characters and at most two embedded documents:

package bookshelf;

import java.nio.file.Path;

import org.apache.tika.config.TimeoutLimits;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.metadata.TikaCoreProperties;
import org.apache.tika.pipes.fork.PipesForkParser;
import org.apache.tika.pipes.fork.PipesForkParserConfig;
import org.apache.tika.pipes.fork.PipesForkResult;

public class ForkLimits {

    public static void main(String[] args) throws Exception {
        Path inbox = Path.of(args[0]);

        // 1. A 3-second document budget against four slow embedded pages (about 2 s each)
        PipesForkParserConfig deadline = new PipesForkParserConfig()
                .setTimeoutLimits(new TimeoutLimits(3_000, 3_000)); // equal values: stall detector inert
        run(deadline, inbox.resolve("slow-bundle.xml"));

        // 2. A 60-character write limit and at most 2 embedded documents
        PipesForkParserConfig limits = new PipesForkParserConfig()
                .setWriteLimit(60)
                .setMaxEmbeddedCount(2);
        run(limits, inbox.resolve("bundle.xml"));
    }

    static void run(PipesForkParserConfig config, Path file) throws Exception {
        try (PipesForkParser parser = new PipesForkParser(config);
             TikaInputStream tis = TikaInputStream.get(file)) {
            PipesForkResult result = parser.parse(tis);
            System.out.println(file.getFileName() + " -> " + result.getStatus()
                    + ", " + result.getMetadataList().size() + " documents");
            for (Metadata m : result.getMetadataList()) {
                System.out.printf("  %-16s deadline=%-5s write-limit=%-5s embedded-limit=%-5s text=[%s]%n",
                        m.get(TikaCoreProperties.RESOURCE_NAME_KEY),
                        m.get(TikaCoreProperties.TASK_DEADLINE_REACHED),
                        m.get(TikaCoreProperties.WRITE_LIMIT_REACHED),
                        m.get(TikaCoreProperties.EMBEDDED_RESOURCE_LIMIT_REACHED),
                        m.get(TikaCoreProperties.TIKA_CONTENT).strip().replace('\n', ' '));
            }
        }
    }
}

Output:

slow-bundle.xml -> PARTIAL_TIMEOUT, 3 documents
  slow-bundle.xml  deadline=true  write-limit=null  embedded-limit=null  text=[Scanned loan forms, one per embedded page. page1.xml  page2.xml]
  page1.xml        deadline=null  write-limit=null  embedded-limit=null  text=[page one]
  page2.xml        deadline=null  write-limit=null  embedded-limit=null  text=[page two]
bundle.xml -> PARSE_SUCCESS, 2 documents
  bundle.xml       deadline=null  write-limit=true  embedded-limit=null  text=[Container text. e1.xml]
  e1.xml           deadline=null  write-limit=true  embedded-limit=null  text=[first embed]

Pages 3 and 4 were never started; the fork stayed up. In the second run the write limit, which the fork parser applies in throwing mode, ended the walk before the embedded count limit could matter. An earlier run with progressTimeoutMillis 10,000 above the 3,000 total logged, from the fork: progressTimeoutMillis (10000) > totalTaskTimeoutMillis (3000) -- the stall detector can never fire; set them equal to disable it deliberately.

In-process, runs F and G of the previous program show the other half of the model. The total budget still works at embedded boundaries: the four-page bundle stopped after 4,011 ms with the deadline flag, because the second page was already running when the budget ran out. But a 5-second in-JVM hang under the same 3-second budget returned after 5,005 ms: no boundary was crossed, and nothing outside the thread could kill it. That is the case the stall detector and a fork exist for.

throwOnDeadline

With throwOnDeadline: true, the moment the total budget is exhausted every embedded document that would be skipped throws EmbeddedLimitReachedException (unchecked) instead. In library use it propagates to your call site. In Pipes the fork catches it, records it in tk:exception:container-exception, stops parsing that document and reports PARTIAL_TIMEOUT, the same status as the default. The documented difference is that the parse stops at once and carries an explicit exception rather than winding down. In a run for this page with PipesForkParserConfig.setTimeoutLimits and setThrowOnDeadline(true), the result was identical to the default run (PARTIAL_TIMEOUT, three documents) and no container exception was recorded; whether the flag reaches the fork through PipesForkParserConfig was not investigated further. Set it in the tika-config.json parse-context if you rely on it, and test.

Reading a timeout

Every per-call timeout message reports the requested budget, the granted budget and which limit did the clipping:

requested=120000ms, granted=120000ms -- budget exhausted        <- the parser's own timeout was binding
requested=120000ms, granted=30000ms (task remaining)            <- the document budget was binding
Observation Action

requested=N, granted=N on an embedded document

Raise that parser’s timeoutMillis, or accept losing that part

PARTIAL_TIMEOUT, tk:exception:task-deadline-reached, children clipped by task remaining

Raise totalTaskTimeoutMillis, or accept truncation and route the document to a slow lane

TIMEOUT, "no progress", reproduces at the same spot

A parser hang: exclude the file, report the bug

TIMEOUT, "no progress", on healthy documents, correlated with size

progressTimeoutMillis too low for long in-process stretches; raise it

TIMEOUT right after an external-call timeout, zombie child processes on the host

An unkillable child (NFS stall, broken binary): an environment problem, no setting helps

TIMEOUT with a client socket timeout and no server messages

The fork died from outside (OS OOM-killer): raise -Xmx or host memory; a retry may succeed

Source: Apache Tika 4.1.x, Timeouts (the seven scenario walkthroughs and the diagnostic table are summarised here).

Misconfiguration the loader catches

Setting Handling Note

Negative totalTaskTimeoutMillis or progressTimeoutMillis

Rejected

At config load; Long.MAX_VALUE is the unbounded sentinel.

progressTimeoutMillis: 0 with a positive total

Rejected

The stall detector would kill every task at once.

pipes.socketTimeoutMillis not greater than pipes.heartbeatIntervalMillis

Rejected

The client would kill healthy forks between heartbeats.

Pre-4.0 names such as timeoutSeconds

Rejected

Unknown fields fail loudly.

progressTimeoutMillis at or above totalTaskTimeoutMillis

Warned

Stall detection is effectively off; this is the supported way to disable it.

Either value above zero and below one second

Warned

Almost always a seconds-for-milliseconds mistake.

Zero total

Accepted

The budget is already spent; useful in tests only. Zero does not disable a timeout.

There is no "off" switch: to disable a bound, use Long.MAX_VALUE (9223372036854775807). Do not set progressTimeoutMillis to Long.MAX_VALUE, which also removes the client’s backstop for a wedged fork; set it equal to the total instead.

Per-request overrides and the server cap

With allowPerRequestConfig: true, a tika-server or pipes request can carry its own timeout-limits in the parse context. Lowering is free; raising is capped at pipes.maxTotalTaskTimeoutMillis (default 1 hour), with a WARN in the fork log, so a client cannot disable the server’s self-protection with an enormous timeout. Limits in the server’s own parse-context are trusted and never clamped. See Tika Server (REST API) for the endpoints.

On the command line

# one file in a fork, 10 minutes total, 60 seconds of silence allowed
java -jar tika-app.jar --fork --task-timeout=600000 --progress-timeout=60000 catalogue.pdf

# pipes mode: --fork does not apply; the timeouts come from tika-config.json
java -jar tika-app.jar --config=tika-config.json -i /bookshelf/inbox -o /bookshelf/extracted

--fork-timeout is gone and passing it is a hard error that names the two replacements. Without --fork, the two timeout flags are parsed but have nothing to apply to. These commands were taken from the 4.1.x Timeouts page and were not run for this page; the CLI is "in flux" in 4.x, as Tika App (Command Line & GUI) explains.

Source: Apache Tika 4.1.x, Timeouts (CLI usage, per-request overrides, misconfiguration handling).

3.x to 4.x timeout renames

None of these changes shows up as a compile error, and the unit changes do not show up as a config error either.

3.x 4.x Trap

tika-server taskTimeoutMillis (default 300,000)

parse-context.timeout-limits.totalTaskTimeoutMillis (default 3,600,000)

The default grew from 5 minutes to 1 hour. Set 300,000 to keep 3.x behaviour.

tika-server taskPulseMillis, minimumTimeoutMillis

removed

Delete them; unknown keys fail startup.

none

progressTimeoutMillis

New axis: the stall detector. Set it deliberately.

timeoutSeconds (Tesseract, Tess4J, strings, VLM, inference, image embedding: 120; LibPst: 600)

timeoutMillis (120,000; 600,000)

Name and unit changed: renaming "timeoutSeconds": 300 to "timeoutMillis": 300 gives a 300 ms timeout.

PipesConfig socketTimeoutMs, startupTimeoutMs, heartbeatIntervalMs

socketTimeoutMillis, startupTimeoutMillis, heartbeatIntervalMillis

Rename only; already milliseconds.

async CLI --timeoutMs

--timeoutMillis

Rename only.

DWG dwgReadTimeout, ExternalParser timeoutMs

timeoutMillis

Rename only.

tika-app --fork-timeout

--task-timeout, --progress-timeout

Old flag is a hard error.

TikaTimeoutException unchecked

TikaTimeoutException checked (extends TikaException)

A catch (RuntimeException e) around parse silently stops catching it.

raising progressTimeoutMillis to let slow OCR finish

raise the per-parser timeoutMillis or the total

The stall detector no longer caps a single external call.

ProcessUtils.checkCommand() probe: 60 s

5 s

A slow binary probe (antivirus, network file system) can now fail; use checkCommandWithTimeout.

Source: Apache Tika 4.1.x, Timeouts, Upgrading from Tika 3.x and Apache Tika 4.1.x, Migrating tika-server to 4.x.

CPU and heap sizing for forks

Each fork is a full JVM that sizes its garbage collector, JIT compiler and common ForkJoinPool from the CPU count it sees, and its default heap from the memory it sees. Four forks on a 16-core host would otherwise each start about 16 GC threads for the same 16 cores, and together claim far more heap than the host has. Tika Pipes therefore injects two JVM flags into every fork in per-client mode, unless you set them yourself in forkedJvmArgs:

  • -XX:ActiveProcessorCount=<slice>, with the slice computed as:

    slice = (hostCores - 2) / numClients        the parent keeps 2 cores
slice is clamped to at least 2              1 parser thread + 1 for GC, JIT, heartbeat, socket I/O

  • -XX:MaxRAMPercentage=75/numClients (60 for a single fork), when no -Xmx, -XX:MaxRAMPercentage or -XX:MaxRAMFraction is set.

The parent logs its decision once at startup; in the run above, with -Xmx256m in the arguments:

pipes-cpu-sizing: hostCores=8, numClients=1, parentReserved=2, autoCap=slice=6, heap=user-set in forkedJvmArgs
Host cores numClients Slice Notes (from the 4.1.x sizing page)

4

1

2

Comfortable single-fork deployment.

8

3

2

Sweet spot for medium pods.

16

4

3

Sweet spot for 16-core hosts; the measured winner in the project’s benchmarks.

16

8

1, clamped to 2

Does not fit; keep 4 or 6.

32

8

3

Same shape as 16 and 4.

The two rules to check by hand:

  • CPU: the largest numClients for which two cores per fork plus two for the parent fit in the host cores. Beyond that, adding forks hurts throughput, and the parent logs a WARN.

  • Memory: numClients times the per-fork heap must stay within 75% of host or container memory. The parent warns if an explicit -Xmx breaks this; each fork warns at startup if its heap is below 256 MB (seen above with -Xmx128m).

A reasonable starting point is about 2 GB of heap per fork (-Xmx2g): a few pathological PDFs in any large corpus allocate hundreds of megabytes, embedded-heavy Office files multiply that, and G1 behaves poorly above about 85% occupancy. To right-size, measure the post-GC live heap with -Xlog:gc* and pick -Xmx at about 1.5 times the peak. On a 16 GB node with numClients 4: 4 × 2 GB plus about 1 GB for the parent, about 9 GB in all.

Further points:

  • Containers: on JDK 17 and later Runtime.availableProcessors() honours cgroup CPU limits. A pod without resources.limits.cpu sees the node’s full core count, so always set CPU limits on pods that run forks (see Docker & Kubernetes Deployment).

  • Lazy start and idle shutdown: numClients is a ceiling. Forks start on demand, the most recently used one is reused first, and a fork idle for socketTimeoutMillis (default 60 s) exits and restarts on next use.

  • Several pipes groups in one JVM (tika-server with /async beside the other endpoints, or two PipesForkParser instances) each size themselves as if alone. Set -XX:ActiveProcessorCount yourself with the combined total in mind.

  • Shared-server mode (useSharedServer: true) runs all clients in one fork to save memory; one crash, OOM or timeout then affects every in-flight request. Use it only when memory forces you to.

  • Recycling: maxFilesProcessedPerProcess (default 10,000; setMaxFilesPerProcess on the fork parser) restarts a fork after that many files to contain slow leaks.

Source: Apache Tika 4.1.x, Forked-JVM CPU and Heap Sizing, Apache Tika 4.1.x, Shared Server Mode and Apache Tika 4.1.x, Performance and Isolation Trade-offs. Only the 8-core, one-fork line was observed here; the table is the project’s.

Testing with MockParser

MockParser lives in the tika-core tests jar (org.apache.tika.parser.mock.MockParser). It parses XML documents whose root element is <mock> and executes each child element as an action, so you can make a "document" that hangs, runs out of memory, exits the JVM, throws any exception or writes a megabyte of text, and prove your integration survives each one before production finds out. The tests jar registers MockParser through META-INF/services/org.apache.tika.parser.Parser and declares the type application/mock+xml (root element mock) in a custom-mimetypes.xml, so with the jar on the classpath AutoDetectParser and the forks pick it up by themselves:

detect: application/mock+xml parsedBy=org.apache.tika.parser.DefaultParser,org.apache.tika.parser.mock.MockParser

Start every mock file with an XML declaration (<?xml version="1.0" encoding="UTF-8"?>). Without it, the first run for this page detected the file as text/plain; charset=windows-1252 and TextAndCSVParser printed the script as text: the root-element match needs the content to be recognised as XML first.

The actions, from the 4.1.0 source:

Element Effect

<metadata action="add" name="dc:title">value</metadata>

Add or set a metadata value. Reserved tk: names are refused by the 4.x write guard.

<write element="p" times="2000">text</write>

Write SAX text, optionally repeated: huge output for write-limit tests.

<hang millis="60000" heavy="false" interruptible="false"/>

Sleep (or, with heavy="true" and pulse_millis, burn CPU), optionally ignoring interrupts: the infinite loop.

<checkpointedSleep millis="2000" intervalMillis="200"/>

Sleep while reporting progress every interval, like a bounded external call: tests the total budget without tripping the stall detector.

<oom/>

Allocate until a genuine OutOfMemoryError.

<throw class="java.lang.NullPointerException">message</throw>

Throw any Throwable (an OutOfMemoryError too, for a fake OOM).

<system_exit/>

System.exit(1): the native-crash stand-in.

<embedded filename="e1.xml" content-type="application/mock+xml">escaped mock XML</embedded>

An embedded document, parsed through the normal extractor (and so subject to EmbeddedLimits).

<print_out>, <print_err>

Write to standard output or error, to test that stray output does not corrupt the pipes protocol.

<fakeload millis="..." cpu="..." mb="..."/>

Synthetic CPU and memory load (needs the fakeload library on the classpath; not tried here).

<thread_interrupt/>, <throwIllegalChars/>, <parentMetadata>

Interrupt the thread, throw a message with control characters, or add metadata to the parent.

The scripts used on this page:

<?xml version="1.0" encoding="UTF-8"?>
<!-- hang.xml: a parser that never returns -->
<mock>
  <write element="p">Page one parsed, then the parser loops.</write>
  <hang millis="60000" heavy="false" interruptible="false"/>
</mock>
<?xml version="1.0" encoding="UTF-8"?>
<!-- oom.xml -->
<mock>
  <write element="p">About to allocate without bound.</write>
  <oom/>
</mock>
<?xml version="1.0" encoding="UTF-8"?>
<!-- slow-bundle.xml: four slow embedded pages that report progress -->
<mock>
  <write element="p">Scanned loan forms, one per embedded page.</write>
  <embedded filename="page1.xml" content-type="application/mock+xml">&lt;mock&gt;&lt;write&gt;page one&lt;/write&gt;&lt;checkpointedSleep millis="2000" intervalMillis="200"/&gt;&lt;/mock&gt;</embedded>
  <!-- page2.xml to page4.xml are the same -->
</mock>

ok.xml, crash.xml (<system_exit/>), big.xml (<write times="2000">), bundle.xml (four <embedded> documents) and npe-embedded.xml (an embedded <throw>) follow the same pattern.

A JUnit 5 test that pins the Bookshelf worker’s contract, written in the same shape as ForkIntake above (not run as a JUnit test for this page; the same assertions were checked by the ForkIntake run):

class ForkIntakeTest {

    static PipesForkParser parser;

    @BeforeAll
    static void start() throws Exception {
        parser = new PipesForkParser(new PipesForkParserConfig()
                .setTimeoutLimits(new TimeoutLimits(60_000, 5_000))
                .addJvmArg("-Xmx256m"));
    }

    @AfterAll
    static void stop() throws Exception {
        parser.close();
    }

    @ParameterizedTest
    @CsvSource({"hang.xml,TIMEOUT", "oom.xml,OOM", "crash.xml,UNSPECIFIED_CRASH"})
    void survivesAndRecovers(String file, PipesResult.RESULT_STATUS expected) throws Exception {
        try (TikaInputStream bad = TikaInputStream.get(Path.of("src/test/resources/mock", file))) {
            PipesForkResult result = parser.parse(bad);
            assertEquals(expected, result.getStatus());
            assertTrue(result.isProcessCrash());
        }
        try (TikaInputStream good = TikaInputStream.get(Path.of("src/test/resources/mock/ok.xml"))) {
            assertEquals(PipesResult.RESULT_STATUS.PARSE_SUCCESS, parser.parse(good).getStatus()); // fresh fork
        }
    }
}

Keep the tests jar in test scope: MockParser would otherwise be registered in production as a parser for any XML file with a <mock> root.

Source: MockParser source (4.1.0), the example mock files under tika-parsers-standard-integration-tests/src/test/resources/test-documents/mock in the 4.1.0 tag, and Apache Tika 4.1.x, The Robustness of Apache Tika ("Testing your integration with MockParser").

Memory, performance and robustness costs

Cost What to expect

Fork start

About 4.5 to 6 seconds per fork with the standard parser package on the test machine. Paid at the first request and after every crash, recycle or idle shutdown.

Per-document overhead

Serialisation over a local socket and, for streams without a file, spooling to a temp file. Small next to most parses; the warm oom.xml round trip above took 89 ms including the crash.

Memory

Parent plus numClients full JVMs. Budget heap per fork, not per application.

Limits

writeLimit, maxCount and the metadata limiter are counters: negligible cost, and they save memory and index space.

Timeouts

Cooperative checks at boundaries and a heartbeat (default every 1 s). Negligible.

Crash recovery

One document lost per crash plus one fork start. A corpus with many crashing files is dominated by fork starts: quarantine repeat offenders.

In-process parsing

Fastest and lightest, and no protection against OOM, hangs or native crashes. Acceptable only for trusted input or for detection-only work with tika-core.

Pitfalls and security

  • Running untrusted files in-process. An OOM or a native crash takes your service with it. Fork, or call a service.

  • Assuming OutputLimits.writeLimit applies to your own handler. It does not (see output-limits); build the handler from the context or give the limit to the constructor, and on PipesForkParser use setWriteLimit.

  • Relying on the zip-bomb ratio in RMETA mode. It is per stream; a single compressible entry passes. Set a write limit.

  • Treating a successful status as a complete document. PARTIAL_TIMEOUT is a success status. Check the flags in Detecting truncation.

  • Using a write-limited RMETA list as an inventory. With throwing write limits, later documents are missing from the list.

  • Renaming timeoutSeconds to timeoutMillis without multiplying by 1000.

  • Setting progressTimeoutMillis to 0 or Long.MAX_VALUE. The first is rejected, the second disables the client’s backstop. Set it equal to the total to disable stall detection.

  • Retrying OOM and TIMEOUT blindly. The same bytes usually crash the same way, and each crash costs a fork start. Retry once at most, then quarantine.

  • Oversubscribing the host. Too many forks for the cores, or forks without container CPU limits, starve the parent and turn into timeouts on healthy documents.

  • Leaking internals in errors. The default exception-reporting level is FULL. Use MESSAGE_REDACTED with a finite maxLength whenever the caller is not the operator.

  • Shipping the tests jar. Keep MockParser in test scope.

  • Mistaking isolation for a security boundary. A fork protects availability; it is not a sandbox against a malicious file exploiting a parser vulnerability, and tika-server and tika-grpc must stay on trusted networks behind authentication. See Security and the Tika security model: denial of service in-process is the caller’s problem, while a failure of the Pipes boundary is treated as a vulnerability.

1.x to 4.x at a glance

Topic 1.x (the book, Tika in Action) 2.x 3.x 4.x

Crash isolation in a library

ForkParser (ch. 9)

ForkParser; Tika Pipes added

ForkParser, Tika Pipes

PipesForkParser over Tika Pipes; ForkParser removed

tika-server

In-process; -spawnChild opt-in watchdog

Forked child by default, --noFork to opt out

Same; /pipes, /async behind enableUnsecureFeatures

Always forked through Pipes; 429 busy, 503 fork lost

Write limit

BodyContentHandler(int)

Handler limit; a writeLimit request header on tika-server

Same; AutoDetectParserConfig thresholds

output-limits.writeLimit, honoured by context-built handlers; headers removed

Zip-bomb guard

SecureContentHandler

SecureContentHandler; AutoDetectParserConfig (outputThreshold, maximumCompressionRatio) from 2.1.1

Same

output-limits (zipBombThreshold, zipBombRatio, maxPackageEntryDepth, maxXmlDepth)

Embedded limits

none built in

maximumDepth, maximumPackageEntryDepth on AutoDetectParserConfig

Same

embedded-limits (maxDepth, maxCount, and their throw flags)

Timeouts

ForkParser server timeouts

Server taskTimeoutMillis, per-parser timeoutSeconds

Same (server default 5 minutes)

timeout-limits (total, progress, throwOnDeadline), per-parser timeoutMillis, PARTIAL_TIMEOUT

Exception detail

full traces

server returnStackTrace

Same

exception-reporting everywhere

The 2.x cells rest on the 2.x/3.x-era wiki pages and the 4.1.x migration guides rather than on release-by-release checks; the AutoDetectParserConfig thresholds carry a "since 2.1.1" note in their Javadoc, and the writeLimit header is named as removed in the 4.x server migration guide. Tika in Action covers ForkParser and the thread-safety and memory concerns in chapter 9; its reasoning ("parse untrusted files out of process") is current, its API is not.

Source: Apache Tika 4.1.x, Migrating to Tika 4.x, Apache Tika 4.1.x, Migrating tika-server to 4.x, Apache Tika wiki, TikaServer and Tika in Action, chapter 9.

Checklist

  • Never parse untrusted files in the JVM of a service whose availability matters; use PipesForkParser, Pipes, tika-server or tika-grpc.

  • Create one PipesForkParser and share it; close it on shutdown; close every TikaInputStream.

  • Branch on the status category: success (check flags), process crash (quarantine), task exception (continue), initialisation failure (retry later), fatal (stop).

  • Set embedded-limits, output-limits (with a writeLimit), timeout-limits and exception-reporting in parse-context, and the metadata limiter for wide or hostile metadata.

  • Make sure the write limit actually reaches your handler (context-built factory, setWriteLimit, or the constructor).

  • Check the truncation flags and PARTIAL_TIMEOUT before indexing a result as complete.

  • Choose totalTaskTimeoutMillis for your slowest legitimate document (3.x users: the default grew to 1 hour), keep progressTimeoutMillis above your longest honest in-process stretch, and size per-parser timeoutMillis in milliseconds.

  • Size forks with both rules: two cores per fork plus two for the parent, and total fork heap within 75% of memory; start at -Xmx2g; set container CPU limits.

  • Watch the logs for pipes-cpu-sizing warnings, low-heap warnings and maxUnpackBytes truncation.

  • Test hang, OOM, crash, huge output and embedded explosions with MockParser before going live.

What was verified and what was not

Verified by running against Tika 4.1.0 on Java 21 (macOS, 8 cores):

  • PipesForkParser statuses and restarts for a healthy file, a hang (TIMEOUT, killed after 5 s of silence, fork exit code 17), an OOM (OOM, exit code 18), System.exit (UNSPECIFIED_CRASH) and recovery afterwards; fork start time; the pipes-cpu-sizing line; the low-heap WARN at 128 MB.

  • PARTIAL_TIMEOUT with tk:exception:task-deadline-reached in a fork, and the cooperative deadline in-process.

  • That an in-process hang is not stopped by TimeoutLimits.

  • setWriteLimit and setMaxEmbeddedCount on PipesForkParserConfig; that output-limits.writeLimit in a setUserConfigPath file and in a plain context with a self-built handler did not limit output, while BasicContentHandlerFactory.newInstance(type, context) did.

  • RecursiveParserWrapper write-limit behaviour in throwing and non-throwing mode, and the absence of a PropertyTypeException in 90 combinations (see the note in the in-process run).

  • embedded-limits.maxCount with and without throwing; the zip-bomb abort in a single handler and its absence in RMETA mode; the three exception-reporting levels and maxLength.

  • MockParser registration and detection through the tika-core tests jar, and the need for an XML declaration.

Read from the 4.1.0 sources and the 4.1.x documentation but not run: unpack-config, the metadata limiter, the hard watchdog timing (total plus progress), per-request override clamping, the misconfiguration rejections other than the progress-above-total WARN, the tika-app --fork flags, tika-server and tika-grpc behaviour (covered on their own pages), the CPU and heap sizing table, shared-server mode, and the JUnit test and isComplete helper as written. The throwOnDeadline run through PipesForkParserConfig did not show the documented container exception; the reason was not investigated. The 3.x ForkParser example is shown for recognition and was not run.

References