Tika Pipes

A pipeline is assembled from four kinds of pluggable component. Each is named in tika-config.json and each is a PF4J plugin, so a source or a destination Tika does not ship can be added without rebuilding Tika.

The FetchEmitTuple is the unit of work. It carries an id (yours, echoed in logs and reports), a FetchKey (fetcher id plus key), an EmitKey (emitter id plus key), optional Metadata to seed the parse (a content-type hint, a resource name, or business fields such as a shelf code), an optional per-document ParseContext (parse mode, limits, emit strategy) and an onParseException choice (EMIT or SKIP). The iterator builds the tuples; with the Java API you can build them yourself.

Source: Apache Tika 4.1.x, Tika Pipes (The four roles), Pipes Iterators and the 4.1.0 FetchEmitTuple source.

The same machinery is reachable at six levels. Pick the highest one that does the job.

PipesClient is the per-fork socket client wrapped by the last three; it is not a user-facing API.

Source: Apache Tika 4.1.x, Tika Pipes from Java and Tika Pipes (Overview).

The default numClients is derived from the host: max(1, mincores - 2) / 2, 4, so roughly half the cores, never more than four (PipesConfig.defaultNumClients() in the source). The batch CLIs (tika-app -i/-o, tika-async-cli) use 2 when neither -n nor a config value is given, and PipesForkParser uses 1.

Each JVM sizes its garbage collector, JIT and fork-join pools from the host CPU count, so four forks on a 16-core host would otherwise start about 64 GC threads. Unless you pass your own, Tika injects -XX:ActiveProcessorCount (a fair slice of the cores, never below 2, with 2 cores reserved for the parent) and -XX:MaxRAMPercentage (so the forks' combined heap stays under the host’s memory) into each fork. The tika-app run on an 8-core laptop logged:

The documented rule of thumb: pick the largest numClients for which twice that number plus two cores still fits on the host; beyond it, more forks reduce throughput. Size -Xmx per fork so that three quarters of it covers the parsing working set, because a quarter is the default in-memory cache budget (tika.pipes.cacheMemoryBudgetBytes, a fork-side system property). The Docker & Kubernetes Deployment page applies this to container limits.

Source: Apache Tika 4.1.x, Forked-JVM CPU and Heap Sizing and Pipes Configuration (Process Management, Embedded-object cache memory budget).

Every document ends in one PipesResult.RESULT_STATUS, grouped into five categories. PipesResult and PipesForkResult have one predicate per category.

Source: the 4.1.0 PipesResult source and Tika Pipes from Java (Reading a result).

These are the options of the pipes mode (TikaAsyncCLI in the 4.1.0 source). The Tika App page covers the rules that decide when tika-app enters this mode and the "in flux" status of the 4.x command line.

A Markdown copy of each document, without any metadata, is one flag away:

And -Z unpacks every attachment of the publisher’s e-mail, including the files inside the attached ZIP:

The extracted files are numbered by their embedded id; the JSON beside them is the RMETA list, in which each embedded object carries tk:embedded-id so you can map 00000002.docx back to /samples.zip/chapter-one.docx. The text note inside the attached ZIP came out as 00000005.bin; the extension comes from the suffixStrategy of unpack-config (see UNPACK and unpack-config).

Source: the 4.1.0 TikaAsyncCLI source and Apache Tika 4.1.x, Parse Modes (CLI usage).

For anything beyond a folder walk, write a tika-config.json and run it with -c, or with -a -c when no -i and -o are given. The next section builds the Bookshelf file. Flags still override the file where they overlap.

The Bookshelf intake service receives manuscripts, catalogues, scans and publisher e-mails into an inbox folder and needs one JSON extract per document plus a record of anything that failed. This is the file used for the AsyncProcessor run later on this page.

Notes on the file:

Source: Apache Tika 4.1.x, Pipes Configuration (Complete examples), Fetchers (Reserved IDs) and File System Plugin.

All defaults below were read from the 4.1.0 PipesConfig and EmitStrategyConfig sources. The pipes section is shared by the parent and the forks; each ignores the keys it does not use.

Source: Apache Tika 4.1.x, Pipes Configuration and the 4.1.0 PipesConfig source.

The parse mode decides how embedded documents are handled and what the emitter writes. It is set as parseMode in the pipes section and can be overridden per document by putting a ParseMode into the tuple’s ParseContext.

The content format is independent of the mode: the top-level content-handler-factory (TEXT, HTML, XML, MARKDOWN, BODY, IGNORE; MARKDOWN when unset) applies to every mode that produces content. With CONTENT_ONLY set the emitter’s fileExtension to match the handler (md, txt, html, xml). The handlers themselves are described in Content Handlers & Output Formats and the RMETA list in Embedded Documents & Recursive Parsing.

Source: Apache Tika 4.1.x, Parse Modes and the 4.1.0 ParseMode source.

When a fork finishes a document, someone has to call the emitter. emitStrategy decides who:

Passed-back extracts wait in the parent until a batch is flushed by size (emitMaxEstimatedBytes) or by time (emitWithinMillis), then the numEmitters threads write them. Batching matters for the search emitters, which send bulk requests.

emitIntermediateResults decides what happens to documents whose parse did not succeed. Two runs of the Bookshelf CSV pipeline with an absurd five-millisecond progressTimeoutMillis, so that every document timed out, showed the difference: with false (the default) the output folder stayed empty and only the reporter knew about the three documents; with true the emitter wrote one file per document, each containing just [{"tk:pipes-result":"TIMEOUT"}]. Use true when the destination itself must hold a record of every attempt, and a JSON-lines reporter otherwise.

Source: Apache Tika 4.1.x, Pipes Configuration (Emit Strategy, Async / Emit Batching) and the 4.1.0 EmitStrategy source.

Two limits in parse-context.timeout-limits bound each document: totalTaskTimeoutMillis (end to end, default one hour) and progressTimeoutMillis (how long the parse may go silent). When the total deadline passes, the fork first tries to stop cleanly and return what it has as PARTIAL_TIMEOUT (also with throwOnDeadline, which only records an exception in the container metadata); a hard watchdog in the parent kills the fork if that does not happen within the progress window, which yields TIMEOUT. The progress timeout produced this result in the failure run below: Server-side progress timeout: no progress for 11ms (limit: 5ms). The complete model, the 3.x renames and the sizing advice are on Robustness, Limits & Timeouts.

Source: Apache Tika 4.1.x, Timeouts.

Reporters are called once per document by the parent, with the tuple, the PipesResult and the elapsed time. Because they run in the parent, they see crashes that never reach an emitter. Every configured reporter receives every call; the first exception any of them throws is rethrown afterwards and stops the run.

The file-system-reporter rewrites a status file every reportUpdateMs (always set it: the value is the writer thread’s sleep, so leaving it at 0 rewrites the file in a tight loop). After the Bookshelf run:

The write is not atomic, so a watcher (a dashboard, the Tika GUI) must treat a JSON parse error as a stale read and try again on the next poll.

The file-system-jsonl-reporter (new in 4.1.0) appends one line per document and flushes it before returning, so it survives the death of the parent too. With includes you can turn it into a crash ledger for a dead-letter process or for tika-eval. Lines from the successful run and from the timeout run:

The other reporters are jdbc-reporter (rows in a table), opensearch-pipes-reporter and es-pipes-reporter (status documents in an index).

Source: Apache Tika 4.1.x, Pipes Reporters and File System Plugin (File System Reporter, File System JSONL Reporter).

Every fetcher, emitter, iterator and reporter in 4.x is a PF4J plugin. A plugin contributes one factory per component, each annotated with @org.pf4j.Extension and listed in META-INF/extensions.idx. The configuration never names a class: it names the factory’s getName(), such as file-system-fetcher.

Plugins reach a Tika host in one of two ways:

The plugin zips come from the distribution zips on the download page; they are not on Maven Central, the jars are. When plugin-roots points at a folder that does not exist, PF4J logs a warning and Tika carries on with the class-path plugins, which is what the runs on this page did:

For plugin development, -Dtika.plugin.dev.mode=true (or TIKA_PLUGIN_DEV_MODE=true) treats each plugin-roots entry as one already exploded plugin folder, so you can point it at a module’s target/classes.

Source: Apache Tika 4.1.x, Tika Pipes from Java (Where plugins come from) and Writing a Pipes Plugin.

The component names come from the 4.1.x plugin pages and the artifact ids from the module folders of the 4.1.0 source tree (tika-pipes/tika-pipes-plugins/tika-pipes-*). The file-system, CSV, HTTP, JDBC, Kafka, S3, Solr, Elasticsearch and OpenSearch jars were also checked on Maven Central.

Source: Apache Tika 4.1.x, Pipes Plugins (Plugin / Interface Matrix).

The file-system plugin is the reference implementation and the one most pipelines start with.

basePath is the only containment. Without it there is none, and allowAbsolutePaths: true is how you assert that the keys come from a fully trusted source. The fetcher re-checks symbolic links with toRealPath(), the emitter does not, so a link already present under the emitter’s basePath can be written through.

Source: Apache Tika 4.1.x, File System Plugin (Security Notes).

The three object-store plugins provide a fetcher, an emitter and an iterator each, with keys relative to a prefix. For Bookshelf, scans uploaded to a bucket could be read and the extracts written back to another prefix. Credentials are not in the file: credentialsProvider: "profile" (or the instance role) for S3, Application Default Credentials for GCS, and an environment variable for the Azure SAS token.

Not run for this page; the key names are the ones on the 4.1.x S3, Azure Blob and GCS plugin pages.

Source: Apache Tika 4.1.x, Amazon S3 Plugin, Azure Blob Storage Plugin and Google Cloud Storage Plugin.

These are fetcher-only plugins; pair them with an iterator that produces the right keys (a CSV of URLs, a JSON-lines work list, your own code) and any emitter. For http-fetcher the fetch key is the URL itself; it supports basic and JWT authentication, connection and timeout settings, extra headers and maxSpoolSize, and since 4.0 it verifies TLS certificates and host names by default (verifySsl: false opts out, which you should not do outside a test).

google-drive-fetcher takes a base64 service-account key (serviceAccountKeyBase64), subjectUser, scopes and applicationName; feed the key from an environment variable as well. An HTTP fetcher that accepts URLs from callers (tika-server /pipes, tika-grpc) is a server-side request forgery risk; see the Security page.

Source: Apache Tika 4.1.x, HTTP Plugin, Microsoft Graph Plugin and Google Drive Plugin.

When the list of documents comes from a system rather than a folder, use a work-list iterator. The Bookshelf intake desk exports the day’s submissions as a CSV, with business fields that should end up in each extract:

Replace the iterator in the Bookshelf file with:

Run with the Intake program of the Java section (and tika-pipes-csv on the class path), the pipeline wrote 2026/BK-0001.json, 2026/BK-0002.json and 2026/BK-0003.json, the ledger used the CSV ids ({"id":"BK-0001","status":"PARSE_SUCCESS",...}), and every column not consumed as a key became metadata: the manuscript’s extract carried "shelf": "fiction" and "submitted_by": "ana.torres" next to "dc:title": "The Lighthouse Keeper". The CSV must be UTF-8 with a header row (Excel format); a column whose header collides with a reserved Tika key is skipped with a log line.

The json-pipes-iterator reads a JSON-lines file in which each line is a whole serialized FetchEmitTuple (id, fetcher, fetchKey, emitter, emitKey, and optionally metadata, parse-context, onParseException, preset), so each line names its own fetcher and emitter; unknown fields are rejected. The jdbc-pipes-iterator walks a SELECT (select, idColumn, fetchKeyColumn, emitKeyColumn, fetchSize), and the jdbc-emitter writes chosen metadata keys into a table through an insert template with typed keys.

Source: Apache Tika 4.1.x, CSV Plugin, JSON Plugin and JDBC Plugin.

The three search emitters send bulk requests, so they benefit from passback batching. Each takes the index or collection URL, an idField (the emit key becomes the document id), an attachmentStrategy for embedded documents (the 4.1.x examples use PARENT_CHILD) and an updateStrategy (ADD for Solr, OVERWRITE for the other two). The Elasticsearch and OpenSearch emitters also take a chunkStrategy for tk:chunks.

Restrict what reaches the index with a metadata filter (see Metadata) instead of indexing every key Tika produces. The solr-pipes-iterator reads documents back out of a collection (for example everything with status:NEEDS_REPARSE) to re-parse them. SolrJ moved to 10.0 in Tika 4.0, so the Solr plugin no longer supports Solr 8. For the index side see Solr indexing & updates and Elasticsearch ingest pipelines. Not run for this page.

Source: Apache Tika 4.1.x, Elasticsearch Plugin, OpenSearch Plugin and Apache Solr Plugin.

The Kafka plugin has two halves with different standing.

The kafka-emitter is a plain producer and a good fit for streaming extracts to downstream indexers. Its keys map onto the standard producer settings (topic, bootstrapServers, acks, lingerMs, batchSize, compressionType, enableIdempotence, maxRequestSize, deliveryTimeoutMs, transactionalId, and so on); Tika sets no values of its own, so an absent key means the Kafka client default. Read from the 4.1.0 KafkaEmitter source: the emitter sends one record per Metadata object, so an RMETA extract of publisher-email.eml becomes six records with the same key (the emit key). The value is a JSON object of the metadata fields, and only the first value of a multi-valued key is kept (with a warning in the log). Use CONCATENATE when downstream consumers expect one record per document.

The kafka-pipes-iterator is documented as a worked example rather than a supported production integration. In 4.1.0 it uses each record’s key as the tuple id, the fetch key and the emit key (the record value is not read). It leaves enable.auto.commit at Kafka’s default, so offsets are committed when records are polled, before Tika has parsed or emitted them: a crash in between loses those documents, which is at-most-once delivery. It drains the topic and stops after drainIdleMs (1000 ms) of quiet, so it suits a periodic batch run, not a long-running consumer. And a slow document holds up its partition. For long-running or OCR-heavy work, write your own consumer that commits after a successful result and drives tika-server, tika-grpc or PipesParser (see the AsyncProcessor section), and keep the Kafka emitter for the results.

The Kafka side of these trade-offs is covered in Kafka Consumers, Kafka Producers, Delivery Guarantees and Kafka Reliability and Exactly-Once Semantics. Not run for this page.

Source: Apache Tika 4.1.x, Apache Kafka Plugin and the 4.1.0 KafkaEmitter and KafkaPipesIterator sources.

A plugin is three kinds of class, one plugin.properties and a zip with a fixed layout. Using a Bookshelf fetcher for a supplier portal as the example:

The zip holds plugin.properties at its root, classes/META-INF/extensions.idx, and lib/ with the plugin’s own jar and its runtime dependencies. Everything on the boundary between host and plugin is declared provided and must not be in lib/: pf4j, tika-core, tika-plugins-core, tika-pipes-api, tika-serialization, the logging API and, unless you deliberately bundle your own, Jackson. A plugin that ships its own tika-core loads cleanly and then finds no extensions or fails with a ClassCastException, because its FetcherFactory is a different class from the host’s. Follow the logging convention of the bundled plugins: literal fetch and emit keys only at TRACE.

The factory above was written from the 4.1.x guide and not compiled for this page. The parser-side extension points are on Extending Tika.

Source: Apache Tika 4.1.x, Writing a Pipes Plugin and tika-pipes-file-system (the reference plugin).

PipesForkParser (module tika-pipes-fork-parser) is the 4.x replacement for the removed ForkParser. It is the simplest way to give an application process isolation: you pass a Path or a TikaInputStream, a fork parses it and the text and metadata come back in a PipesForkResult. Internally it writes a temporary tika-config.json with an internal file-system fetcher (UUID-named, allowAbsolutePaths: true), the PASSBACK_ALL emit strategy and your settings, and loads a PipesParser from it.

The Bookshelf intake check below parses every file in the inbox in Markdown and RMETA mode, lists embedded documents and handles the three outcome classes. It also parses a stream with no file behind it, which travels inline to the fork.

The whole run took about 7 seconds of wall time, most of it the start of the first fork. renamed-invoice.png was detected as the PDF it is. In the RMETA list the embedded documents come after the container, deepest first (the image inside the attached manuscript precedes the manuscript itself). getContent() is only the container’s text: for the e-mail, 112 characters out of a much larger total, so iterate getMetadataList() and read TikaCoreProperties.TIKA_CONTENT from each element when you need everything.

Source: PipesForkParserExample.java and Apache Tika 4.1.x, Tika Pipes from Java (PipesForkParser).

PipesParser (in tika-pipes-core) takes a FetchEmitTuple and returns a PipesResult, with everything else coming from a config file. The same code reads from a folder today and from a bucket tomorrow. It is thread-safe: each call borrows one of the numClients forks and waits up to maxWaitForClientMillis for one to free up.

Against the Bookshelf file (fetcher inbox, emitter extracts), this program writes one extract from the fork, gets one back instead, and asks for a file that is not there:

The first call wrote /srv/bookshelf/extracts/single/manuscript.json (the emitter appended .json), which begins:

(shortened: the real file also lists fs:accessed, the detection keys, tk:parsed-by and the ZIP detector flag.) A passed-back result under PipesParser is yours to write: there is no emitter thread behind it.

Source: Apache Tika 4.1.x, Tika Pipes from Java (PipesParser, Getting the result back instead).

AsyncProcessor adds the queue, the worker threads that feed the forks, the emitter threads and the reporters. You offer tuples; it does the rest. With an iterator in the config, the tuples come from the config too:

The extracts folder then held the nine .json files, status.json showed "statusCounts":{"PARSE_SUCCESS":9} and the JSONL ledger had nine lines (shown in the Reporters section). Swapping the iterator for the CSV one produced the 2026/BK-000n.json extracts described there.

An iterator is optional. Pass null as the second argument of AsyncProcessor.load and offer tuples you build yourself, from a Kafka consumer, a database cursor or an upload queue. offer blocks while the queue (queueSize) is full and returns false if it never drains. Per-document outcomes go to the reporters, not back to the caller; checkActive only surfaces failures that stop the pipeline. Because AsyncProcessor emits passed-back extracts itself, the default DYNAMIC strategy is the right one here.

Source: Apache Tika 4.1.x, Tika Pipes from Java (AsyncProcessor: the whole pipeline).

The Spring Boot Integration page of this section covers PipesForkParser as a Spring bean with a matching shutdown.

Forks inherit the parent’s stdout and stderr, which suits containers: everything reaches the container log. The bundled fork logging configuration (pipes-fork-server-default-log4j2.xml) marks fork lines with [fork], and both sides print pipesClientId=N, so you can follow one fork across the boundary:

The 4.1.0 forks in these runs also logged one WORKER_TIMING line per document at INFO, with the deserialization, fetch, parse and emit times in microseconds and the tuple id. For PipesForkParser the id is the absolute file path, so treat these lines as containing file names:

A property that "does nothing" is usually set on the wrong JVM: fork-side properties belong in forkedJvmArgs, and set on the parent (java -D..., MAVEN_OPTS, Surefire argLine) they are silently ignored by the forks.

Logging levels follow a convention: ERROR and WARN never contain literal fetch or emit keys (documents are identified by the tuple id), INFO is lifecycle, DEBUG is per-document progress and counts, TRACE adds the literal keys (paths, URLs, object keys). HTTP fetchers may still include response bytes in exception messages, limited by maxErrMsgSize.

Source: Apache Tika 4.1.x, Pipes Troubleshooting and Pipes Configuration (Where temporary files go).