Search Indexing with Lucene, Solr & Elasticsearch

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

Baseline date: 2026-10-13. This page was written against Apache Tika 4.1.0 (the latest release on Maven Central on that date) with the 3.3.x differences noted, Java 21 and later for the examples (Tika 4.x needs Java 17; Lucene 10 needs Java 21), Spring Boot 4.x on Spring Framework 7, the Docker image apache/tika:4.1.0, Apache Lucene 10.5.2, Apache Solr 10.0 and current Elasticsearch and OpenSearch releases. Search is the oldest use of Tika: Tika in Action spent its chapter 10 on feeding Lucene and Solr, and Tika was born as a Lucene subproject. The idea has not changed. A search engine indexes text and fields, the files people want to find are PDFs, Office documents, e-mails and archives, and Tika is the stage in between that turns bytes into text and metadata. What changed is everything around it: the Lucene 3.0 API of the book is gone, Solr Cell no longer runs Tika inside Solr, Tika Pipes has emitters that write straight to Solr, Elasticsearch and OpenSearch, the search clusters carry their own embedded Tika for small jobs, and Tika 4.x can attach chunked embedding vectors to every document.

This page walks the Bookshelf intake through each route: an embedded Lucene index built in-process, Solr Cell, the Tika Pipes search emitters, a client that calls tika-server and the bulk API, and the in-cluster attachment processors. It ends with chunk strategies for vector search, the field-shaping filters every route needs, and the 1.x to 4.x changes. It links the index-side pages of this site rather than repeat them: Apache Lucene, Solr indexing & updates (with the Solr Cell parameter overview), Elasticsearch ingest pipelines and Lucene vs. Solr vs. Elasticsearch vs. OpenSearch.

Related Tika pages: Tika Pipes (fetchers, emitters, the forked-JVM model), Metadata (the tk: keys), Embedded Documents & Recursive Parsing (the list-of-metadata model every route below relies on), tika-server REST API, VLM Parsers & Inference (how tk:chunks is produced), RAG & LLM Ingestion and Performance & Scaling.

The Lucene example and the metadata-filter configuration on this page were compiled and run against Tika 4.1.0 and Lucene 10.5.2 on Java 25, with the same small Bookshelf stand-in files used across this section. The Solr, Elasticsearch and OpenSearch configurations were checked against the Tika 4.1.0 documentation and emitter sources and the Solr 10.0.0 sources, but not run against a live cluster. What was verified and what was not has the details.

Tika as the extraction stage of a search pipeline

Every search pipeline that ingests files has the same four steps: find the files, extract text and metadata, shape the result into the fields of an index, and write it. Tika does the second step, and in Tika Pipes it can also do the first (fetchers and iterators) and the last (emitters). Where the extraction runs is the main design decision, because parsing is the step that crashes, hangs and runs out of memory.

flowchart LR subgraph sources["Files"] fs["Folder or share"] obj["S3, GCS, Azure Blob"] up["Uploads from an app"] end subgraph tika["Tika"] detect["Detect media type"] parse["Parse: text and metadata, embedded documents"] list["List of metadata: container first, then attachments"] filters["Metadata filters: keep, rename, drop"] detect --> parse --> list --> filters end subgraph index["Index"] lucene["Lucene IndexWriter (in-process)"] solr["Solr /update"] es["Elasticsearch or OpenSearch _bulk"] end fs --> detect obj --> detect up --> detect filters -->|"Lucene Document"| lucene filters -->|"solr-emitter or SolrJ"| solr filters -->|"es-emitter, opensearch-emitter or a bulk client"| es

The diagram shows the external-Tika shape. Two routes move Tika inside the search engine instead: Solr Cell (which in Solr 10 calls a separate tika-server for you) and the Elasticsearch and OpenSearch attachment processors (which run Tika in the cluster’s ingest JVM). The table compares the five routes this page covers.

Route Where Tika runs Isolation Best for

Tika in your Java code, writing a Lucene index

In your JVM, as a library (AutoDetectParser, RecursiveParserWrapper), or in a fork with PipesForkParser

None in-process; a forked JVM per worker with PipesForkParser

Desktop and embedded search, small corpora, tests, applications that already own a Lucene index

Solr Cell (/update/extract)

In a tika-server process that Solr 10 calls over HTTP (Solr 9 could also run Tika inside Solr)

Separate process, but the request blocks a Solr update thread

Moderate volumes, uploads straight into Solr, teams that want no extra code

Tika Pipes with a search emitter

In forked Tika JVMs, driven by tika-app -a, tika-server /async or your own AsyncProcessor

A fork per worker; crash, timeout and OOM are reported per file

Bulk and continuous ingestion into Solr, Elasticsearch or OpenSearch

Your client: tika-server, then the index API

In tika-server (forked parsing in 4.x)

Separate service; your client decides retries and batching

Applications that need custom enrichment between extraction and indexing

Elasticsearch attachment / OpenSearch ingest-attachment

Inside the search cluster’s ingest nodes, on base64 input

None from the cluster: a bad file costs ingest-node heap and CPU

Occasional small attachments when no separate service is wanted

The rest of the page goes through them in that order. Whichever route you pick, the same three questions decide the index design: which text to index, which metadata becomes fields, and what happens to embedded documents.

What a search index needs from Tika

The text

Tika returns the text through a content handler (see Content Handlers & Output Formats). For a classic full-text index use the TEXT handler type (/rmeta/text on tika-server, "type": "TEXT" in a basic-content-handler-factory). Markdown, the 4.x default, keeps headings and tables that a chunker for embeddings uses, but adds # and | characters that a plain full-text field does not need. When you build tk:chunks for vector search, keep Markdown, because the Markdown chunker splits on headings first.

Two details matter in every route:

  • Write limit. BodyContentHandler stops at 100,000 characters by default and BasicContentHandlerFactory takes an explicit limit. A search index rarely needs the full text of a 2,000-page PDF, so set a limit that matches the field and decide whether reaching it is an error (throwOnWriteLimitReached) or a truncation you record. The Elasticsearch attachment processor has its own limit (indexed_chars, 100,000 by default), and so does Solr Cell with the Tika server backend (tikaserver.maxChars). See Robustness, Limits & Timeouts.

  • Whitespace. The text Tika returns starts with blank lines where the XHTML <head> and empty block elements were. An analyzer ignores them, but a stored field shown as a snippet looks odd. Trim before storing.

The metadata fields

Tika produces dozens of keys per document; an index wants a handful. The table lists the keys worth indexing for the Bookshelf files, with the field class or type each one maps to. The 4.x key names are used throughout; 3.x used X-TIKA:content and resourceName where 4.x has tk:content and tk:resource-name (see Metadata for the full mapping).

Tika 4.x key Index field (this page) Lucene 10 field class Solr field type Elasticsearch / OpenSearch type

tk:content

content

TextField, not stored

text_general

text

dc:title

title

TextField, stored

text_general

text with a keyword sub-field

dc:creator (multi-valued)

author

KeywordField per value

string, multiValued

keyword

dcterms:created, dcterms:modified

created, modified

LongField (epoch milliseconds)

pdate

date

Content-Type (base type only)

content_type

KeywordField

string

keyword

tk:resource-name

file_name

KeywordField or TextField

string or text_general

keyword or text

tk:embedded-id-path

embedded_path

KeywordField

string

keyword

dc:language or tk:detected-language

language

KeywordField

string

keyword

xmpTPg:NPages

pages

IntField

pint

integer

tk:parsed-by, tk:exception:*

parsed_by, parse_error

StoredField

string, stored only

keyword, "index": false

Three rules keep this manageable:

  1. Choose the fields before the first run, and make Tika emit only those (see Metadata filters). An Elasticsearch index with dynamic mapping otherwise grows a field for every key any file ever carried, and a strict Solr schema rejects the first unknown one.

  2. Treat document metadata as untrusted. dc:title and dc:creator are claims the file makes about itself: a spammer can stuff them, and they can be empty, wrong or in another script. Weight them accordingly in relevance.

  3. Index the base media type. Content-Type arrives as text/plain; charset=windows-1252; facet on text/plain.

Embedded documents

The Bookshelf publisher-email.eml holds a zip with a .docx (which holds a picture), a PDF and a text note. RecursiveParserWrapper, /rmeta and the pipes RMETA parse mode return that as a list of six metadata objects, container first (see Embedded Documents & Recursive Parsing). An index has three ways to store such a list:

Shape How Trade-off

Concatenate

One index document per file; the text of all attachments appended to the container’s text (pipes CONCATENATE, /tika, AutoDetectParser with a plain handler)

Simplest. A hit says "this e-mail", never "the PDF inside it"; per-attachment metadata is lost.

Separate documents

One index document per element of the list, each with a reference to its container (attachmentStrategy: SEPARATE_DOCUMENTS)

Each attachment is a hit of its own. Group results by container at query time (collapse in Elasticsearch, grouping in Solr and Lucene).

Parent and children

The attachments as nested or joined child documents of the container (attachmentStrategy: PARENT_CHILD)

Queries can find containers by their children and the other way round. Needs a schema for it (Solr nested documents, an Elasticsearch join field with routing) and the joins cost query time.

The emitters support the last two; your own code can do any of them. The Lucene example below uses separate documents.

Tika to Lucene

Lucene is a library, so this route is just two libraries in one JVM: Tika parses, Lucene’s IndexWriter indexes. It is the modern form of the book’s chapter 10 example.

Dependencies

<dependencies>
  <dependency>
    <groupId>org.apache.tika</groupId>
    <artifactId>tika-core</artifactId>
    <version>4.1.0</version>
  </dependency>
  <dependency>
    <groupId>org.apache.tika</groupId>
    <artifactId>tika-parsers-standard-package</artifactId>
    <version>4.1.0</version>
    <type>pom</type>
  </dependency>
  <dependency>
    <groupId>org.apache.lucene</groupId>
    <artifactId>lucene-core</artifactId>
    <version>10.5.2</version>
  </dependency>
</dependencies>
dependencies {
    implementation("org.apache.tika:tika-core:4.1.0")
    implementation(platform("org.apache.tika:tika-parsers-standard-package:4.1.0"))
    implementation("org.apache.lucene:lucene-core:10.5.2")
}

Source: Apache Tika 4.1.x, Java API: Getting Started and Apache Lucene 10.5.2 documentation. StandardAnalyzer lives in lucene-core; language analyzers need lucene-analysis-common (see Lucene language analysis). The Gradle form of a pom-type dependency depends on how your build resolves it; check that the parser modules arrive on the runtime class path. See Setup & Installation for the dependency options.

Indexing the Bookshelf intake

The program walks the intake folder, parses each file with RecursiveParserWrapper so that attachments become their own Lucene documents, maps the metadata to Lucene 10 field classes and writes the index. It then runs two term queries.

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Date;
import java.util.List;
import java.util.stream.Stream;

import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.KeywordField;
import org.apache.lucene.document.LongField;
import org.apache.lucene.document.StoredField;
import org.apache.lucene.document.TextField;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.ScoreDoc;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.search.TopDocs;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;
import org.apache.tika.exception.TikaException;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.HttpHeaders;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.metadata.TikaCoreProperties;
import org.apache.tika.parser.AutoDetectParser;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.parser.RecursiveParserWrapper;
import org.apache.tika.sax.BasicContentHandlerFactory;
import org.apache.tika.sax.RecursiveParserWrapperHandler;
import org.xml.sax.SAXException;

public class BookshelfLuceneIndexer {

    private static final int WRITE_LIMIT = 1_000_000;   // characters of text per document

    public static void main(String[] args) throws Exception {
        Path intake = Path.of(args[0]);
        Path indexDir = Path.of(args[1]);
        RecursiveParserWrapper parser = new RecursiveParserWrapper(new AutoDetectParser());

        try (Directory dir = FSDirectory.open(indexDir);
             IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig(new StandardAnalyzer()));
             Stream<Path> files = Files.list(intake)) {
            for (Path file : files.sorted().toList()) {
                for (Metadata m : parse(parser, file)) {
                    Document doc = toLuceneDocument(intake.relativize(file).toString(), m);
                    // updateDocument = delete-then-add by id, so a re-run replaces instead of duplicating
                    writer.updateDocument(new Term("id", doc.get("id")), doc);
                }
            }
            writer.commit();
        }

        try (Directory dir = FSDirectory.open(indexDir);
             DirectoryReader reader = DirectoryReader.open(dir)) {
            IndexSearcher searcher = new IndexSearcher(reader);
            System.out.println("documents in index: " + reader.numDocs());
            for (String word : args.length > 2 ? List.of(args).subList(2, args.length) : List.of("catalogue")) {
                TopDocs top = searcher.search(new TermQuery(new Term("content", word)), 10);
                System.out.println("content:" + word + " -> " + top.totalHits);
                for (ScoreDoc sd : top.scoreDocs) {
                    Document hit = searcher.storedFields().document(sd.doc);
                    System.out.printf("  %-45s %-20s %s%n", hit.get("id"), hit.get("content_type"), hit.get("title"));
                }
            }
        }
    }

    static List<Metadata> parse(RecursiveParserWrapper parser, Path file)
            throws IOException, SAXException, TikaException {
        RecursiveParserWrapperHandler handler = new RecursiveParserWrapperHandler(
                new BasicContentHandlerFactory(BasicContentHandlerFactory.HANDLER_TYPE.TEXT, WRITE_LIMIT));
        Metadata metadata = new Metadata();
        try (TikaInputStream tis = TikaInputStream.get(file, metadata)) {
            parser.parse(tis, handler, metadata, new ParseContext());
        }
        return handler.getMetadataList();
    }

    static Document toLuceneDocument(String containerId, Metadata m) {
        String idPath = m.get(TikaCoreProperties.EMBEDDED_ID_PATH);   // null for the container itself
        String id = idPath == null ? containerId : containerId + "!" + idPath;
        Document doc = new Document();
        doc.add(new KeywordField("id", id, Field.Store.YES));
        doc.add(new KeywordField("container_id", containerId, Field.Store.YES));
        String type = m.get(HttpHeaders.CONTENT_TYPE);
        if (type != null) {
            // "text/plain; charset=windows-1252" -> "text/plain": facet on the base type only
            doc.add(new KeywordField("content_type", type.split(";")[0].strip(), Field.Store.YES));
        }
        String title = m.get(TikaCoreProperties.TITLE);
        if (title != null) {
            doc.add(new TextField("title", title, Field.Store.YES));
        }
        for (String creator : m.getValues(TikaCoreProperties.CREATOR)) {
            doc.add(new KeywordField("creator", creator, Field.Store.YES));
        }
        Date created = m.getDate(TikaCoreProperties.CREATED);
        if (created != null) {
            doc.add(new LongField("created", created.getTime(), Field.Store.YES));
        }
        String content = m.get(TikaCoreProperties.TIKA_CONTENT);
        if (content != null) {
            doc.add(new TextField("content", content, Field.Store.NO));   // searchable, not stored
        }
        String parsedBy = String.join(",", m.getValues(TikaCoreProperties.TIKA_PARSED_BY));
        doc.add(new StoredField("parsed_by", parsedBy));
        return doc;
    }
}
$ java -cp target/classes:$(cat cp.txt) BookshelfLuceneIndexer intake idx catalogue spring
documents in index: 10
content:catalogue -> 4 hits
  catalogue.pdf                                 application/pdf      Spring catalogue
  publisher-email.eml!/1/4                      application/pdf      Spring catalogue
  renamed-invoice.png                           application/pdf      Spring catalogue
  publisher-email.eml!/1/5                      text/plain           null
content:spring -> 5 hits
  catalogue.pdf                                 application/pdf      Spring catalogue
  publisher-email.eml!/1/4                      application/pdf      Spring catalogue
  renamed-invoice.png                           application/pdf      Spring catalogue
  publisher-email.eml                           message/rfc822       Spring submissions: samples attached
  publisher-email.eml!/1/5                      text/plain           null

Source: RecursiveParserWrapper, Javadoc 4.1.0, KeywordField, Lucene 10.5.2 Javadoc, IndexWriter, Lucene 10.5.2 Javadoc and the tika-example module on GitHub (LuceneIndexerExtended is its 1.x-era equivalent). Run on Java 25 with five intake files (catalogue.pdf, legacy-catalogue.txt, manuscript.docx, publisher-email.eml, renamed-invoice.png); Lucene also printed an INFO line about its vectorization provider, left out here.

What the run shows:

  • Ten Lucene documents from five files. The e-mail produced six (container, zip, .docx, picture, PDF, note); the other four files one each. The PDF inside the zip and the stand-alone catalogue.pdf are separate hits with different ids.

  • renamed-invoice.png is a PDF. Detection looked at the bytes, not the extension, so it was indexed as application/pdf with the PDF’s title. That is why Content-Type from Tika, not the file name, belongs in the content_type field.

  • Ids are stable. The container id is the relative path and an attachment adds its tk:embedded-id-path (publisher-email.eml!/1/4). A second run of the same program still reports ten documents, because updateDocument deletes by the id term before it adds.

  • Nothing matched manuscript. Term queries match analyzed tokens of the text; the file name is not in the content field. Index tk:resource-name as its own field if users search by file name.

Choosing the Lucene field classes

The field class decides what a field can do (see Documents, fields & the capability matrix for the full matrix):

  • TextField for anything people type words into: content, title. Store content only if you show snippets or highlight; the stored copy roughly doubles the index size for text-heavy corpora, and highlighting a long field also wants offsets (see Highlighting, suggesters and more).

  • KeywordField for ids, media types, authors and languages: exact match plus doc values for sorting and faceting in one field. This is the 10.x replacement for the 3.0 idiom Field.Index.NOT_ANALYZED.

  • LongField (or IntField) for dates and page counts: range queries and sorting. Convert dates with Metadata.getDate(Property), which parses the ISO 8601 value Tika writes.

  • StoredField for diagnostics such as tk:parsed-by: retrievable, never searchable.

  • KnnFloatVectorField for embeddings: decode a tk:chunks vector (base64, big-endian float32) and add one Lucene document per chunk; see kNN vector search.

To keep attachments together with their container, Lucene offers block indexing (IndexWriter.addDocuments with the parent last) and ToParentBlockJoinQuery; see Grouping and joins. The example uses separate documents with a container_id instead, which GroupingSearch can group on.

Make it robust before you point it at real files

In-process parsing shares your JVM with the parser: a malformed file can loop, exhaust the heap or crash the process, and with it the IndexWriter. For anything that indexes files you did not create, parse in a fork with PipesForkParser (see Tika Pipes) or call tika-server, and set write, embedded and timeout limits (see Robustness, Limits & Timeouts). Commit in batches rather than per file, and log the files that failed with the reason, so that a re-run can pick them up.

Metadata filters: emit only the indexed fields

A metadata filter rewrites the metadata list after parsing and before it is written or emitted. Filters are configured in tika-config.json, apply in tika-app, tika-server and Tika Pipes alike, and are the supported way to make Tika’s output match an index schema. The built-in ones live in tika-core (org.apache.tika.metadata.filter); the general reference is in Metadata and Configuration.

The Bookshelf index configuration drops pictures, strips the charset from the media type and renames the seven keys the index uses, dropping everything else:

{
  "metadata-filters": [
    { "remove-by-mime-metadata-filter": { "mimes": ["image/png", "image/jpeg"] } },
    {
      "capture-group-metadata-filter": {
        "sourceField": "Content-Type",
        "targetField": "Content-Type",
        "regex": "\\A([^;]+)"
      }
    },
    {
      "field-name-mapping-filter": {
        "excludeUnmapped": true,
        "mappings": {
          "tk:content": "content",
          "Content-Type": "content_type",
          "dc:title": "title",
          "dc:creator": "author",
          "dcterms:created": "created",
          "tk:embedded-id-path": "embedded_path",
          "tk:resource-name": "file_name"
        }
      }
    }
  ]
}
$ java -jar tika-app-4.1.0.jar --config=index-fields.json -J -t publisher-email.eml
[
  {"author": "Marta Ruiz <marta.ruiz@press.example>", "file_name": "Spring submissions: samples attached.eml",
   "created": "2026-10-12T09:30:00Z", "title": "Spring submissions: samples attached",
   "content": "\n\n...Spring submissions: samples attached\n\nHello,\n\nPlease find the spring samples...",
   "content_type": "message/rfc822"},
  {"title": "Chapter one, with a figure", "content": "...", "embedded_path": "/1/2",
   "content_type": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
   "author": "Ana Torres", "file_name": "chapter-one.docx"},
  {"author": "Bookshelf Press", "title": "Spring catalogue", "content": "...", "embedded_path": "/1/4",
   "content_type": "application/pdf", "file_name": "catalogue.pdf"},
  {"content": "...", "embedded_path": "/1/5", "content_type": "text/plain", "file_name": "notes/cover-note.txt"},
  {"file_name": "samples.zip", "content": "...", "embedded_path": "/1", "content_type": "application/zip"}
]

Source: Apache Tika 4.1.x, Metadata Filters and FieldNameMappingFilter, Javadoc 4.1.0. Run with the tika-app 4.1.0 classes on the class path; output shortened (...) and reformatted. Without the first filter the list had six entries, the sixth being figure1.png with only a type and a name.

Points the run confirms and the docs explain:

  • Order is literal. Each filter sees the previous one’s output. The rename runs last, so the capture-group filter is configured against Content-Type, not content_type. An allow-list (include-field-metadata-filter) placed before a filter that reads other fields removes what that filter needed.

  • excludeUnmapped defaults to true, so field-name-mapping-filter is an allow-list and a rename in one step. include-field-metadata-filter keeps fields under their Tika names, which is enough when the index uses Tika’s keys.

  • No section, no filtering. There is no default filter; an empty or missing metadata-filters emits every key.

  • Filters do not run by themselves in the Java API. RecursiveParserWrapper returns the raw list. Load the chain with TikaLoader and apply it, as the next example does.

  • CONTENT_ONLY parse mode in Pipes applies its own filter that keeps tk:content, and ignores the top-level metadata-filters chain; a parse-context filter list replaces it (see Tika Pipes).

  • Per-request lists replace, not add. A metadata-filters list in a request’s parse-context replaces the configured chain for that request.

The same chain applied in Java, for code that indexes in-process:

import java.nio.file.Path;
import java.util.List;

import org.apache.tika.config.loader.TikaLoader;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.parser.RecursiveParserWrapper;
import org.apache.tika.sax.BasicContentHandlerFactory;
import org.apache.tika.sax.RecursiveParserWrapperHandler;

public class FilteredForIndex {
    public static void main(String[] args) throws Exception {
        TikaLoader loader = TikaLoader.load(Path.of(args[0]));            // index-fields.json
        RecursiveParserWrapper parser = new RecursiveParserWrapper(loader.loadAutoDetectParser());
        RecursiveParserWrapperHandler handler = new RecursiveParserWrapperHandler(
                new BasicContentHandlerFactory(BasicContentHandlerFactory.HANDLER_TYPE.TEXT, 1_000_000));
        try (TikaInputStream tis = TikaInputStream.get(Path.of(args[1]))) {
            parser.parse(tis, handler, new Metadata(), loader.loadParseContext());
        }
        List<Metadata> list = handler.getMetadataList();
        loader.loadMetadataFilters().filter(list);                         // the library path runs no filter by itself
        for (Metadata m : list) {
            System.out.println(String.join(", ", m.names()));
        }
    }
}
author, file_name, created, title, content, content_type
title, content, embedded_path, content_type, author, file_name
author, title, content, embedded_path, content_type, file_name
content, embedded_path, content_type, file_name
file_name, content, embedded_path, content_type

Source: Apache Tika 4.1.x, Metadata Filters and the TikaCLI 4.1.0 source, which applies the chain the same way. TikaLoader is in tika-serialization. Run against Tika 4.1.0.

Solr

Solr offers two ways in: Solr Cell, which receives the file and calls Tika for you, and ordinary /update requests fed by something that already ran Tika, such as the Tika Pipes solr-emitter. For the update handlers themselves, commits and the Solr Cell parameter reference see Solr indexing & updates; this section covers the Tika side.

Solr Cell: /update/extract with a remote tika-server

Solr Cell is the ExtractingRequestHandler in Solr’s optional extraction module. Its history explains its current shape:

Solr How Solr Cell runs Tika

1.4 to 8.x

Tika jars inside Solr (a contrib, then a module); parsing ran in the Solr JVM.

9.x

extraction.backend chooses local (Tika in the Solr JVM, the default and deprecated) or tikaserver (an external tika-server at tikaserver.url).

10.0

Only tikaserver. tikaserver.url is required and set only in solrconfig.xml.

So in Solr 10 the Tika that parses your files is a tika-server you run, size and secure, and Solr is its HTTP client. Enable the module (SOLR_MODULES=extraction or -Dsolr.modules=extraction) and point the handler at the server:

<requestHandler name="/update/extract" class="solr.extraction.ExtractingRequestHandler">
  <str name="tikaserver.url">http://tika.bookshelf.internal:9998</str>
  <lst name="defaults">
    <str name="lowernames">true</str>
    <str name="fmap.content">content</str>
    <str name="uprefix">ignored_</str>
  </lst>
</requestHandler>
curl "http://localhost:8983/solr/bookshelf/update/extract?literal.id=catalogue.pdf&literal.source=intake&commit=true" \
  -F "file=@catalogue.pdf"

Source: Indexing with Solr Cell and Apache Tika, Solr Reference Guide 10.0 and the same page for Solr 9.10 (the local backend). Not run.

The parameters the example uses, in one line each (the full list is in the Solr guide):

  • literal.<field>=<value> adds a fixed field value: literal.id supplies the unique key, which a PDF does not have.

  • fmap.<source>=<target> renames an extracted field; fmap.content decides where the body text goes (the default configsets map it to _text_).

  • uprefix=ignored_ prefixes every field the schema does not know, so a dynamic ignored_* field can swallow them instead of the update failing on an unknown field.

  • lowernames=true lower-cases names and replaces characters that are awkward in Solr field names.

The Tika-side settings of the tikaserver backend:

Setting Default Meaning

tikaserver.url

required

Base URL of tika-server. Set in solrconfig.xml only.

tikaserver.timeoutSeconds

180

Per-request timeout towards tika-server.

tikaserver.maxChars

100 MB

Cap on the response read from tika-server; solrconfig.xml only.

tikaserver.recursive

false

true asks for the recursive (/rmeta) output so embedded documents are included.

passwordsFile, resource.password

none

Passwords for encrypted files, forwarded to tika-server. The file is clear text; restrict who can read it.

Source: Indexing with Solr Cell and Apache Tika, Solr Reference Guide 10.0.

Which tika-server version Solr Cell expects

The Solr 10.0 guide shows apache/tika:3.2.3.0-full in its examples and states no compatibility range. Reading the Solr 10.0.0 source of the tikaserver backend shows why the Tika version matters:

  • Non-recursive requests are PUT /tika with Accept: text/xml, and Solr parses the reply as XHTML. Tika 4.x removed Accept-header routing on /tika: a bare PUT /tika returns Markdown, and XHTML needs /tika/xml (see tika-server REST API).

  • Recursive requests are PUT /rmeta, and Solr reads the body text from the key X-TIKA:content. Tika 4.x writes tk:content.

  • Encrypted-file passwords are sent in a Password request header.

From this reading, Solr 10.0 Solr Cell expects a Tika 3.x server. Run it against the 3.3.x line (supported until early 2027; see the section index for the dates) until Solr documents 4.x support, or feed Solr from Tika 4.x through the Pipes emitter below. Configuring legacy-key-migration-filter on a 4.x server would restore the X-TIKA:content key for the recursive mode but not the XHTML default. This combination was not run for this page.

Source: TikaServerExtractionBackend, Solr 10.0.0 source and Migrating Tika Server to 4.x.

When Solr Cell fits

The Solr guide itself positions Solr Cell for moderate volumes at index time. It is convenient (one HTTP call per file, no extra code) but each upload holds a Solr update thread while tika-server parses, there is no per-file status store, and the extracted text cannot be enriched before indexing beyond what update request processors can do. Use it for user uploads and small collections; use Pipes or your own client for bulk loads, re-indexing and anything that needs OCR tuning or chunking. Either way the tika-server is a parsing service, not a security boundary: keep it on a private network and follow Security.

The Tika Pipes Solr emitter

The solr-emitter (plugin tika-pipes-solr, built on SolrJ 10, so Solr 8 is no longer supported) writes each parsed file to a collection through /update. A complete Bookshelf pipeline: walk the inbox, parse in two forks, keep the index fields, and emit to Solr.

{
  "content-handler-factory": {
    "basic-content-handler-factory": { "type": "TEXT", "writeLimit": 1000000, "throwOnWriteLimitReached": false }
  },
  "fetchers": {
    "inbox": { "file-system-fetcher": { "basePath": "/srv/bookshelf/inbox" } }
  },
  "emitters": {
    "solr": {
      "solr-emitter": {
        "solrCollection": "bookshelf",
        "solrUrls": ["http://solr1.bookshelf.internal:8983/solr"],
        "idField": "id",
        "commitWithin": 10000,
        "attachmentStrategy": "SEPARATE_DOCUMENTS",
        "updateStrategy": "ADD",
        "userName": "tika-indexer",
        "password": "${env:BOOKSHELF_SOLR_PASSWORD}"
      }
    }
  },
  "pipes-iterator": {
    "file-system-pipes-iterator": { "basePath": "/srv/bookshelf/inbox", "fetcherId": "inbox", "emitterId": "solr" }
  },
  "pipes": { "parseMode": "RMETA", "onParseException": "EMIT", "numClients": 2 },
  "metadata-filters": [
    { "remove-by-mime-metadata-filter": { "mimes": ["image/png", "image/jpeg"] } },
    { "field-name-mapping-filter": { "excludeUnmapped": true, "mappings": {
        "tk:content": "content", "Content-Type": "content_type", "dc:title": "title", "dc:creator": "author",
        "dcterms:created": "created", "tk:embedded-id-path": "embedded_path", "tk:resource-name": "file_name",
        "tk:exception:container-exception": "parse_error" } } }
  ]
}
java -jar tika-app-4.1.0.jar -a -c /srv/bookshelf/tika-config.json

Source: Apache Tika 4.1.x, Apache Solr Plugin (its pipes-solr-pipeline.json example), Metadata Filters and Configuration (the ${env:NAME} interpolation). Keys checked against the 4.1.0 documentation and SolrEmitter source; not run against Solr.

The emitter settings that matter for the index design:

Key Default Effect on the index

solrUrls, or solrZkHosts with solrZkChroot

one required

Direct node URLs, or ZooKeeper for SolrCloud routing. Setting both or neither fails validation.

solrCollection

required

Target collection or core.

idField

id

Receives the emit key (the file’s relative path from the iterator).

attachmentStrategy

PARENT_CHILD

PARENT_CHILD nests attachments under the container in the field named by embeddedFileFieldName (embedded); SEPARATE_DOCUMENTS writes them as top-level documents.

updateStrategy

ADD

ADD replaces a document with the same id. UPDATE_MUST_EXIST and UPDATE_MUST_NOT_EXIST send atomic set updates with _version_ 1 or -1, so Solr rejects a missing or an existing document.

commitWithin

1000 ms

Passed on every batch. One second is aggressive for bulk loads; raise it, or rely on autoSoftCommit.

userName, password

none

Basic authentication; read the password from the environment.

verifySsl

true

Keep it; add a private CA to the JVM trust store instead of disabling it.

Behaviour read from the 4.1.0 SolrEmitter source, which the schema has to accommodate:

  • Field names are the metadata keys, verbatim. Without a filter, Solr receives tk:content, dc:title, pdf:producer and every other key; colons in field names work but need escaping in queries, and a schemaless collection guesses a type for each. Rename with field-name-mapping-filter as above.

  • Every value is a string. created arrives as 2026-10-12T09:30:00Z, which a pdate field parses; numbers arrive as strings, which numeric field types also accept.

  • Attachment ids are random. An attachment’s id is the container id plus a hyphen and a random UUID, minted on every run. Re-emitting a container under SEPARATE_DOCUMENTS therefore adds a second copy of each attachment rather than replacing it. Delete the old ones first (for example by a container_id field you add with a metadata filter, or by id prefix), or use PARENT_CHILD, where replacing the parent replaces its block of children.

  • PARENT_CHILD needs a nested-document schema: _root_, and _nest_path_ for the labelled embedded children (see Indexing Nested Documents).

  • A batch either succeeds or fails. The emitter sends one UpdateRequest per batch with failOnVersionConflicts=false; a Solr error fails the batch, and the pipes status tells you which files were in it.

A matching set of fields for the collection, as Schema API calls:

curl -X POST -H 'Content-Type: application/json' http://localhost:8983/solr/bookshelf/schema -d '{
  "add-field": [
    {"name": "content",       "type": "text_general", "stored": false},
    {"name": "title",         "type": "text_general"},
    {"name": "author",        "type": "string", "multiValued": true},
    {"name": "created",       "type": "pdate"},
    {"name": "content_type",  "type": "string"},
    {"name": "file_name",     "type": "string"},
    {"name": "embedded_path", "type": "string"},
    {"name": "parse_error",   "type": "string", "indexed": false}
  ]
}'

Source: Schema API, Solr Reference Guide. Field design and analyzers are covered in Solr schema and fields and Solr text analysis. Not run.

The plugin also has a solr-pipes-iterator, which reads ids back out of a collection (for example everything matching status:NEEDS_REPARSE) so you can re-parse them after a Tika upgrade or a parser fix; pair it with updateStrategy: UPDATE_MUST_EXIST so a re-parse never creates a document that was deleted in the meantime. The tk:chunks field is not treated specially by the Solr emitter: it arrives as one JSON string, so vector search in Solr needs your own code to decode the vectors into a DenseVectorField (see Solr dense vector search).

Elasticsearch and OpenSearch

The Tika Pipes es-emitter and opensearch-emitter

The es-emitter (plugin tika-pipes-es, new in 4.0) and the opensearch-emitter (tika-pipes-opensearch, since 2.x) post each file’s metadata list to the index’s _bulk endpoint over REST. The configuration differs from Solr’s in the URL (it names the index) and the strategies:

{
  "emitters": {
    "bookshelf-es": {
      "es-emitter": {
        "esUrl": "https://search.bookshelf.internal:9200/bookshelf-docs",
        "idField": "id",
        "attachmentStrategy": "SEPARATE_DOCUMENTS",
        "updateStrategy": "OVERWRITE",
        "chunkStrategy": "INLINE",
        "apiKey": "${env:BOOKSHELF_ES_API_KEY}",
        "httpClientConfig": { "connectionTimeoutMillis": 10000, "socketTimeoutMillis": 60000 }
      }
    },
    "bookshelf-os": {
      "opensearch-emitter": {
        "openSearchUrl": "https://opensearch.bookshelf.internal:9200/bookshelf-docs",
        "idField": "id",
        "attachmentStrategy": "SEPARATE_DOCUMENTS",
        "updateStrategy": "OVERWRITE",
        "httpClientConfig": { "userName": "tika-indexer", "password": "${env:BOOKSHELF_OS_PASSWORD}" }
      }
    }
  },
  "pipes-reporters": {
    "es-pipes-reporter": {
      "esUrl": "https://search.bookshelf.internal:9200/bookshelf-intake-status",
      "includes": ["PARSE_SUCCESS_WITH_EXCEPTION", "PARSE_EXCEPTION_NO_EMIT", "OOM", "TIMEOUT"],
      "apiKey": "${env:BOOKSHELF_ES_API_KEY}"
    }
  }
}

Source: Apache Tika 4.1.x, Elasticsearch Plugin and OpenSearch Plugin. The fetcher, iterator and filters are as in the Solr pipeline. Not run against a cluster.

Key es-emitter opensearch-emitter Notes

Index URL

esUrl

openSearchUrl

Full URL including the index name; the emitter appends /_bulk.

idField

required

required

Validated but not read: _id is always the emit key, and an attachment’s _id is the emit key plus a random UUID.

attachmentStrategy

no default

no default

Set it. Unset, attachments get neither a parent field nor a join relation.

updateStrategy

UPSERT

UPSERT

OVERWRITE sends index actions; UPSERT sends update with doc_as_upsert, merging into an existing document.

chunkStrategy

INLINE

INLINE

Where tk:chunks goes; see tk:chunks.

commitWithin

ignored

ignored

Accepted but unused; visibility follows the index refresh_interval.

Authentication

apiKey (preferred) or basic auth in httpClientConfig

basic auth in httpClientConfig (required block)

The OpenSearch emitter reads only userName, password and verifySsl from httpClientConfig; its timeouts and proxy keys are not applied.

Reporter

es-pipes-reporter

opensearch-pipes-reporter

Writes one status record per file to a separate index, useful for an intake dashboard.

How the strategies map to Elasticsearch documents (read from the 4.1.0 ESClient source):

  • SEPARATE_DOCUMENTS: the container is written as is; each attachment gets a parent field holding the container’s emit key. Map parent as keyword and collapse or filter on it.

  • PARENT_CHILD: the container gets relation_type: container, each attachment relation_type: {name: <embeddedFileFieldName>, parent: <emit key>}, and every action is routed by the emit key. The index needs a join field and required routing:

PUT /bookshelf-docs
{
  "mappings": {
    "dynamic": "strict",
    "_routing": { "required": true },
    "properties": {
      "content":       { "type": "text" },
      "title":         { "type": "text", "fields": { "raw": { "type": "keyword" } } },
      "author":        { "type": "keyword" },
      "created":       { "type": "date" },
      "content_type":  { "type": "keyword" },
      "file_name":     { "type": "keyword" },
      "embedded_path": { "type": "keyword" },
      "parse_error":   { "type": "keyword", "index": false },
      "relation_type": { "type": "join", "relations": { "container": "embedded" } }
    }
  }
}

Source: Apache Tika 4.1.x, Elasticsearch Plugin, the elasticsearch-parent-child-mappings.json test resource of the Tika 4.1.0 ES integration tests, and Join field type, Elasticsearch reference. The relation name on the child side must equal embeddedFileFieldName; set it to embedded explicitly, because the ES emitter has no default for it. Not run.

"dynamic": "strict" makes the index reject any field you did not map, which turns a missing metadata filter into an immediate bulk error instead of a slowly growing mapping. Mapping details are in Elasticsearch mapping and field types, and joins and their cost in Elasticsearch joins and relationships.

Your own client: tika-server, then the bulk API

When the documents need enrichment Tika does not do (a catalogue lookup, access-control fields, a classification), let your service call tika-server and write to the index itself. /rmeta/text returns the same list of metadata as tika-app -J, so the transformation can be developed against either. With the metadata filters above configured on the server, a jq program turns the list into _bulk lines:

curl -s -T publisher-email.eml -H "Content-Disposition: attachment; filename=publisher-email.eml" \
  http://tika.bookshelf.internal:9998/rmeta/text > rmeta.json

jq -c --arg id publisher-email.eml '.[]
  | {index: {_id: ($id + (if .embedded_path then "!" + .embedded_path else "" end))}},
    (. + {container_id: $id, content: ((.content // "") | gsub("^\\s+|\\s+$"; ""))})' rmeta.json > bulk.ndjson

curl -s -H "Content-Type: application/x-ndjson" -H "Authorization: ApiKey $BOOKSHELF_ES_API_KEY" \
  -X POST "https://search.bookshelf.internal:9200/bookshelf-docs/_bulk" --data-binary @bulk.ndjson
{"index":{"_id":"publisher-email.eml"}}
{"author":"Marta Ruiz <marta.ruiz@press.example>","file_name":"Spring submissions: samples attached.eml",...}
{"index":{"_id":"publisher-email.eml!/1/2"}}
{"title":"Chapter one, with a figure","content":"Chapter one, with a figure\n\nChapter one\nThe lamp was lit at dusk...
{"index":{"_id":"publisher-email.eml!/1/4"}}
{"author":"Bookshelf Press","title":"Spring catalogue","content":"Spring catalogue\n\n\nBookshelf spring catalogue...
...

Source: Apache Tika 4.1.x, tika-server and Bulk API, Elasticsearch. The jq step was run on the tika-app -J output of the previous section (shown shortened); the two curl calls were not run. Unlike the emitter, this keeps stable attachment ids (!/1/4), so a re-run overwrites instead of duplicating.

A real client does the same in Java with the JDK HttpClient or Spring RestClient (see tika-server REST API) and the Elasticsearch Java client (see Elasticsearch clients and REST conventions), and handles tika-server’s 429 (all forks busy) with a back-off and 503 (crash, timeout or OOM in a fork) as a per-file failure.

The Elasticsearch attachment and OpenSearch ingest-attachment processors

Both engines can run Tika themselves, as an ingest processor: you send the file base64-encoded in a JSON field, and the processor extracts text and a few metadata fields before the document is indexed. In Elasticsearch the attachment processor is part of the default distribution since 8.4 (it was the ingest-attachment plugin before). In OpenSearch it is still a plugin, installed on every node with bin/opensearch-plugin install ingest-attachment.

PUT _ingest/pipeline/bookshelf-attachment
{
  "description": "Extract text and metadata from the base64 field 'data'",
  "processors": [
    {
      "attachment": {
        "field": "data",
        "target_field": "attachment",
        "indexed_chars": 200000,
        "properties": ["content", "title", "author", "date", "content_type", "language"],
        "remove_binary": true
      }
    }
  ]
}

PUT bookshelf-uploads/_doc/catalogue.pdf?pipeline=bookshelf-attachment
{ "data": "JVBERi0xLjQK..." }

Source: Attachment processor, Elasticsearch reference and Ingest-attachment plugin, OpenSearch documentation. Not run; the base64 value is truncated.

Option Default Meaning

field

required

Field with the base64-encoded file (or a CBOR byte array, which skips base64 decoding).

target_field

attachment

Object that receives the extracted fields (attachment.content, attachment.title and so on).

indexed_chars

100000

Characters of text kept; -1 for no limit, which needs heap for the largest document.

indexed_chars_field

none

A document field that overrides indexed_chars per document.

properties

all

Subset to store: content, title, name, author, keywords, date, content_type, content_length, language.

remove_binary

false

true drops the base64 field after extraction. Elasticsearch warns when it is not set; set it, or every document keeps its file in _source.

resource_name

none

Field with the file name, passed to Tika as a detection hint.

ignore_missing

false

true skips documents without field.

max_field_bytes (Elasticsearch)

-1

Size limit for the source field, checked before decoding; the node-wide ingest.attachment.max_field_size does the same for all pipelines.

The OpenSearch plugin documents field, target_field, properties, indexed_chars, indexed_chars_field and ignore_missing, and a smaller set of properties. For a document with an array of attachments, wrap the processor in a foreach processor and set target_field to _ingest._value.attachment, otherwise only the first element is extracted. Pipelines, on_failure handling and the _simulate API are covered in Elasticsearch ingest pipelines.

When to prefer an external Tika

The processors are convenient but put the parser in the cluster. The comparison decides most cases:

Concern In-cluster processor External Tika (Pipes, tika-server)

Isolation

Parses in the ingest node’s JVM. A pathological file costs heap and CPU on a node that also indexes and searches. Elastic recommends dedicated ingest nodes.

Forked JVMs per worker; crashes, timeouts and OOMs are contained and reported per file.

Tika version and parsers

Whatever the cluster release bundles (the OpenSearch page links Tika 3.2.2 format lists); you upgrade Tika by upgrading the cluster.

You choose the release, the parser configuration and the optional modules.

OCR, VLM, chunking

Not described for the processors.

Tesseract or a VLM recognizer, tk:chunks with embeddings (see tk:chunks).

Embedded documents

One text field per document; archives and e-mail attachments are not split into documents.

Separate or parent/child documents per attachment.

Transport

Base64 in the JSON body: about a third larger than the file, plus the binary in _source unless remove_binary.

The file stays in its store; only text and fields travel to the index.

Operations

Nothing extra to run.

One more service (or a batch job) to deploy, size and monitor.

Use the processor for occasional, small, trusted uploads where an extra service is not worth it. Use external Tika for bulk loads, untrusted files, archives and e-mail, OCR, vector search, or whenever you need to control the Tika version.

tk:chunks: chunk strategies and the mapping

Tika 4.x can split each document’s text into chunks and attach embedding vectors to them during the parse. The result is one metadata field, tk:chunks, holding a JSON array. How the chunks are produced (engines, TEXT, IMAGES, PAGES and MEDIA bindings, the Markdown chunker) is explained in VLM Parsers & Inference; this section is about getting them into Elasticsearch or OpenSearch. The inference module and the tk:chunks layout are documented as experimental and may change in a minor release.

A chunk carries a vector (base64 of big-endian float32 values), its text for text chunks, producer (the binding id), modality and locators (character offsets, a page, a time range, or the embedded file it was lifted from):

{
  "text": "Spring catalogue\n\nBookshelf spring catalogue Forty titles from independent publishers.",
  "vector": "PczMzT5MzM0+mZmaPszMzQ==",
  "producer": "text-vectors",
  "modality": "text",
  "locators": { "text": [ { "start_offset": 0, "end_offset": 88 } ] }
}

The Elasticsearch and OpenSearch emitters recognise tk:chunks and write it as raw JSON rather than as an escaped string. Their chunkStrategy decides the shape:

chunkStrategy What is indexed Search

INLINE (default)

The file’s document carries tk:chunks as an array of objects.

Map tk:chunks as nested and run a nested kNN query; a hit is the file, scored by its best chunk.

DOCUMENTS

The file’s document without tk:chunks, plus one document per unit, _id <file id>-chunk-N, with file_id, container_id, correlator, flattened locators (start_offset, end_offset, page, start_ms, end_ms, embedded_id_path) and each vector decoded to a float array under v.<producer>.

A hit is a chunk; collapse on container_id groups them back into files. Requires attachmentStrategy: SEPARATE_DOCUMENTS.

A mapping for INLINE, assuming a 1,024-dimension text embedding:

PUT /bookshelf-docs
{
  "mappings": {
    "properties": {
      "content":      { "type": "text" },
      "title":        { "type": "text" },
      "content_type": { "type": "keyword" },
      "parent":       { "type": "keyword" },
      "tk:chunks": {
        "type": "nested",
        "properties": {
          "text":     { "type": "text" },
          "producer": { "type": "keyword" },
          "modality": { "type": "keyword" },
          "vector":   { "type": "dense_vector", "dims": 1024, "index": true, "similarity": "cosine" },
          "locators": {
            "properties": {
              "text":      { "type": "nested", "properties": {
                               "start_offset": { "type": "integer" }, "end_offset": { "type": "integer" } } },
              "paginated": { "type": "nested", "properties": { "page": { "type": "integer" } } }
            }
          }
        }
      }
    }
  }
}

A mapping for DOCUMENTS (chunk documents share the index with the file documents):

PUT /bookshelf-docs
{
  "mappings": {
    "properties": {
      "file_id":      { "type": "keyword" },
      "container_id": { "type": "keyword" },
      "correlator":   { "type": "keyword" },
      "page":         { "type": "integer" },
      "start_offset": { "type": "integer" },
      "end_offset":   { "type": "integer" },
      "text":         { "type": "text" },
      "content":      { "type": "text" },
      "parent":       { "type": "keyword" },
      "v": { "properties": {
        "text-vectors": { "type": "dense_vector", "dims": 1024, "index": true, "similarity": "cosine" },
        "page-vectors": { "type": "dense_vector", "dims": 1024, "index": true, "similarity": "cosine" }
      } }
    }
  }
}

Source: Apache Tika 4.1.x, Chunk Strategies for Search Engines and Elasticsearch Plugin. The text-vectors and page-vectors names are the binding ids of the Bookshelf vector intake on the inference page; the field under v is always the producer. Not run against a cluster.

Points from the Tika documentation that decide between the two:

  • Vector encoding. Inline, the emitter writes each vector as the base64 string Tika produced. The Tika documentation says Elasticsearch 9.3 and later accept that encoding for dense_vector, and that earlier versions need an ingest pipeline to decode it or chunkStrategy: DOCUMENTS, which decodes to float arrays. This was not verified against an Elasticsearch release here; check your version before relying on it, and OpenSearch’s knn_vector separately.

  • Nested kNN caveats. Inline chunks need nested kNN support and make documents with many pages large. DOCUMENTS gives one hit per chunk and lets several knn clauses (for example text and page vectors) add up per chunk.

  • Re-indexing. DOCUMENTS replaces chunk documents by id but never deletes the surplus chunks of an earlier, longer run, and under UPSERT a removed binding’s v.<producer> field survives. Delete by container_id before re-indexing a changed file.

  • Grouping key. Collapse on container_id (the emit key, stable across runs), not file_id, which for an attachment is minted per run.

  • Images land on the parent. A picture’s vector inside a .docx or an e-mail body is written to the containing document with an embedded locator, so pictures do not become hits of their own; attachments keep their own chunks.

Solr and Lucene get tk:chunks as a JSON string. Decode it yourself and write one DenseVectorField (Solr) or KnnFloatVectorField (Lucene) document per chunk; the vector-store and RAG side is covered on RAG & LLM Ingestion and Elasticsearch vector and semantic search.

Re-indexing, updates and deletes

Indexes outlive Tika versions. Plan for re-parsing from the start:

  • Keep the originals (or a fetcher that can reach them). Text extraction improves between releases; a re-parse is the only way to benefit, and the index alone cannot be re-parsed.

  • Record what parsed each document. Index tk:parsed-by (or at least the Tika version) as a stored field, so you can select documents parsed by an old parser after a fix.

  • Make ids deterministic. Use the emit key or the relative path for containers. The emitters' random attachment ids mean attachments must be deleted by their container before a re-emit with SEPARATE_DOCUMENTS; the Lucene and jq examples above derive attachment ids from tk:embedded-id-path instead.

  • Handle deletes outside Tika. An iterator only lists files that exist. Deleting a file from the source needs a delete in the index (by id and by container), driven by your own change feed.

  • Re-parse from the index. The solr-pipes-iterator turns a Solr query into a work list; for Elasticsearch, a scroll or point-in-time query writing a JSON-lines file for the json-pipes-iterator does the same (see Tika Pipes).

Memory, performance and robustness

  • Parsing dominates. Indexing a document in Lucene, Solr or Elasticsearch takes milliseconds; parsing a large PDF or OCR-ing a scanned page takes seconds. Size the Tika side (forks, CPU, heap) for the parse and the index side for the write rate. See Performance & Scaling.

  • Bound the text. Set a write limit that matches the field. The full text of a large book can exceed what an Elasticsearch document should carry, and highlighting cost grows with field length.

  • Batch writes. The emitters batch through the pipes emit strategy; a client should send _bulk requests of a few megabytes and commit Solr with commitWithin or auto-commit rather than per file.

  • Isolate the parser. In-process Tika (the Lucene example, the in-cluster processors) shares the JVM with the index writer. Pipes forks or tika-server keep a bad file from taking the index down.

  • Track failures. Index tk:exception:container-exception into a non-indexed field, or use a pipes reporter, so that "file not found by search" can be answered with "file failed to parse".

Pitfalls and security

  • Unfiltered metadata. Emitting every key creates hundreds of fields, colon-named fields in Solr and mapping explosions in Elasticsearch. Always configure a metadata-filters allow-list.

  • Leaking restricted documents. Search exposes whatever was indexed, including text from embedded files, document properties, comments and hidden sheets. Add access-control fields (from your own system, not from the file) and filter every query by them.

  • Trusting file metadata. Titles, authors and dates come from the file. Do not use them for access control or as authoritative dates.

  • Solr Cell with the wrong Tika. The Solr 10 backend speaks the 3.x tika-server API, as described above.

  • Exposed tika-server. Solr Cell and your own clients call tika-server over HTTP. It has no authentication of its own and is not a security boundary; keep it on a private network or behind a gateway (see Security).

  • Secrets in configuration. Use ${env:NAME} for apiKey and password, as above, and keep verifySsl: true.

  • Base64 in _source. Without remove_binary: true the attachment processor leaves the whole file in every document.

  • Duplicate attachments on re-emit. See Re-indexing.

From the book to 4.x

Tika in Action (Tika 1.0, 2011) shows Tika feeding Lucene 3.0 in chapter 10 and describes Tika in Solr, Nutch and other Apache projects in the case-study chapters 12 to 15. Treat that material as history; the ideas carry over, the code does not.

Book era Today Where on this site

Lucene 3.0 Field with Field.Store and Field.Index.ANALYZED / NOT_ANALYZED

Lucene 10 field classes: TextField, KeywordField, LongField, StoredField, KnnFloatVectorField

Documents, fields & the capability matrix

Tika.parseToString into one Lucene field

RecursiveParserWrapper or Pipes RMETA: one index document per embedded file, with tk:content

Tika to Lucene

Solr Cell in contrib/extraction, Tika inside Solr

Solr 9 extraction module with local or tikaserver; Solr 10 tikaserver only

Solr Cell, Solr indexing & updates

Custom crawlers and batch code

Tika Pipes fetchers, iterators and the Solr, Elasticsearch and OpenSearch emitters (Pipes arrived in 2.0; the ES emitter in 4.0)

Tika Pipes

Metadata keys such as resourceName, X-TIKA:content (3.x)

tk:resource-name, tk:content; field-name-mapping-filter to rename for the index

Metadata

Elasticsearch, first released in 2010, is not covered by the book

attachment processor in the default Elasticsearch distribution since 8.4; OpenSearch ingest-attachment plugin

Attachment processors

Keyword search only

tk:chunks with embeddings for vector and hybrid search (4.x, experimental)

tk:chunks

The projects in the case studies have different fates. Apache Nutch, Jackrabbit and Mahout are active Apache projects; Apache OODT has been retired to the Apache Attic. Check each project’s site before reusing anything from those chapters.

What was verified and what was not

Run against Tika 4.1.0 (and Lucene 10.5.2 where relevant) on Java 25:

  • The BookshelfLuceneIndexer program and its output, including the stable document count on a second run.

  • The index-fields.json metadata-filter chain through tika-app 4.1.0 (-J -t), with and without remove-by-mime-metadata-filter, and the same chain applied in Java with TikaLoader.loadMetadataFilters().

  • The jq transformation to _bulk lines, on the tika-app -J output (the /rmeta/text format).

Read from sources, not run:

  • The Solr emitter behaviour (field names, string values, random attachment ids, atomic set updates, nested children) from the 4.1.0 SolrEmitter source; the Elasticsearch strategies (parent field, relation_type join, routing, doc_as_upsert, raw tk:chunks) from the 4.1.0 ESClient source and the integration-test mappings.

  • The Solr Cell tikaserver request shapes (PUT /tika with Accept: text/xml, /rmeta with X-TIKA:content, Password header) from the Solr 10.0.0 source. That Solr 10 Solr Cell fails against a Tika 4.x server is an inference from that source and the Tika 4.x server migration guide, not a tested result.

Not verified:

  • No configuration on this page was run against a live Solr, Elasticsearch or OpenSearch cluster; the Solr schema, the Elasticsearch mappings, the attachment pipeline and the curl calls are unrun.

  • The claim that Elasticsearch 9.3 and later accept base64 float32 vectors for dense_vector comes from the Tika documentation only.

  • The Elasticsearch attachment processor options come from the current Elastic reference; the "default distribution since 8.4" date comes from the Elasticsearch 8.4.0 release notes ("Move the ingest attachment processor to the default distribution", #87989). The OpenSearch option list comes from the plugin page (which documents fewer options and properties).

  • Which Tika version a given Elasticsearch or OpenSearch release embeds was not checked.

  • The book chapter numbers (10, 12 to 15) were not re-checked against a copy of the book.

References