Architecture & data flow

This section documents the current Apache Lucene 10.x line — Lucene 10 requires Java 21 — as published at the Apache Lucene documentation and Javadoc, which is the reference these pages are written and verified against. No specific patch version is pinned; examples target lucene-core 10.x and the companion modules. Some areas (the Panama foreign-memory / Vector API internals, codec file-format internals, and the nightly benchmark harness) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, as Lucene iterates quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

A Lucene index is a directory of immutable segment files, written by an IndexWriter and read through a point-in-time DirectoryReader. This page describes the on-disk model, the pluggable codec that defines the file formats, and the two class chains — one for indexing, one for searching — that meet at the Directory.

The index: a Directory of immutable segments

An index is one Directory holding one or more segments. A segment is a complete mini-index over a subset of the documents, and once written it is never modified — only merged away or deleted. At its core each segment holds an inverted index: the analyzed terms, and for every term the list of documents that contain it (the postings list, with frequencies and positions).

Three source documents are analyzed into lowercased term streams with positions; the inverted index then stores each term once — brown

Each segment is a set of files sharing a name prefix (_0, _1, …​), one file per data structure:

Files Structure

.tim / .tip / .doc / .pos

Term dictionary and postings (which documents contain a term, with frequencies and positions).

.fdt / .fdx

Stored field values and their index, returned by storedFields().

.dvd / .dvm

Doc values — the columnar per-document values used for sort, facet, and function scoring.

.kdd / .kdi / .kdm

BKD trees for *Point fields (numeric and geo range search).

.tvd / .tvx

Term vectors (per-document term lists, used by some highlighters).

.nvd / .nvm

Norms (per-field length normalization for scoring).

.vec / .vex / .vem

Nearest-neighbour vector data for Knn*VectorField.

.si

SegmentInfo: the segment’s metadata and which codec wrote it.

.liv

Live-docs bitset — deletions are recorded here, not by rewriting the segment.

Deletes and updates never edit a segment in place: a delete flips a bit in .liv, and an updated document is a delete plus a fresh add in a new segment. Space is reclaimed only when merges rewrite several segments into one.

Generations and commits

The file segments_N (the segment infos, class SegmentInfos) lists the segments that make up one commit — a durable, consistent point-in-time view. N is the generation: every commit writes a new segments_N with a higher number, and a DirectoryReader opened on the directory reads the highest one. IndexWriter.commit() fsyncs the segment files and then the new segments_N; until then, new segments are visible only to a near-real-time reader (below), not after a crash.

// Inspect the latest commit without opening a full reader.
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/SegmentInfos.html
import org.apache.lucene.index.SegmentInfos;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;
import java.nio.file.Path;

try (Directory dir = FSDirectory.open(Path.of("/var/data/books-index"))) {
    SegmentInfos infos = SegmentInfos.readLatestCommit(dir);
    System.out.println("generation   = " + infos.getGeneration());   // the N in segments_N
    System.out.println("segments     = " + infos.size());
    System.out.println("total docs   = " + infos.totalMaxDoc());
    infos.forEach(c -> System.out.println("  " + c.info.name + "  codec=" + c.info.getCodec().getName()));
}

The pluggable codec

Every file format above is produced by a Codec — an assembly of a PostingsFormat, DocValuesFormat, KnnVectorsFormat, and the rest. The current default is the Lucene<NN>Codec class (e.g. Lucene100Codec on the 10.0.x line — each minor release ships its own numbered default); Codec.getDefault() returns it. Because the codec that wrote a segment is recorded in its .si file, an index can contain segments written by different codecs at once — which is how an older segment stays readable after an upgrade.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/codecs/package-summary.html
// current default codec: https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/codecs/lucene100/Lucene100Codec.html
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.index.IndexWriterConfig;

Codec current = Codec.getDefault();               // e.g. "Lucene100"
System.out.println(current.getName());

// Pin a codec explicitly (e.g. to keep a format stable across a minor upgrade).
IndexWriterConfig cfg = new IndexWriterConfig(analyzer);
cfg.setCodec(current);

lucene-backward-codecs supplies the read-only codecs of the previous major version, and Lucene’s compatibility rule is exactly one major version back: Lucene 10 reads indexes written by Lucene 9, but not Lucene 8. To move further, run IndexUpgrader once per major hop (it rewrites every segment with the current codec), or reindex. CheckIndex verifies an index’s structural integrity and can drop unreadable segments as a last resort.

// One-time upgrade of an index written by the previous major version.
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/IndexUpgrader.html
import org.apache.lucene.index.IndexUpgrader;
import org.apache.lucene.index.CheckIndex;
import org.apache.lucene.store.FSDirectory;
import java.nio.file.Path;

try (var dir = FSDirectory.open(Path.of("/var/data/old-index"))) {
    new IndexUpgrader(dir).upgrade();             // rewrites all segments with the current codec

    try (CheckIndex checker = new CheckIndex(dir)) {
        CheckIndex.Status status = checker.checkIndex();
        System.out.println("clean = " + status.clean);
    }
}
// CLI equivalent: java -cp lucene-core.jar org.apache.lucene.index.CheckIndex /var/data/old-index
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/CheckIndex.html

See the Elasticsearch documents & indices page for how a server layers a "logical index" and shards over this same segment model.

The indexing chain and the search chain

Two chains of objects meet at the Directory.

Left lane: Document and Field feed an Analyzer held by IndexWriterConfig

Indexing: a Document is an ordered list of Field objects (Documents, fields & the capability matrix). An IndexWriterConfig carries the Analyzer and the other write-time settings; an IndexWriter built from it and a Directory buffers added documents in RAM, flushes them to a new segment when the buffer fills, and merges segments in the background.

Searching: DirectoryReader.open(dir) (or .open(writer) for near-real-time) produces an immutable IndexReader over one commit. An IndexSearcher wraps that reader, rewrites a Query against it, and returns TopDocs — an array of ScoreDoc (doc id + score). Field values for the hits come from searcher.storedFields(); see Retrieving results.

// Both chains, side by side.
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/package-summary.html
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/search/package-summary.html
IndexWriterConfig cfg = new IndexWriterConfig(new StandardAnalyzer());
try (IndexWriter writer = new IndexWriter(dir, cfg)) {
    Document d = new Document();
    d.add(new TextField("title", "Lucene in the small", Field.Store.YES));
    writer.addDocument(d);
    writer.commit();
}

try (DirectoryReader reader = DirectoryReader.open(dir)) {
    IndexSearcher searcher = new IndexSearcher(reader);
    TopDocs hits = searcher.search(new TermQuery(new Term("title", "lucene")), 10);
    StoredFields stored = searcher.storedFields();
    for (ScoreDoc sd : hits.scoreDocs) {
        System.out.println(stored.document(sd.doc).get("title"));
    }
}

Reader lifecycle and reference counting

An IndexReader is reference-counted. Opening one sets its count to 1; incRef() adds a hold and decRef() releases one, and the reader closes its files only when the count reaches 0. This lets a new searcher take over while in-flight searches still hold the old reader. DirectoryReader.openIfChanged(old) returns a new reader that shares unchanged segment readers with the old one and null if nothing changed — the basis of SearcherManager, covered in Near-real-time search.

// Hand a reader to another thread safely: hold a reference, release it in finally.
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/IndexReader.html
reader.incRef();
try {
    IndexSearcher searcher = new IndexSearcher(reader);
    searcher.search(query, 10);
} finally {
    reader.decRef();          // never close() a shared reader directly
}

// Swap in a fresher view without disturbing running searches.
DirectoryReader refreshed = DirectoryReader.openIfChanged(reader);
if (refreshed != null) {
    reader.decRef();          // drop our hold on the old one
    reader = refreshed;
}

From buffered documents to merged segments

Adds accumulate in an in-memory buffer. A flush (buffer full, an explicit flush(), or a commit()) turns the buffer into one new on-disk segment. Because every flush makes another segment and a search must visit them all, the MergePolicy continuously picks small segments and rewrites them into fewer, larger ones in the background; deleted documents are dropped during the merge. This flush-then-merge cycle is what Indexing & merge policies tunes, and it is the Lucene-level view of Solr’s indexing internals & performance.

flowchart TD A["addDocument: in-RAM buffer"] -->|buffer fills / flush / commit| B["flush: write one new segment"] B --> C["Directory: many small segments"] C -->|MergePolicy selects| D["merge: rewrite N segments into 1"] D --> E["Directory: fewer, larger segments, deletes purged"] E -->|more adds| A B -.->|commit fsyncs| F["segments_N: durable commit point"]
// The MergePolicy is a write-time setting; the default is TieredMergePolicy.
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/MergePolicy.html
IndexWriterConfig cfg = new IndexWriterConfig(analyzer);
System.out.println(cfg.getMergePolicy());        // TieredMergePolicy(...)

try (IndexWriter writer = new IndexWriter(dir, cfg)) {
    // ... bulk load ...
    writer.commit();
    // Optional one-off consolidation after a load: collapse to a single segment.
    // Expensive -- not something to run routinely.
    writer.forceMerge(1);
}

Related pages