Indexing operations & merge policies

This section documents the current Apache Lucene 10.x line — Lucene 10 requires Java 21 — as published at the Apache Lucene documentation and Javadoc, which is the reference these pages are written and verified against. No specific patch version is pinned; examples target lucene-core 10.x and the companion modules. Some areas (the Panama foreign-memory / Vector API internals, codec file-format internals, and the nightly benchmark harness) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, as Lucene iterates quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

IndexWriter is the only class that changes an index. It buffers documents in RAM, flushes them to immutable segments, records deletes as a side file, and — through a merge policy and a merge scheduler — rewrites small or delete-heavy segments into larger clean ones in the background. This page covers the write operations, the two-phase commit protocol, the IndexWriterConfig settings that matter, and how merging is controlled.

Writing documents

Every write goes through one long-lived IndexWriter. addDocument appends; updateDocument atomically deletes every document matching a Term and then adds the replacement in one operation, so a reader never sees zero or two copies; deleteDocuments takes either Term s or Query s. Lucene has no true in-place update — an "update" is always a delete plus an add, and the old document’s space is reclaimed only when its segment is later merged.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/IndexWriter.html
var config = new IndexWriterConfig(analyzer);
try (var dir = FSDirectory.open(Path.of("/var/index/products"));
     var writer = new IndexWriter(dir, config)) {

    var doc = new Document();
    doc.add(new StringField("id", "p-1", Field.Store.YES));
    doc.add(new TextField("name", "Trail running shoe", Field.Store.YES));
    doc.add(new IntPoint("stock", 41));
    doc.add(new NumericDocValuesField("stock", 41));
    writer.addDocument(doc);

    // Replace p-1 in one atomic step (delete-by-Term + add).
    var updated = new Document();
    updated.add(new StringField("id", "p-1", Field.Store.YES));
    updated.add(new TextField("name", "Trail running shoe v2", Field.Store.YES));
    writer.updateDocument(new Term("id", "p-1"), updated);

    writer.deleteDocuments(new Term("id", "p-legacy"));
    writer.deleteDocuments(IntPoint.newRangeQuery("stock", 0, 0)); // delete-by-Query
    writer.commit();
}

Soft deletes

A soft delete marks a document as deleted for search but keeps it retrievable until a retention rule lets it go — the mechanism behind Elasticsearch’s _seq_no recovery and useful for change-data-capture or "show me the previous version" features. Enable it by naming a field with IndexWriterConfig.setSoftDeletesField(…​) and pairing it with a SoftDeletesRetentionMergePolicy that decides which soft-deleted docs a merge may finally drop. softUpdateDocument then replaces a document while writing the soft-delete marker instead of a hard delete.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/SoftDeletesRetentionMergePolicy.html
var config = new IndexWriterConfig(analyzer);
config.setSoftDeletesField("__soft_deleted");
config.setMergePolicy(new SoftDeletesRetentionMergePolicy(
        "__soft_deleted",
        () -> LongPoint.newRangeQuery("version", 5L, Long.MAX_VALUE), // keep versions >= 5
        new TieredMergePolicy()));

try (var writer = new IndexWriter(dir, config)) {
    var v = new Document();
    v.add(new StringField("id", "p-1", Field.Store.YES));
    v.add(new LongPoint("version", 6L));
    v.add(new TextField("name", "Trail running shoe v3", Field.Store.YES));
    // Marker field flags the OLD p-1 as soft-deleted rather than removing it.
    writer.softUpdateDocument(new Term("id", "p-1"), v,
            new NumericDocValuesField("__soft_deleted", 1));
}

Commits, flushes & rollback

A flush pushes the RAM buffer into a new on-disk segment; it happens automatically as the buffer fills and is not durable on its own. A commit fsync s all pending segments plus a new segments file, producing a recovery point that survives a crash and becomes visible to a freshly opened DirectoryReader. Near-real-time readers opened on the writer see flushed-but-uncommitted changes too — see Near-real-time search.

prepareCommit then commit is a two-phase commit: prepareCommit does all the expensive fsync work, after which the final commit is cheap and near-atomic, which lets an IndexWriter participate in a distributed transaction alongside another resource. rollback closes the writer and discards everything written since the last commit.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/IndexWriter.html#prepareCommit()
writer.addDocument(doc);

// Attach opaque metadata to the commit point (checkpoint, source offset, schema hash...).
writer.setLiveCommitData(Map.of(
        "kafka.offset", "48213",
        "indexer.version", "2025.09").entrySet());

writer.prepareCommit();   // phase 1: fsync segments + prepare a new segments_N
try {
    externalResource.commit();
    writer.commit();      // phase 2: cheap, flips the pointer
} catch (Exception e) {
    writer.rollback();    // discards since the last successful commit; writer is now closed
    throw e;
}

Read commit user data back with DirectoryReader.open(dir).getIndexCommit().getUserData().

Tuning IndexWriterConfig

IndexWriterConfig is single-use — one config per IndexWriter — and every setter returns this. The knobs that change throughput and segment shape:

Setting Effect

setOpenMode(OpenMode.CREATE | APPEND | CREATE_OR_APPEND)

Whether to overwrite, require an existing index, or create-if-absent (the default).

setRAMBufferSizeMB(double)

Flush when buffered documents reach this heap size. The primary throughput lever; the default is 16 MB, and large bulk loads often use 256 MB or more. Takes precedence over a document count.

setMaxBufferedDocs(int)

Flush after this many documents regardless of size. Disabled by default; set one or the other, not usually both.

setUseCompoundFile(boolean)

Pack each small segment’s files into a single .cfs file (default true) to limit open file handles; large merged segments stay multi-file regardless.

setMergePolicy(…​) / setMergeScheduler(…​)

Covered below.

setIndexSort(Sort)

Physically order documents within every segment (see below).

 
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/IndexWriterConfig.html
var config = new IndexWriterConfig(analyzer)
        .setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND)
        .setRAMBufferSizeMB(256.0)
        .setUseCompoundFile(true);

forceMerge is not a routine operation

forceMerge(1) rewrites the whole index into a single segment. It was called optimize before Lucene 4 and the rename was deliberate: it is enormous I/O, it temporarily needs up to 3x the index size on disk, and the one giant segment it produces is never itself merged again, so deletes accumulate in it with nothing to reclaim them. Only run it on an index that is now read-only. forceMergeDeletes() is the narrower, occasionally-justified cousin — it rewrites only segments whose deleted-document percentage is high.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/IndexWriter.html#forceMerge(int)
// Do this ONLY on an index that will not be written to again.
writer.forceMerge(1);

// Steady-state alternative: reclaim space in delete-heavy segments without collapsing to one.
writer.forceMergeDeletes();

Index-time sorting

setIndexSort(Sort) stores documents inside each segment in a chosen order (for example newest first). Range and top-N queries that follow the same sort can then stop early, and the sort is preserved across merges. The constraint: updateDocument(Iterable) blocks — parent/child documents indexed as a contiguous group — are only compatible with an index sort when the sort is on the parent documents, because a block must stay contiguous after the segment is reordered.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/IndexWriterConfig.html#setIndexSort(org.apache.lucene.search.Sort)
config.setIndexSort(new Sort(new SortField("timestamp", SortField.Type.LONG, true)));
// Requires a matching NumericDocValuesField("timestamp", ...) on every document.

Why merging exists, and the policies that drive it

Segments are immutable and deletes are recorded out-of-band, so a busy index drifts toward many small segments each carrying dead documents — more files to open, more term dictionaries to seek, wasted disk. Merging is the compaction that fixes this: a merge policy (findMerges) decides which segments to combine, and a merge scheduler decides when and on which threads the merge runs.

Merge policy Use

TieredMergePolicy

The default. Groups segments into size tiers and merges non-adjacent segments within a tier, so it copes well with the irregular segment sizes a real workload produces.

LogByteSizeMergePolicy

Merges only adjacent segments, which preserves insertion order on disk. Chiefly of interest when you need that ordering and are not using setIndexSort.

NoMergePolicy

Never merges. For a one-shot build where you control merging manually, or a throwaway index.

FilterMergePolicy

A pass-through wrapper to override one method of another policy (the base class of SoftDeletesRetentionMergePolicy, OneMergeWrappingMergePolicy, and others).

SoftDeletesRetentionMergePolicy

Wraps another policy and holds back soft-deleted documents that a retention Query still selects (see soft deletes above).

TieredMergePolicy 's main knobs: setMaxMergedSegmentMB (default ~5 GB) caps how large a merged segment may get, which bounds worst-case merge cost; setSegmentsPerTier (default 10) sets how many segments accumulate in a tier before a merge is triggered — lower means fewer segments but more merge work; setDeletesPctAllowed (default 20, min 5) is the index-wide dead-document percentage above which the policy merges purely to reclaim deletes.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/TieredMergePolicy.html
var mergePolicy = new TieredMergePolicy();
mergePolicy.setMaxMergedSegmentMB(2_048);   // cap merged segments at 2 GB
mergePolicy.setSegmentsPerTier(10);
mergePolicy.setDeletesPctAllowed(15);       // merge sooner to keep deletes low
config.setMergePolicy(mergePolicy);

The scheduler is a separate choice. ConcurrentMergeScheduler (default) runs merges on background threads and throttles their I/O so they do not starve searches; setMaxMergesAndThreads(maxMerges, maxThreads) sizes it, and it auto-tunes the I/O rate for SSDs. SerialMergeScheduler runs each merge on the calling thread — deterministic, used mostly in tests.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/ConcurrentMergeScheduler.html
var scheduler = new ConcurrentMergeScheduler();
scheduler.setMaxMergesAndThreads(4, 2);     // queue up to 4 merges, run 2 at once
config.setMergeScheduler(scheduler);
flowchart TD A[Segment flushed or a merge finished] --> B[MergeScheduler calls MergePolicy.findMerges] B --> C[Sort segments by size, group into tiers] C --> D{A tier exceeds
segmentsPerTier segments?} D -- yes --> F[Choose the lowest-cost eligible merge
skipping segments past maxMergedSegmentMB] D -- no --> E{Index-wide deletes
above deletesPctAllowed?} E -- yes --> F E -- no --> G[No merge this round] F --> H[ConcurrentMergeScheduler runs it
on a background thread, I/O throttled] H --> A

Tuning merges for bulk load vs steady state

For a large one-shot load, fewer, larger flushes and deferred merging win: raise setRAMBufferSizeMB (256 MB+), leave the default TieredMergePolicy, give ConcurrentMergeScheduler more threads, and call forceMergeDeletes() (not forceMerge(1)) once at the end only if the load produced many deletes. For a steady-state index taking continuous writes, keep the defaults and let background merging run — lowering setSegmentsPerTier or setDeletesPctAllowed trades extra merge I/O for a leaner index and faster searches. Solr’s partial-update and commit tuning and Elasticsearch’s bulk indexing both sit directly on top of this machinery.

Related pages