Directory implementations & storage

This section documents the current Apache Lucene 10.x line — Lucene 10 requires Java 21 — as published at the Apache Lucene documentation and Javadoc, which is the reference these pages are written and verified against. No specific patch version is pinned; examples target lucene-core 10.x and the companion modules. Some areas (the Panama foreign-memory / Vector API internals, codec file-format internals, and the nightly benchmark harness) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, as Lucene iterates quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

A Directory is Lucene’s abstraction over the place index files live — almost always a filesystem directory. It exposes create/open/delete/rename plus a lock, and hides whether reads go through mmap, positional read, or a byte array. Choosing the right implementation and understanding how it uses memory is most of what storage tuning involves.

Directory implementations

Call FSDirectory.open(Path) and let Lucene pick: on a 64-bit JVM it returns MMapDirectory, which is the right default on every supported platform. The concrete types:

Implementation When it is used

MMapDirectory

Memory-maps each file into the process address space through the Java Platform’s foreign-memory (Panama) API, as MemorySegment slices of up to 16 GiB each. Reads become ordinary memory access served from the OS page cache; no per-read syscall. The default from FSDirectory.open on 64-bit JVMs.

NIOFSDirectory

Positional FileChannel.read per access. The fallback when memory-mapping is unwanted (e.g. a 32-bit JVM, or an environment that limits mapped memory). Slower under load than MMapDirectory.

ByteBuffersDirectory

Holds the whole "index" in heap byte buffers. Not persistent — for tests and short-lived in-memory indexes. This is the modern replacement for the removed RAMDirectory; do not look for RAMDirectory in 10.x.

 
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/store/FSDirectory.html
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/store/MMapDirectory.html
import org.apache.lucene.store.*;
import java.nio.file.Path;

// Recommended: let FSDirectory choose (MMapDirectory on a 64-bit JVM).
Directory dir = FSDirectory.open(Path.of("/var/data/idx"));

// Force a specific implementation only for a concrete reason.
Directory nio = new NIOFSDirectory(Path.of("/var/data/idx"));

// In-memory, non-persistent -- tests and scratch indexes.
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/store/ByteBuffersDirectory.html
Directory mem = new ByteBuffersDirectory();

MMapDirectory can be told which files to keep resident and which to leave on-demand; see its Javadoc for setPreload and the group-by-extension preload predicate. For test code, the test framework's newDirectory() returns a randomly chosen, assertion-wrapped implementation so tests exercise all of them.

Locking and write.lock

At most one IndexWriter may hold an index open at a time. On open, the writer acquires write.lock in the directory through its LockFactory; a second writer — in this JVM or another process — fails with LockObtainFailedException. FSDirectory defaults to NativeFSLockFactory, which uses an OS advisory file lock that the kernel releases even if the JVM is killed, so a stale lock file left on disk is not by itself a problem.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/store/LockFactory.html
import org.apache.lucene.store.*;
import java.nio.file.Path;

// Default: native advisory lock, released by the OS on process death.
Directory dir = FSDirectory.open(Path.of("/var/data/idx"));   // NativeFSLockFactory

// Explicit choice, e.g. a single-JVM in-process guard.
Directory jvm = FSDirectory.open(Path.of("/var/data/idx"), new SingleInstanceLockFactory());

try (IndexWriter w = new IndexWriter(dir, new IndexWriterConfig(analyzer))) {
    // ... only writer holding /var/data/idx ...
}   // close() releases write.lock

// Recover from a crashed writer ONLY after proving no live writer exists.
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/IndexWriter.html#isLocked(org.apache.lucene.store.Directory)
if (IndexWriter.isLocked(dir)) {
    // investigate; do not blindly delete write.lock while a writer may be running
}

Never point two writers at one directory expecting them to cooperate — that is a job for a search server (Solr’s update concurrency, Elasticsearch bulk indexing) or for your own single-writer design.

Prefetch and ReadAdvice

Lucene 10 added I/O hints on the read path. IndexInput#prefetch(offset, length) asks the OS to start pulling a byte range into the page cache (via madvise(MADV_WILLNEED) on Linux) before Lucene actually reads it, overlapping I/O with computation — query execution uses this to warm postings and doc-values blocks it is about to touch. The ReadAdvice enum (NORMAL, RANDOM, SEQUENTIAL, RANDOM_PRELOAD) is set per IndexInput open via an IOContext so the OS can tune read-ahead: bulk merge reads are SEQUENTIAL, term-dictionary lookups are RANDOM.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/store/IndexInput.html
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/store/ReadAdvice.html
import org.apache.lucene.store.*;

try (Directory dir = FSDirectory.open(java.nio.file.Path.of("/var/data/idx"))) {
    IOContext ctx = IOContext.DEFAULT.withReadAdvice(ReadAdvice.RANDOM);
    try (IndexInput in = dir.openInput("_0.dvd", ctx)) {
        in.prefetch(0L, Math.min(in.length(), 1 << 20));   // hint: read the first 1 MiB soon
        // ... seek + read; the page cache is already being warmed ...
    }
}

Most code never calls these directly — the codecs do — but the ReadAdvice default is worth knowing when profiling merge or cold-cache behaviour.

Storage uses the OS page cache, not the heap

MMapDirectory reads are served from mapped pages the operating system caches outside the JVM heap — the class Javadoc, MMapDirectory, spells this out. The practical sizing rule is the inverse of most Java services: give the JVM only the heap it needs (query structures, IndexWriter RAM buffer, doc-values accessors) and leave the rest of physical memory free for the OS to cache index files. An oversized -Xmx starves the page cache and makes search slower. Mapped memory also shows up as large virtual size (VIRT) and shared resident pages — expected, not a leak.

// Nothing special to configure -- FSDirectory.open already returns MMapDirectory on a 64-bit JVM,
// and its reads come from OS-cached pages, not the heap. Size -Xmx small; leave RAM for the cache.
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/store/MMapDirectory.html
Directory dir = FSDirectory.open(Path.of("/var/data/idx"));
System.out.println(dir.getClass().getSimpleName());   // MMapDirectory

// A 32 GiB host serving a 40 GiB index: -Xmx6g for the app leaves ~26 GiB of page cache for
// hot index files. Raising -Xmx to 24g would evict those pages and regress query latency.

Sizing the IndexWriter RAM buffer, warming, and merge-time I/O are covered in Performance tuning.

Related pages