Test framework, tools & a module map

This section documents the current Apache Lucene 10.x line — Lucene 10 requires Java 21 — as published at the Apache Lucene documentation and Javadoc, which is the reference these pages are written and verified against. No specific patch version is pinned; examples target lucene-core 10.x and the companion modules. Some areas (the Panama foreign-memory / Vector API internals, codec file-format internals, and the nightly benchmark harness) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, as Lucene iterates quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

Lucene ships far more than lucene-core. This page covers the tooling you reach for when working on an index rather than through it — the randomized test framework, the corruption checker, the version upgrader, the Luke GUI, and the benchmark, replicator and misc modules — and ends with a one-line description of every module so you know what exists before you go looking.

The lucene-test-framework

lucene-test-framework is the harness Lucene tests itself with, published for you to use the same way. Add it in test scope:

<dependency>
  <groupId>org.apache.lucene</groupId>
  <artifactId>lucene-test-framework</artifactId>
  <version>10.0.0</version>
  <scope>test</scope>
</dependency>
<!-- https://lucene.apache.org/core/10_0_0/test-framework/index.html -->

Extend LuceneTestCase and you inherit a RandomizedRunner that seeds every run and randomizes the codec, Directory implementation, locale, time zone, and merge policy, printing the seed on failure so you can reproduce it with -Dtests.seed=…​. newDirectory() returns a randomly chosen, assertion-heavy Directory that also verifies every file is closed. RandomIndexWriter is a drop-in for IndexWriter that randomly flushes, commits, and reopens mid-test to shake out segment-boundary bugs.

// https://lucene.apache.org/core/10_0_0/test-framework/org/apache/lucene/tests/util/LuceneTestCase.html
public class TitleSearchTest extends LuceneTestCase {

  public void testFindsByTitle() throws Exception {
    try (Directory dir = newDirectory();                              // random Directory impl
         RandomIndexWriter w = new RandomIndexWriter(random(), dir)) { // random flush/commit
      Document doc = new Document();
      doc.add(new TextField("title", "the left hand of darkness", Field.Store.YES));
      w.addDocument(doc);
      try (IndexReader r = w.getReader()) {
        IndexSearcher s = newSearcher(r);                             // random concurrency slices
        assertEquals(1, s.count(new TermQuery(new Term("title", "darkness"))));
      }
    }
  }
}

For analyzer tests, BaseTokenStreamTestCase asserts a TokenStream produces an exact token / offset / position sequence, and independently fuzzes it with random strings for reset() / close() correctness:

// https://lucene.apache.org/core/10_0_0/test-framework/org/apache/lucene/tests/analysis/BaseTokenStreamTestCase.html
public class MyAnalyzerTest extends BaseTokenStreamTestCase {
  public void testTokens() throws Exception {
    Analyzer a = new StandardAnalyzer();               // no stop words by default
    assertAnalyzesTo(a, "The 2 QUICK Brown-Foxes",
        new String[] {"the", "2", "quick", "brown", "foxes"});
    checkRandomData(random(), a, 200);      // fuzz with 200 random inputs
    a.close();
  }
}

Tag slow or large tests @Nightly (or @Slow); they run only with -Dtests.nightly=true, keeping the default build fast. The randomized engine underneath is documented at randomizedtesting.

CheckIndex: detect and triage corruption

CheckIndex walks every segment and verifies the term dictionary, postings, doc-values, points, norms, and stored fields against their checksums. Run it from the command line against a closed index:

java -ea -cp lucene-core-10.0.0.jar \
  org.apache.lucene.index.CheckIndex /var/lib/app/index -verbose

# add -exorcise to drop unrecoverable segments (destructive; back up first)

Or call it from a health check and act on the result:

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/CheckIndex.html
try (Directory dir = FSDirectory.open(path);
     CheckIndex checker = new CheckIndex(dir)) {
  CheckIndex.Status status = checker.checkIndex();
  if (!status.clean) {
    log.error("index has broken segments: {} docs would be lost", status.totLoseDocCount);
  }
}

IndexUpgrader and lucene-backward-codecs

Lucene reads indexes written by the previous major version only. lucene-backward-codecs (a transitive dependency of lucene-core) carries the read-only codecs that make that one-version bridge work; IndexUpgrader rewrites every segment with the current codec so the index stays readable after the next major upgrade.

# One hop per major version: 8.x -> 9.x on Lucene 9, then 9.x -> 10.x on Lucene 10.
java -cp lucene-core-10.0.0.jar:lucene-backward-codecs-10.0.0.jar \
  org.apache.lucene.index.IndexUpgrader -verbose /var/lib/app/index
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/IndexUpgrader.html
new IndexUpgrader(dir).upgrade();
// module: https://lucene.apache.org/core/10_0_0/backward-codecs/index.html

Skipping a major version means exporting and reindexing from source instead.

Luke: browse an index by hand

The luke module is a Swing GUI for opening an index directory and inspecting it — term frequencies, per-field statistics, document contents, the effect of an analyzer on a string, and ad-hoc queries. It ships in the Lucene binary distribution under luke/:

./luke.sh          # or luke.bat on Windows; opens a directory chooser

Use it to answer "what actually got indexed?" without writing a probe program. Module docs: https://lucene.apache.org/core/10_0_0/luke/index.html.

The benchmark module

lucene-benchmark runs repeatable indexing and search workloads described by an .alg task file — handy for a local before/after when tuning.

# collection.alg -- excerpt
analyzer=org.apache.lucene.analysis.standard.StandardAnalyzer
content.source=org.apache.lucene.benchmark.byTask.feeds.LineDocSource
docs.file=work/enwiki.txt
{ "Rounds"
  ResetSystemErase
  { "Populate" CreateIndex { "MAddDocs" AddDoc > : 200000 CloseIndex }
  { "Search"   OpenReader { "SearchSameRdr" Search > : 500 CloseReader }
}

Run it with the module’s Benchmark main class. Module docs and the full task vocabulary: https://lucene.apache.org/core/10_0_0/benchmark/index.html. For the project’s own continuous numbers see Performance tuning.

The replicator module

lucene-replicator copies a committed (or NRT) index from a primary to one or more replicas, either in-process or over HTTP with the bundled Jetty handler. IndexReplicationHandler / ReplicationClient do the commit-point-based copy; the nrt package (PrimaryNode / ReplicaNode) does segment-level NRT replication without forcing a commit on the primary.

// Primary side: publish each commit.
// https://lucene.apache.org/core/10_0_0/replicator/index.html
Replicator replicator = new LocalReplicator();
replicator.publish(new IndexRevision(writer));

// Replica side: pull into a local directory, then refresh the searcher.
ReplicationClient client = new ReplicationClient(
    replicator,
    new IndexReplicationHandler(replicaDir, () -> searcherManager.maybeRefresh()),
    new PerSessionDirectoryFactory(workDir));
client.updateNow();

This is the building block a home-grown distributed setup uses; see Near-real-time search for the reopen side.

Misc command-line tools

lucene-misc bundles small utilities:

java -cp lucene-core-10.0.0.jar:lucene-misc-10.0.0.jar \
  org.apache.lucene.misc.HighFreqTerms -t /var/lib/app/index 25 body

A map of every module

Every artifact is org.apache.lucene:lucene-<name>. Most projects need only core plus analysis-common and queryparser.

Module What it gives you

core

The index, the analysis API, Directory, the core queries, BM25 scoring, and codecs.

analysis-common

StandardAnalyzer, the language-neutral tokenizers, and most TokenFilter implementations.

analysis-icu / -kuromoji / -nori / -smartcn / -stempel / -morfologik / -opennlp / -phonetic

ICU segmentation and folding; Japanese; Korean; Chinese; Polish (algorithmic); dictionary stemming; OpenNLP tokenizers; phonetic (Soundex / Metaphone) filters.

queryparser

The classic query-string parser plus the flexible, standard, complex-phrase, and surround parsers.

queries

FunctionScoreQuery, MoreLikeThis, and other extra Query implementations (DoubleValuesSource itself lives in lucene-core).

sandbox

Newer, not-yet-stable queries and features.

facet

Taxonomy and SortedSetDocValues faceting, range facets, and drill-sideways.

grouping

Result grouping and block / parent grouping.

join

Query-time and index-time (block) joins.

highlighter

The unified, plain, and postings-based highlighters.

suggest

Autocomplete and "did you mean" — AnalyzingInfixSuggester, FST suggesters, spellchecker.

expressions

Compile a JavaScript expression into a DoubleValuesSource for sorting / scoring.

spatial-extras / spatial3d

Geospatial shapes and prefix-tree strategies; 3D / geodesic maths.

misc

IndexMergeTool, HighFreqTerms, GetTermInfo, index-sorting helpers, native-directory glue.

codecs

Alternative and experimental codecs (direct, bloom-filter postings, and others).

backward-codecs

Read-only codecs for the previous major version (pulled in by core).

classification

k-NN and naive-Bayes document classifiers over an index.

memory

MemoryIndex — index one document in RAM to test it against many queries.

monitor

Reverse search: register many queries, stream documents past them — see Monitor / reverse search.

replicator

Primary / replica index replication, in-process or over HTTP.

benchmark

.alg-driven indexing and search benchmarks.

test-framework

LuceneTestCase, RandomIndexWriter, newDirectory(), and the randomized runner.

luke

The Swing index-inspection GUI.

demo

The IndexFiles / SearchFiles sample programs.

The full module list with Javadoc links is on the Lucene 10 documentation index.

Related pages