Highlighting, suggesters, spellcheck, MoreLikeThis & classification

This section documents the current Apache Lucene 10.x line — Lucene 10 requires Java 21 — as published at the Apache Lucene documentation and Javadoc, which is the reference these pages are written and verified against. No specific patch version is pinned; examples target lucene-core 10.x and the companion modules. Some areas (the Panama foreign-memory / Vector API internals, codec file-format internals, and the nightly benchmark harness) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, as Lucene iterates quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

Once a query matches, several optional modules turn hits into user-facing features: snippet highlighting (lucene-highlighter), autocomplete and "did you mean" (lucene-suggest), find-similar (MoreLikeThis in lucene-queries), and text categorisation (lucene-classification). Each is a thin layer over the same index and IndexSearcher.

Highlighting with the UnifiedHighlighter

UnifiedHighlighter is the current highlighter. The older Highlighter (query scorer), FastVectorHighlighter (term-vector based), and PostingsHighlighter are superseded — their capabilities are folded into this one class, which picks the cheapest offset source available per field. Build it from a searcher and analyzer, then highlight a TopDocs.

// https://lucene.apache.org/core/10_0_0/highlighter/org/apache/lucene/search/uhighlight/UnifiedHighlighter.html
import org.apache.lucene.search.uhighlight.UnifiedHighlighter;
import java.text.BreakIterator;
import java.util.Locale;

Query query = new QueryParser("body", analyzer).parse("sharding strategy");
TopDocs top = searcher.search(query, 10);

UnifiedHighlighter highlighter = UnifiedHighlighter.builder(searcher, analyzer)
        .withBreakIterator(() -> BreakIterator.getSentenceInstance(Locale.ROOT))
        .build();

// one snippet string per hit (null when the field had no match), <b>..</b> by default
String[] snippets = highlighter.highlight("body", query, top, 3);   // up to 3 passages per hit

Passages are scored with a BM25-like PassageScorer (term weight x passage-local frequency, with a pivot on passage length) so the returned fragments are the most query-relevant sentences, not merely the first ones. withFormatter, withScorer, and withMaxLength on the builder override the markup, the scoring, and the per-field character budget.

The offset-source requirement

The highlighter needs character offsets for the matched terms. Provide them one of three ways, in order of speed:

Source How to enable

Postings offsets

Index the field with IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS. Fastest; no re-analysis.

Term vectors with offsets

FieldType.setStoreTermVectorOffsets(true) (also positions + vectors). Bigger index, but useful if you already store vectors.

Analysis (default fallback)

Nothing indexed; the highlighter re-analyzes the stored field value at query time. Requires Field.Store.YES. Simplest, slowest.

 
// A TextField variant that carries offsets in its postings.
import org.apache.lucene.document.FieldType;
import org.apache.lucene.document.TextField;
import org.apache.lucene.index.IndexOptions;

FieldType bodyType = new FieldType(TextField.TYPE_STORED);
bodyType.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);
bodyType.freeze();

doc.add(new org.apache.lucene.document.Field("body", articleText, bodyType));

The lucene-highlighter module overview documents the uhighlight package. Solr’s highlighter component (Solr highlighting) and Elasticsearch’s highlight block (Elasticsearch search extras) both drive this class.

Autocomplete: the suggest module

lucene-suggest provides Lookup implementations backed by an FST (or, for the infix suggesters, an auxiliary Lucene index). Build one from a Dictionary / InputIterator, then call lookup.

Lookup Matches

AnalyzingSuggester

Prefix of the whole analyzed entry ("san fr" → "San Francisco"). Applies an analyzer at build and query time.

FuzzySuggester

Like AnalyzingSuggester but tolerates edits in the prefix (typos).

AnalyzingInfixSuggester

Any token prefix inside the entry ("fran" → "San Francisco"). Backed by its own index directory.

BlendedInfixSuggester

AnalyzingInfixSuggester plus a position-based score penalty so earlier matches rank higher.

FreeTextSuggester

Next-word prediction from an n-gram language model over the corpus — completes the sentence, not the term.

 
// https://lucene.apache.org/core/10_0_0/suggest/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggester.html
import org.apache.lucene.search.suggest.InputArrayIterator;
import org.apache.lucene.search.suggest.Lookup.LookupResult;
import org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester;
import org.apache.lucene.util.BytesRef;
import java.util.List;

AnalyzingInfixSuggester suggester =
        new AnalyzingInfixSuggester(FSDirectory.open(Path.of("/var/data/suggest")), analyzer);

suggester.build(new InputArrayIterator(new org.apache.lucene.search.suggest.Input[] {
        new org.apache.lucene.search.suggest.Input(new BytesRef("San Francisco"), 9, null),
        new org.apache.lucene.search.suggest.Input(new BytesRef("San Jose"), 4, null),
}));

List<LookupResult> hits = suggester.lookup("fran", false, 5);   // -> "San Francisco"

A DocumentDictionary / DocumentValueSourceDictionary feeds entries and weights straight from an existing index instead of an array.

NRT completion: SuggestField and friends

The suggest.document package is a faster, near-real-time alternative: the completions live in the main index as a special postings format, so they update with your normal IndexWriter commits and respect deletes. Index a SuggestField (or ContextSuggestField to carry filter contexts), write with a CompletionAnalyzer and a codec that routes the suggest field through a CompletionPostingsFormat, and query through a SuggestIndexSearcher.

// https://lucene.apache.org/core/10_0_0/suggest/org/apache/lucene/search/suggest/document/SuggestField.html
import org.apache.lucene.search.suggest.document.*;
import org.apache.lucene.codecs.PostingsFormat;
import org.apache.lucene.codecs.lucene100.Lucene100Codec;

// --- indexing ---
CompletionAnalyzer completionAnalyzer = new CompletionAnalyzer(analyzer);
IndexWriterConfig iwc = new IndexWriterConfig(completionAnalyzer);
iwc.setCodec(new Lucene100Codec() {            // route only the suggest field through completion postings
    final PostingsFormat completion = new Completion912PostingsFormat();
    @Override public PostingsFormat getPostingsFormatForField(String field) {
        return field.startsWith("title_suggest") ? completion : super.getPostingsFormatForField(field);
    }
});
try (IndexWriter w = new IndexWriter(dir, iwc)) {
    Document d = new Document();
    d.add(new SuggestField("title_suggest", "Sharding strategies for Lucene", 9));
    d.add(new ContextSuggestField("title_suggest_ctx", "Sharding strategies", 9, "en"));
    w.addDocument(d);
}

// --- querying ---
try (DirectoryReader reader = DirectoryReader.open(dir)) {
    SuggestIndexSearcher searcher = new SuggestIndexSearcher(reader);
    CompletionQuery q = new PrefixCompletionQuery(completionAnalyzer,
            new Term("title_suggest", "shard"));
    TopSuggestDocs suggestions = searcher.suggest(q, 5, false);
    // also: FuzzyCompletionQuery (typos), RegexCompletionQuery, ContextQuery (wrap + filter)
}

Spell checking

Three tools, none of which need a rebuild pipeline beyond an analyzer:

// https://lucene.apache.org/core/10_0_0/suggest/org/apache/lucene/search/spell/DirectSpellChecker.html
import org.apache.lucene.search.spell.*;

// 1. DirectSpellChecker -- runs against the LIVE index terms, no side index to maintain
DirectSpellChecker direct = new DirectSpellChecker();
SuggestWord[] fix = direct.suggestSimilar(new Term("body", "recieve"), 5, reader);

// 2. SpellChecker -- builds a separate n-gram index from a dictionary
SpellChecker spell = new SpellChecker(FSDirectory.open(Path.of("/var/data/spell")));
spell.indexDictionary(new LuceneDictionary(reader, "body"),
        new IndexWriterConfig(analyzer), false);            // or PlainTextDictionary(Path) for a word list
String[] byGram = spell.suggestSimilar("recieve", 5);

// 3. WordBreakSpellChecker -- split "newyork" -> "new york", or combine the reverse
WordBreakSpellChecker breaker = new WordBreakSpellChecker();
CombineSuggestion[] combined = breaker.suggestWordCombinations(
        new Term[] { new Term("body", "new"), new Term("body", "york") },
        5, reader, SuggestMode.SUGGEST_WHEN_NOT_IN_INDEX);

DirectSpellChecker is the low-maintenance default; SpellChecker + PlainTextDictionary / LuceneDictionary is the choice when the vocabulary is a curated list. This is a different technique from the index-time n-gram token filters in Token filters & recipes: those expand the terms in the main index so a wrong spelling still matches; the spell checkers keep a compact edit-distance structure on the side and suggest a correction instead. The lucene-suggest module overview covers the spell and suggest packages together. Compare Solr spell check & suggest, which exposes exactly these classes as search components.

MoreLikeThis

MoreLikeThis (in lucene-queries) reads the term vectors or re-analyzes a seed document, keeps the highest-tf-idf "interesting terms", and builds a BooleanQuery of SHOULD clauses from them — the "related articles" query.

// https://lucene.apache.org/core/10_0_0/queries/org/apache/lucene/queries/mlt/MoreLikeThis.html
import org.apache.lucene.queries.mlt.MoreLikeThis;

MoreLikeThis mlt = new MoreLikeThis(reader);
mlt.setAnalyzer(analyzer);
mlt.setFieldNames(new String[] { "title", "body" });
mlt.setMinTermFreq(1);
mlt.setMinDocFreq(3);
mlt.setMaxQueryTerms(25);

Query similar = mlt.like(seedDocId);                       // from a doc already in the index
// or from arbitrary text:  mlt.like("body", new StringReader(pastedText));

// inspect what drove the query
String[] terms = mlt.retrieveInterestingTerms(seedDocId);

Wrap similar in a BooleanQuery.Builder with a MUST_NOT on the seed’s id to exclude the document itself. Elasticsearch’s more_like_this query (Elasticsearch search extras) is this class over the DSL.

The lucene-classification module

lucene-classification treats an indexed, labelled corpus as training data: a Classifier assigns a class to new text using only index statistics — no external ML runtime.

Classifier Model

KNearestNeighborClassifier

Runs the input text as a MoreLikeThis query, takes the majority label among the top k hits.

KNearestFuzzyClassifier

kNN as above but with fuzzy term matching, for noisy / short input.

SimpleNaiveBayesClassifier

Multinomial naive Bayes from term frequencies per class.

BM25NBClassifier

Naive Bayes with BM25-weighted term contributions instead of raw counts.

 
// https://lucene.apache.org/core/10_0_0/classification/org/apache/lucene/classification/KNearestNeighborClassifier.html
import org.apache.lucene.classification.*;
import org.apache.lucene.util.BytesRef;

Classifier<BytesRef> knn = new KNearestNeighborClassifier(
        reader, null /* similarity */, analyzer, null /* query filter */,
        10 /* k */, 1 /* minDocFreq */, 1 /* minTermFreq */,
        "category" /* class field */, "body" /* text field(s) */);

ClassificationResult<BytesRef> result = knn.assignClass("how do I re-shard a live index?");
String label = result.getAssignedClass().utf8ToString();
double confidence = result.getScore();

ConfusionMatrixGenerator in the same module evaluates a classifier against a held-out slice of the index. See the lucene-classification module overview.

Related pages