Reading hits back

This section documents the current Apache Lucene 10.x line — Lucene 10 requires Java 21 — as published at the Apache Lucene documentation and Javadoc, which is the reference these pages are written and verified against. No specific patch version is pinned; examples target lucene-core 10.x and the companion modules. Some areas (the Panama foreign-memory / Vector API internals, codec file-format internals, and the nightly benchmark harness) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, as Lucene iterates quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

A TopDocs gives back doc ids and scores, nothing else. To show results you fetch the stored copy of each document, and to sort, group or compute over them you read the columnar (doc-values) copy. Lucene 10 exposes both through small per-reader accessor objects obtained from the searcher.

Stored fields

searcher.storedFields() returns a StoredFields bound to the current reader. document(docId) materialises a Document from the stored fields written at index time. The old IndexSearcher.doc(int) / IndexReader.document(int) methods are deprecated — use the accessor.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/StoredFields.html
import org.apache.lucene.index.StoredFields;
import org.apache.lucene.document.Document;

StoredFields storedFields = searcher.storedFields();   // not thread-safe: one per thread
for (ScoreDoc sd : hits.scoreDocs) {
    Document doc = storedFields.document(sd.doc);
    String title = doc.get("title");
    String url   = doc.get("url");
}

Selective loading with StoredFieldVisitor

Loading only the fields a result page displays avoids decompressing the rest of the stored block. The Set overload is the shortcut; a StoredFieldVisitor gives full control, including aborting once the wanted fields are in hand.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/StoredFieldVisitor.html
import java.util.Set;
import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.StoredFieldVisitor;

Document slim = storedFields.document(docId, Set.of("title", "url"));

class TitleOnly extends StoredFieldVisitor {
    String title;
    @Override public Status needsField(FieldInfo fi) {
        return fi.name.equals("title") ? Status.YES : Status.STOP;   // fields are visited in order
    }
    @Override public void stringField(FieldInfo fi, String value) { title = value; }
}
TitleOnly v = new TitleOnly();
storedFields.document(docId, v);

StoredFieldVisitor has a typed callback per stored type (stringField, intField, longField, floatField, doubleField, binaryField); needsField returns YES, NO or STOP.

Doc values for computed columns

Stored fields are the display copy; doc values are the per-segment columnar copy used for sorting, faceting and any custom per-hit computation. Read them by segment, mapping a global doc id to the segment-local id with leaf.docBase.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/DocValues.html
import org.apache.lucene.index.DocValues;
import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.index.NumericDocValues;

for (LeafReaderContext leaf : searcher.getIndexReader().leaves()) {
    NumericDocValues price = DocValues.getNumeric(leaf.reader(), "price");
    int local = globalDocId - leaf.docBase;
    if (local >= 0 && local < leaf.reader().maxDoc() && price.advanceExact(local)) {
        long value = price.longValue();
    }
}

Field types and how to index the doc-values variant are on Documents & fields.

Term vectors

If a field was indexed with term vectors, searcher.termVectors().get(docId, field) returns a per-document Terms — the terms, frequencies and (optionally) positions/offsets for that one document. It is the input to the fast highlighters and to "more like this".

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/TermVectors.html
import org.apache.lucene.index.TermVectors;
import org.apache.lucene.index.Terms;
import org.apache.lucene.index.TermsEnum;
import org.apache.lucene.util.BytesRef;

TermVectors termVectors = searcher.termVectors();
Terms tv = termVectors.get(docId, "body");   // null if the field has no term vectors
if (tv != null) {
    TermsEnum te = tv.iterator();
    BytesRef term;
    while ((term = te.next()) != null) {
        long freq = te.totalTermFreq();
    }
}

Paging with searchAfter

searchAfter(ScoreDoc after, Query, n) resumes ranking just past a hit from the previous page, avoiding the growing cost of search(query, pageStart + pageSize). With a sort, after must be the FieldDoc returned for the last hit so its sort values travel with it.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/search/IndexSearcher.html#searchAfter(org.apache.lucene.search.ScoreDoc,org.apache.lucene.search.Query,int)
int PAGE = 20;

TopDocs page1 = searcher.search(query, PAGE);
ScoreDoc last = page1.scoreDocs[page1.scoreDocs.length - 1];
TopDocs page2 = searcher.searchAfter(last, query, PAGE);

// Sorted paging: cast the cursor to FieldDoc and pass the same Sort.
TopFieldDocs s1 = (TopFieldDocs) searcher.search(query, PAGE, sort);
FieldDoc sLast = (FieldDoc) s1.scoreDocs[s1.scoreDocs.length - 1];
TopFieldDocs s2 = (TopFieldDocs) searcher.searchAfter(sLast, query, PAGE, sort);

A stable searcher across pages

Every page of one result set must run against the same IndexSearcher. Doc ids are only stable within a reader generation: after a merge or a reopen they are renumbered, so a cursor captured against the old reader points at the wrong documents and counts drift. Hold the searcher for the pagination session — keep the IndexSearcher acquired from SearcherManager (see Near-real-time search) until the user stops paging, or cache it under a short-lived search token — and release it when done. Reopen the reader only when the user starts a fresh query, or deliberately to surface new documents, accepting that deep cursors are then invalidated. The equivalent trade-off in a distributed engine — search_after plus a point-in-time reader — is covered in Elasticsearch: Search API & pagination.

Related pages