Documents, fields & the capability matrix

This section documents the current Apache Lucene 10.x line — Lucene 10 requires Java 21 — as published at the Apache Lucene documentation and Javadoc, which is the reference these pages are written and verified against. No specific patch version is pinned; examples target lucene-core 10.x and the companion modules. Some areas (the Panama foreign-memory / Vector API internals, codec file-format internals, and the nightly benchmark harness) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, as Lucene iterates quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

A Lucene Document has no schema: it is just an ordered List of Field objects, and two documents in the same index may carry entirely different fields. What a field lets you do — search it, retrieve it, sort or facet on it, run a range query, do nearest-neighbour — is decided entirely by which Field subclass you add and how its FieldType is configured.

Document: an ordered list of fields

You build a Document, add Field instances to it, and hand it to IndexWriter.addDocument. A field name may appear more than once (the values are all indexed). Nothing validates field names or types across documents — consistency is your responsibility, or your framework’s.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/document/Document.html
import org.apache.lucene.document.*;

Document doc = new Document();
doc.add(new StringField("isbn", "9780134685991", Field.Store.YES));
doc.add(new TextField("title", "Effective Java", Field.Store.YES));
doc.add(new TextField("author", "Joshua Bloch", Field.Store.YES));
doc.add(new IntField("year", 2018, Field.Store.YES));          // point + doc values
doc.add(new TextField("tag", "java", Field.Store.NO));
doc.add(new TextField("tag", "best-practices", Field.Store.NO)); // same name, second value
writer.addDocument(doc);

The field types

The modern (post-4.x) field classes each switch on a specific capability. Pick the combination that matches how the field will be queried and read back; add two fields with the same name when you need two capabilities (e.g. a TextField to search and a SortedDocValuesField to sort).

Field class What it enables

TextField

Value is run through the Analyzer and indexed as terms — full-text search, phrase and interval queries. Field.Store.YES also keeps the original for retrieval.

StringField

The whole value is indexed as a single un-analyzed term — exact match, prefix, term filters. For ids, codes, enums.

KeywordField

Like StringField for search, and also adds SortedSetDocValues so the same field sorts and facets without a second field. The preferred exact-string field in 10.x.

StoredField

Stored for retrieval only — never indexed, not searchable. For payload you show but never query.

IntPoint / LongPoint / FloatPoint / DoublePoint

Value indexed into a BKD tree for efficient range and exact numeric search. Not stored, not sortable on their own.

IntField / LongField / FloatField / DoubleField

Convenience field that bundles the matching *Point (range search) with SortedNumericDocValues (sort / facet) in one instance; add Field.Store.YES to also retrieve it.

NumericDocValuesField

A single columnar long per document — sort, numeric facet ranges, function scoring. Not searchable.

SortedDocValuesField

One columnar BytesRef per document (single-valued) — sort and facet on a string.

SortedSetDocValuesField

Multiple columnar BytesRef values per document — multi-valued string facets.

SortedNumericDocValuesField

Multiple columnar numeric values per document — multi-valued numeric sort / facet.

KnnFloatVectorField / KnnByteVectorField

A dense vector per document, indexed in an HNSW graph for approximate nearest-neighbour search — kNN vector search.

FeatureField

A named (feature, value) pair whose value boosts score cheaply at query time (FeatureField.newSaturationQuery, etc.) — e.g. a pagerank or freshness signal.

LatLonPoint

An encoded latitude/longitude point for geo distance and bounding-box / polygon queries — Spatial search. Pair with LatLonDocValuesField to sort by distance.

 
// A field's capability follows from its class -- add several with the same name for several
// capabilities.
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/document/package-summary.html
import org.apache.lucene.document.*;

Document d = new Document();

// exact id: searchable + sortable/facetable in one field
d.add(new KeywordField("status", "PUBLISHED", Field.Store.YES));

// numeric: range search AND sort/facet
d.add(new IntField("price_cents", 1299, Field.Store.YES));

// full-text search to match, plus doc values to sort alphabetically
d.add(new TextField("title", "The Go Programming Language", Field.Store.YES));
d.add(new SortedDocValuesField("title_sort", new org.apache.lucene.util.BytesRef("the go programming language")));

// approximate nearest-neighbour
d.add(new KnnFloatVectorField("embedding", new float[] {0.12f, -0.03f, /* ... */ 0.44f}));

// cheap score signal
d.add(new FeatureField("features", "pagerank", 4.2f));

writer.addDocument(d);

FieldType, IndexOptions and term vectors

Every field carries an IndexableFieldType. The built-in classes above ship sensible instances, but a custom FieldType lets you dial exactly what goes into the index:

  • IndexOptions — how much postings detail to store: DOCS (which documents), DOCS_AND_FREQS (+ term frequency), DOCS_AND_FREQS_AND_POSITIONS (+ positions, needed for phrase and interval queries), DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS (+ character offsets, used by the postings highlighter). NONE means the field is not indexed.

  • setStored(true) — keep the original value for storedFields().

  • setStoreTermVectors(true) (with …​Positions / …​Offsets) — a per-document mini term list, used by some highlighters and "more like this".

  • setOmitNorms(true) — drop the per-document length norm; saves a byte per doc per field but disables length normalization in scoring. StringField omits norms by default.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/index/IndexableFieldType.html
import org.apache.lucene.document.*;
import org.apache.lucene.index.IndexOptions;

FieldType bodyType = new FieldType();
bodyType.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);
bodyType.setStored(true);
bodyType.setStoreTermVectors(true);
bodyType.setStoreTermVectorPositions(true);
bodyType.setStoreTermVectorOffsets(true);
bodyType.setOmitNorms(false);
bodyType.freeze();

Document d = new Document();
d.add(new Field("body", "full article text ...", bodyType));   // generic Field + custom type
writer.addDocument(d);

Before Lucene 4, indexing was configured with the Field.Store and Field.Index enums (and numbers went through a single NumericField); today the capability is the Field subclass plus an optional custom FieldType, and only Field.Store survives.

Capability matrix

Read down the column for the capability you need, then pick a field (or a pair) whose row has it.

Field Searchable (term / match) Range Stored (retrieve) Sort / facet / function Vector

TextField

analyzed

 — 

opt.

 — 

 — 

StringField

exact term

 — 

opt.

 — 

 — 

KeywordField

exact term

 — 

opt.

yes (SortedSet)

 — 

StoredField

 — 

 — 

yes

 — 

 — 

IntPoint / LongPoint / DoublePoint

exact

yes (BKD)

 — 

 — 

 — 

IntField / LongField / DoubleField

exact

yes (BKD)

opt.

yes (SortedNumeric)

 — 

NumericDocValuesField

 — 

 — 

 — 

yes (single numeric)

 — 

SortedDocValuesField

 — 

 — 

 — 

yes (single string)

 — 

SortedSetDocValuesField

 — 

 — 

 — 

yes (multi string)

 — 

SortedNumericDocValuesField

 — 

 — 

 — 

yes (multi numeric)

 — 

KnnFloatVectorField / KnnByteVectorField

 — 

 — 

 — 

 — 

yes (HNSW)

FeatureField

boost only

 — 

 — 

score signal

 — 

LatLonPoint

geo predicate

geo box / distance

 — 

with LatLonDocValuesField

 — 

The recurring pattern: a *Point/*Field for range, a *DocValuesField for sort and facet, and Field.Store.YES (or a StoredField) for retrieval — often all three under one name. Points & range queries, Filtering & faceting, and Retrieving results each use one leg of it.

This is the library-level version of a server’s field mapping: compare Elasticsearch mapping & field types (where text vs. keyword and doc_values are mapping options) and Solr field types (where the same choices are <fieldType> declarations in a schema).

Related pages