Apache Lucene Reference

Getting Started — what Lucene is as an embedded in-process Java search library, adding lucene-core and the companion modules to a Maven or Gradle build, and a first end-to-end index-and-search round-trip.

Architecture & Data Flow — the index model (a Directory of immutable segments, generations and commits, the pluggable codec) and the indexing and search class chains that write to and read from it.

Directories & Storage — the Directory abstraction and its implementations (MMapDirectory, NIOFSDirectory, ByteBuffersDirectory), write.lock and LockFactory, IndexInput prefetch and ReadAdvice, and why Lucene storage leans on the OS page cache rather than the JVM heap.

Documents & Fields — the Document as a schema-free ordered list of Field objects, the modern field types and what each one enables, FieldType and IndexOptions, and a capability matrix for picking searchable vs. stored vs. sortable vs. range vs. vector.

Indexing & Merge Policies — how IndexWriter adds, updates, soft-deletes and commits documents, the IndexWriterConfig knobs that shape a segment, and why and how Lucene merges segments (TieredMergePolicy and the other policies, plus the merge schedulers).

Near-Real-Time Search — making uncommitted IndexWriter changes searchable with DirectoryReader.open(writer) and openIfChanged, managing readers with SearcherManager and ControlledRealTimeReopenThread, stable paging with SearcherLifetimeManager, and the reference-counting rules that keep it safe.

Analysis Pipeline — how an Analyzer turns text into terms: the CharFilter / Tokenizer / TokenFilter chain, the TokenStream lifecycle and stream reuse, and the attribute API that carries term text, positions and offsets.

Built-in Analyzers & CustomAnalyzer — the ready-made analyzers in lucene-analysis-common (StandardAnalyzer, EnglishAnalyzer and the language set), PerFieldAnalyzerWrapper, and building an analyzer from factory names with CustomAnalyzer.builder().

Token Filters & Recipes — practical token-filter chains for folding, stop words, synonym graphs, stemming, autocomplete n-grams, shingles, word-delimiter splitting and payloads.

Language Analysis — the optional analysis modules for non-Latin scripts and language-specific matching (ICU, Kuromoji, Nori, SmartCN, Stempel, Morfologik, phonetic and OpenNLP).

Query Parsers — the classic QueryParser syntax and its pitfalls, SimpleQueryParser, the flexible StandardQueryParser framework, ComplexPhraseQueryParser, and when to skip parsing entirely.

Core Queries — the org.apache.lucene.search Query hierarchy (term, boolean, phrase, multi-term, range, wrapper and set queries), with construction examples and notes on QueryVisitor and rewriting.

Points & Range Queries — how Lucene indexes numeric and multi-dimensional values as points in a BKD tree, and the IntPoint / LongPoint / DoublePoint queries, IndexOrDocValuesQuery, doc-values skip lists, and the IntField / LongField convenience types.

Interval & Span Queries — positional and proximity matching with the modern Intervals / IntervalQuery API (ordered, unordered, maxgaps, containing, containedBy, before, after) and the legacy SpanQuery family it supersedes.

Filtering & Faceting — filtering with BooleanClause.Occur.FILTER, ConstantScoreQuery and TermInSetQuery cached by the searcher’s LRUQueryCache, and the lucene-facet module (FacetsConfig, taxonomy vs. SortedSetDocValues counting, range and value facets, drill-down / drill-sideways).

Scoring & Similarity — BM25Similarity as the default ranking model (its k1 and b parameters, field-length norms and SmallFloat quantisation), the alternative Similarity implementations, PerFieldSimilarityWrapper, and the Explanation API.

Function & Custom Scoring — reshaping relevance with FunctionScoreQuery and DoubleValuesSource, FeatureField relevance signals, the lucene-expressions module, and sorting by field, by script and by score.

Collectors & Concurrent Search — how Lucene gathers hits: the CollectorManager model that replaces bare Collector, running one query across an Executor, the total-hits threshold and TotalHits.Relation, early termination, and writing a custom CollectorManager.

Retrieving Results — turning search hits into data: the StoredFields API and selective loading with StoredFieldVisitor, reading doc values and term vectors, and paging with searchAfter against a stable searcher.

kNN Vector Search — dense-vector retrieval with KnnFloatVectorField and KnnByteVectorField, the HNSW graph and its quantised codec formats, KnnFloatVectorQuery with pre-filtering, exact kNN and hybrid lexical + vector search, and the Panama Vector API acceleration.

Spatial Search — LatLonPoint and LatLonShape in lucene-core, the prefix-tree and serialized strategies in lucene-spatial-extras, geo3d geometry on the ellipsoid in lucene-spatial3d, and WKT / GeoJSON parsing.

Highlighting, Suggesters & More — the lucene-highlighter UnifiedHighlighter and its offset-source requirement, the lucene-suggest Lookup and NRT completion APIs, spell checking, MoreLikeThis from lucene-queries, and the lucene-classification module.

Grouping & Joins — result grouping with the lucene-grouping module (GroupingSearch, first-pass / second-pass collectors, block grouping) and relating documents with the lucene-join module (query-time JoinUtil joins and index-time block joins).

Monitor / Reverse Search — reverse search with lucene-monitor: registering MonitorQuery objects in a Monitor and matching incoming documents against all of them, with a Presearcher to scale to large query sets and CandidateMatcher variants for the match detail you need.

Performance Tuning — tuning for indexing throughput (RAM buffer sizing, one shared IndexWriter, deferred merges, forceMerge for a static index, parallel build with addIndexes) and for search latency (a reused IndexSearcher via SearcherManager, reader warming, the page-cache / heap split for MMapDirectory, the LRUQueryCache, DocValues sorting, capped hit counting), plus how to measure with the nightly benchmarks.

Testing, Tools & Modules — the lucene-test-framework (LuceneTestCase, BaseTokenStreamTestCase, newDirectory, RandomIndexWriter, randomized and Nightly testing), CheckIndex for corruption, IndexUpgrader with lucene-backward-codecs, the Luke index browser, and the benchmark, replicator and misc tool modules — plus a one-line map of every Lucene module.

Lucene vs. Solr vs. Elasticsearch vs. OpenSearch — how the embedded Lucene library compares with the three search servers built on it on deployment model, clustering, API style, licensing and operational weight, with a decision list for choosing one.

Cheat Sheet (PDF) — a single-page, printable summary of everything in this section, with a downloadable PDF.