Scoring & similarity

This section documents the current Apache Lucene 10.x line — Lucene 10 requires Java 21 — as published at the Apache Lucene documentation and Javadoc, which is the reference these pages are written and verified against. No specific patch version is pinned; examples target lucene-core 10.x and the companion modules. Some areas (the Panama foreign-memory / Vector API internals, codec file-format internals, and the nightly benchmark harness) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, as Lucene iterates quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

Lucene ranks matching documents with a Similarity: a pluggable model that turns per-term corpus statistics (term frequency, document frequency, field length) into a score. The default is BM25Similarity; several probabilistic and language-model alternatives ship in org.apache.lucene.search.similarities, and any of them can be set globally or per field.

BM25Similarity, the default

BM25Similarity has been the default since Lucene 6 — earlier lines used the TF/IDF vector-space DefaultSimilarity (now ClassicSimilarity, kept only for compatibility). BM25 scores a term with a saturating term-frequency component and a length normalisation:

score(term, doc) = idf * (k1 + 1) * tf
                   -------------------------------------------
                   tf + k1 * (1 - b + b * fieldLen / avgFieldLen)

idf = log(1 + (N - df + 0.5) / (df + 0.5))

  • k1 (default 1.2) — how fast term frequency saturates. Higher k1 keeps rewarding repeated terms for longer; k1 = 0 ignores term frequency entirely.

  • b (default 0.75) — how strongly a long field is penalised. b = 0 disables length normalisation; b = 1 applies it fully.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/search/similarities/BM25Similarity.html
import org.apache.lucene.search.similarities.BM25Similarity;

BM25Similarity sim = new BM25Similarity(1.5f, 0.6f); // custom k1, b

Field-length norms and SmallFloat

fieldLen is not read exactly at query time — at index time each field’s length (roughly, its token count) is quantised to a single byte by SmallFloat.intToByte4 and written as the field norm. BM25Similarity.computeNorm decodes that byte back to an approximate length. This is why turning norms off (FieldType.setOmitNorms(true), implicit on StringField) removes length normalisation for that field, and why changing b does not require reindexing but changing whether norms are stored does. See SmallFloat.

Other Similarity implementations

Class Model

BooleanSimilarity

score is the query boost only — no tf, idf, or norms (a pure filter-style match)

ClassicSimilarity

the legacy TF/IDF vector-space model (pre-6 default), for compatibility

DFRSimilarity

Divergence From Randomness — pick a BasicModel, AfterEffect, Normalization

IBSimilarity

Information-Based model — a Distribution, Lambda, Normalization

LMDirichletSimilarity

language model with Dirichlet smoothing (parameter mu, default 2000)

LMJelinekMercerSimilarity

language model with Jelinek-Mercer smoothing (parameter lambda)

 
// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/search/similarities/package-summary.html
import org.apache.lucene.search.similarities.*;

Similarity dfr = new DFRSimilarity(
        new BasicModelG(), new AfterEffectB(), new NormalizationH2());
Similarity lmd = new LMDirichletSimilarity(2000f);

PerFieldSimilarityWrapper

PerFieldSimilarityWrapper routes each field to its own Similarity — e.g. BM25 for body, a language model for title, BooleanSimilarity for a tags field that should not influence ranking.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/search/similarities/PerFieldSimilarityWrapper.html
import org.apache.lucene.search.similarities.*;

Similarity perField = new PerFieldSimilarityWrapper() {
    final Similarity bm25 = new BM25Similarity();
    final Similarity lm    = new LMDirichletSimilarity();
    @Override
    public Similarity get(String field) {
        return field.equals("title") ? lm : bm25;
    }
};

Setting a Similarity: indexing vs. querying

A Similarity is consulted in two places, and both should normally use the same implementation:

  • IndexWriterConfig.setSimilarity(…​) — used at index time to compute and store field norms. Changing this affects only segments written afterwards.

  • IndexSearcher.setSimilarity(…​) — used at query time to score matches. Changing this takes effect immediately, with no reindex, for anything it can compute from stored statistics (so k1, b, mu, and switching between models all work live; enabling norms on a field that omitted them does not).

// indexing
IndexWriterConfig iwc = new IndexWriterConfig(analyzer);
iwc.setSimilarity(new BM25Similarity(1.5f, 0.6f));

// querying
IndexSearcher searcher = new IndexSearcher(reader);
searcher.setSimilarity(new BM25Similarity(1.5f, 0.6f));

The Explanation API

IndexSearcher.explain(Query, int docId) returns an Explanation tree: the final score at the root and every additive/multiplicative contribution beneath it, each with a human-readable description. It is the tool for answering "why did this document score what it did" and for verifying a custom Similarity.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/search/IndexSearcher.html#explain(org.apache.lucene.search.Query,int)
import org.apache.lucene.search.Explanation;

TopDocs hits = searcher.search(query, 10);
int docId = hits.scoreDocs[0].doc;
Explanation e = searcher.explain(query, docId);
System.out.println(e);          // full nested breakdown
System.out.println(e.getValue()); // the score as a Number

A BM25 explanation bottoms out in the term statistics below; the diagram shows how they combine:

flowchart LR tf["tf: term occurrences in this field of this doc"] --> sat["tf saturation term"] k1["k1 parameter"] --> sat b["b parameter"] --> sat fl["fieldLen: decoded from the stored norm byte"] --> sat afl["avgFieldLen: mean over the collection"] --> sat df["df: docs containing the term"] --> idf["idf component"] n["N: total docs in the field"] --> idf sat --> score["BM25 score = idf x (k1 + 1) x saturation"] idf --> score

Related pages