Function scoring, expressions & sorting

This section documents the current Apache Lucene 10.x line — Lucene 10 requires Java 21 — as published at the Apache Lucene documentation and Javadoc, which is the reference these pages are written and verified against. No specific patch version is pinned; examples target lucene-core 10.x and the companion modules. Some areas (the Panama foreign-memory / Vector API internals, codec file-format internals, and the nightly benchmark harness) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, as Lucene iterates quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

Once a query has selected the matching documents, three tools reshape their order: FunctionScoreQuery folds a numeric signal into the score, the lucene-expressions module compiles a short JavaScript formula into a value source for scoring, sorting or faceting, and Sort replaces score order entirely with field order. All three are built on one abstraction — DoubleValuesSource, a per-document double computed lazily from doc values, the score, or another source.

FunctionScoreQuery and DoubleValuesSource

FunctionScoreQuery wraps an inner Query and derives the final score from a DoubleValuesSource instead of (or on top of) BM25. It replaces the pre-8.x CustomScoreQuery / FunctionQuery pair; those classes are gone. It lives in the lucene-queries module.

<dependency>
  <groupId>org.apache.lucene</groupId>
  <artifactId>lucene-queries</artifactId>
  <version>10.0.0</version>
</dependency>
// https://lucene.apache.org/core/10_0_0/queries/org/apache/lucene/queries/function/FunctionScoreQuery.html
import org.apache.lucene.queries.function.FunctionScoreQuery;
import org.apache.lucene.search.DoubleValuesSource;
import org.apache.lucene.search.Query;

Query text = parser.parse("wireless headphones");   // ordinary BM25 relevance

// boostByValue: multiply the inner score by a bounded function of a numeric field.
DoubleValuesSource popularity = DoubleValuesSource.fromLongField("popularity");
Query boosted = FunctionScoreQuery.boostByValue(text, popularity);

// boostByQuery: multiply the score by `boost` only for docs matching a second query.
Query recent = org.apache.lucene.document.LongPoint
        .newRangeQuery("published", cutoffMillis, Long.MAX_VALUE);
Query boostRecent = FunctionScoreQuery.boostByQuery(text, recent, 2.0f);

// General form: replace the score with an arbitrary source (see expressions, below).
Query custom = new FunctionScoreQuery(text, myDoubleValuesSource);

TopDocs hits = searcher.search(boosted, 10);

DoubleValuesSource has factory methods for the common cases — fromLongField / fromDoubleField / fromIntField read NumericDocValues, SCORES exposes the relevance score, and constant, fromQuery and arithmetic combinators (multiply, add, min, max) build larger expressions. See DoubleValuesSource.

FeatureField: numeric relevance signals

FeatureField stores one weight per named feature in a shared field, packed for cheap access at score time. Query it with a bounded transform — saturation, log or sigmoid — so a raw count (popularity, pagerank, recency) contributes a well-behaved, diminishing score term rather than swamping the text relevance.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/document/FeatureField.html
import org.apache.lucene.document.FeatureField;
import org.apache.lucene.search.BooleanClause.Occur;
import org.apache.lucene.search.BooleanQuery;

// Index time: many features, one field.
doc.add(new FeatureField("features", "pagerank", 21.4f));
doc.add(new FeatureField("features", "popularity", 3.2f));

// Query time: pick a saturation curve per feature.
Query byPagerank = FeatureField.newSaturationQuery("features", "pagerank");
Query byPopular  = FeatureField.newLogQuery("features", "popularity", 1f, 4.5f);
Query byRecency  = FeatureField.newSigmoidQuery("features", "recency", 1f, 0.5f, 0.6f);

Query q = new BooleanQuery.Builder()
        .add(text, Occur.MUST)          // must still match the text
        .add(byPagerank, Occur.SHOULD)  // features only add score
        .add(byPopular, Occur.SHOULD)
        .build();

newSaturationQuery with no pivot picks a sensible pivot from the field statistics; passing an explicit pivot fixes the value at which the score reaches half its maximum. Full curve maths and parameter meanings: FeatureField.

The lucene-expressions module

lucene-expressions compiles a JavaScript-syntax string into an Expression, then binds each free variable to a DoubleValuesSource. The result drives scoring, sorting or faceting without a custom Java class.

<dependency>
  <groupId>org.apache.lucene</groupId>
  <artifactId>lucene-expressions</artifactId>
  <version>10.0.0</version>
</dependency>
// https://lucene.apache.org/core/10_0_0/expressions/org/apache/lucene/expressions/js/JavascriptCompiler.html
import org.apache.lucene.expressions.Expression;
import org.apache.lucene.expressions.SimpleBindings;
import org.apache.lucene.expressions.js.JavascriptCompiler;
import org.apache.lucene.search.DoubleValuesSource;
import org.apache.lucene.search.Sort;

// Compile once; Expression is thread-safe, so cache it.
Expression expr = JavascriptCompiler.compile("sqrt(_score) + ln(popularity + 1) * 0.3");

SimpleBindings bindings = new SimpleBindings();
bindings.add("_score", DoubleValuesSource.SCORES);
bindings.add("popularity", DoubleValuesSource.fromLongField("popularity"));

// (a) sort by the expression, high value first
Sort sort = new Sort(expr.getSortField(bindings, true));
TopDocs bySort = searcher.search(query, 10, sort);

// (b) score by the expression
DoubleValuesSource score = expr.getDoubleValuesSource(bindings);
Query rescored = new org.apache.lucene.queries.function.FunctionScoreQuery(query, score);

The built-in function set (abs, ceil, ln, log, pow, sqrt, min, max, trigonometric, haversin, …​) and the grammar are listed on the expressions.js package page; JavascriptCompiler.compile can also be given a Map of extra Method references to expose custom functions. Expression itself: Expression.

The same DoubleValuesSource feeds a dynamic facet — for example a range facet over the computed relevance value:

// https://lucene.apache.org/core/10_0_0/facet/org/apache/lucene/facet/range/DoubleRangeFacetCounts.html
import org.apache.lucene.facet.range.DoubleRange;
import org.apache.lucene.facet.range.DoubleRangeFacetCounts;

DoubleRange[] buckets = {
    new DoubleRange("low",  0.0, true, 1.0, false),
    new DoubleRange("high", 1.0, true, Double.POSITIVE_INFINITY, false)
};
Facets facets = new DoubleRangeFacetCounts("relevance", score, facetsCollector, buckets);

More on the faceting collectors in Filtering & faceting.

Sorting: Sort, SortField, SortedNumericSortField

IndexSearcher.search(Query, n, Sort) returns TopFieldDocs ordered by field values rather than score. Each SortField names a doc-values field, a SortField.Type, and a reverse flag; a Sort is an ordered list of them, with SortField.FIELD_SCORE and SortField.FIELD_DOC available as tie-breakers. Sorting a field requires it to be indexed with a *DocValuesField (see Documents & fields).

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/search/SortField.html
import org.apache.lucene.search.Sort;
import org.apache.lucene.search.SortField;
import org.apache.lucene.search.SortedNumericSelector;
import org.apache.lucene.search.SortedNumericSortField;

// Single-valued numeric field, descending, then by score.
SortField byPrice = new SortField("price", SortField.Type.LONG, true);
byPrice.setMissingValue(Long.MAX_VALUE);          // docs with no price sort last
Sort sort = new Sort(byPrice, SortField.FIELD_SCORE);

// Multi-valued field: choose which of the values represents the document.
SortField byRating = new SortedNumericSortField(
        "rating", SortField.Type.INT, true, SortedNumericSelector.Type.MAX);

// doDocScores=true also computes _score for the returned hits (extra work).
TopFieldDocs page = searcher.search(query, 10, new Sort(byRating), true);

Missing-value handling: for numeric types call setMissingValue(…​) with the sentinel that should sort first or last; for SortField.Type.STRING use the constants SortField.STRING_FIRST / SortField.STRING_LAST. Without a missing value set, documents lacking the field sort as if they had the type’s minimum. Sorting on _score alone is just new Sort() (the default) or new Sort(SortField.FIELD_SCORE). Reference: Sort and SortedNumericSortField.

Paging a sorted result set with searchAfter and a FieldDoc cursor is covered in Retrieving results.

Related pages