Grouping & joins

This section documents the current Apache Lucene 10.x line — Lucene 10 requires Java 21 — as published at the Apache Lucene documentation and Javadoc, which is the reference these pages are written and verified against. No specific patch version is pinned; examples target lucene-core 10.x and the companion modules. Some areas (the Panama foreign-memory / Vector API internals, codec file-format internals, and the nightly benchmark harness) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, as Lucene iterates quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

Lucene has no tables and no GROUP BY, but two modules cover the same ground: lucene-grouping collapses a flat hit list into per-key groups (like SQL grouping or a "collapse" feature), and lucene-join relates one set of documents to another either at query time through a shared field value or at index time through a contiguous parent/children block.

Result grouping (lucene-grouping)

GroupingSearch is the high-level entry point. Point it at a SortedDocValues / SortedSetDocValues field and it returns TopGroups — the top N groups, each with its own top k documents.

// https://lucene.apache.org/core/10_0_0/grouping/org/apache/lucene/search/grouping/GroupingSearch.html
import org.apache.lucene.search.grouping.GroupingSearch;
import org.apache.lucene.search.grouping.TopGroups;
import org.apache.lucene.search.grouping.GroupDocs;
import org.apache.lucene.search.Sort;
import org.apache.lucene.util.BytesRef;

// field "author" indexed with new SortedDocValuesField("author", new BytesRef(name))
GroupingSearch grouping = new GroupingSearch("author");
grouping.setGroupSort(Sort.RELEVANCE);        // order of the groups
grouping.setSortWithinGroup(Sort.RELEVANCE);  // order of docs inside each group
grouping.setGroupDocsLimit(3);                // top 3 hits per group
grouping.setAllGroups(true);                  // also count the total number of distinct groups

Query query = new QueryParser("body", analyzer).parse("sharding");
TopGroups<BytesRef> groups = grouping.search(searcher, query, 0 /* groupOffset */, 10 /* topNGroups */);

for (GroupDocs<BytesRef> g : groups.groups) {
    System.out.println(g.groupValue.utf8ToString() + "  (" + g.totalHits.value() + " hits)");
    for (var sd : g.scoreDocs) { /* sd.doc, sd.score */ }
}

First pass and second pass

GroupingSearch runs the query twice. The first pass (FirstPassGroupingCollector) finds which groups have the best-scoring documents; the second pass (TopGroupsCollector) re-runs the query knowing those groups and fills in the top k documents and hit counts for each. Drive the passes directly when you need to slot grouping into a custom CollectorManager pipeline:

// https://lucene.apache.org/core/10_0_0/grouping/org/apache/lucene/search/grouping/FirstPassGroupingCollector.html
import org.apache.lucene.search.grouping.*;
import org.apache.lucene.search.Sort;
import org.apache.lucene.util.BytesRef;
import java.util.Collection;

GroupSelector<BytesRef> selector = new TermGroupSelector("author");

FirstPassGroupingCollector<BytesRef> first =
        new FirstPassGroupingCollector<>(selector, Sort.RELEVANCE, 10);
searcher.search(query, first);
Collection<SearchGroup<BytesRef>> topGroups = first.getTopGroups(0);

TopGroupsCollector<BytesRef> second = new TopGroupsCollector<>(
        selector, topGroups, Sort.RELEVANCE, Sort.RELEVANCE,
        3 /* maxDocsPerGroup */, true /* getMaxScores */);
searcher.search(query, second);
TopGroups<BytesRef> result = second.getTopGroups(0);

Grouping by document block

If the grouping key is fixed at index time and each group’s documents were written together with IndexWriter.addDocuments, grouping needs no doc-values field and no first pass — construct GroupingSearch from a Query that matches the last document of every block:

// blocks written as: children..., then the group-marker doc last.
// The GroupingSearch(Query) constructor takes a query matching each block's LAST doc.
Query lastDocPerBlock = new TermQuery(new Term("docType", "groupEnd"));
GroupingSearch blockGrouping = new GroupingSearch(lastDocPerBlock);
TopGroups<?> byBlock = blockGrouping.search(searcher, query, 0, 10);

The lucene-grouping module overview lists every collector and selector. Solr’s Result Grouping and the collapse query parser (Solr query parsers) are built on this module.

Relating documents (lucene-join)

Query-time join by field value versus a parent-plus-children block in one segment

Query-time joins

JoinUtil.createJoinQuery turns "documents whose fromField value matches the toField value of a document that satisfies fromQuery`" into a single `Query. The two document sets are indexed independently, in any order, and the join is resolved per query from the field’s global ordinals.

// https://lucene.apache.org/core/10_0_0/join/org/apache/lucene/search/join/JoinUtil.html
import org.apache.lucene.search.join.JoinUtil;
import org.apache.lucene.search.join.ScoreMode;

// child docs carry  KeywordField("cust", ...) ; parent docs carry  KeywordField("id", ...)
Query childQuery = new TermQuery(new Term("status", "OPEN"));

Query joinToParents = JoinUtil.createJoinQuery(
        "cust",          // fromField  (on the child docs matched by childQuery)
        false,           // multiple values per document?
        "id",            // toField    (on the parent docs to return)
        childQuery,
        searcher,        // searcher over the index holding the child docs
        ScoreMode.Max);  // how child scores roll up: None, Min, Max, Avg, Total

TopDocs parents = searcher.search(joinToParents, 20);

ScoreMode.None makes it a pure filter (fastest). Other createJoinQuery overloads take a numeric join field, or an OrdinalMap built over a MultiReader, so the two document sets need not sit in the same index. The cost is per-query ordinal work and memory proportional to the join field’s cardinality.

Index-time block joins

A block join is much faster because the relationship is baked into doc-id adjacency. Write a parent and all of its children in one addDocuments call, parent last; the children get consecutive doc ids immediately before the parent’s.

// https://lucene.apache.org/core/10_0_0/join/org/apache/lucene/search/join/ToParentBlockJoinQuery.html
import org.apache.lucene.search.join.*;
import org.apache.lucene.document.*;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.*;
import java.util.List;

Document child1 = new Document();
child1.add(new StringField("docType", "sku", Field.Store.NO));
child1.add(new StringField("size", "M", Field.Store.YES));
Document child2 = new Document();
child2.add(new StringField("docType", "sku", Field.Store.NO));
child2.add(new StringField("size", "L", Field.Store.YES));
Document parent = new Document();
parent.add(new StringField("docType", "product", Field.Store.NO));
parent.add(new TextField("name", "Rain jacket", Field.Store.YES));

writer.addDocuments(List.of(child1, child2, parent));   // one contiguous block, parent last

// a filter selecting the parent doc of every block
BitSetProducer parents = new QueryBitSetProducer(new TermQuery(new Term("docType", "product")));

// children -> parents
Query q1 = new ToParentBlockJoinQuery(
        new TermQuery(new Term("size", "M")), parents, ScoreMode.Avg);

// parents -> children
Query q2 = new ToChildBlockJoinQuery(
        new TermQuery(new Term("name", "jacket")), parents);

// the children of one already-known parent doc id
Query q3 = new ParentChildrenBlockJoinQuery(
        parents, new MatchAllDocsQuery(), knownParentDocId);

Because block membership is positional, changing one child means re-indexing the whole block (delete by a shared key, addDocuments again). After building or merging such an index, validate that every block is intact:

// https://lucene.apache.org/core/10_0_0/join/org/apache/lucene/search/join/CheckJoinIndex.html
import org.apache.lucene.search.join.CheckJoinIndex;

CheckJoinIndex.check(reader, parents);   // throws if any block is malformed

The lucene-join module overview covers both query families. Elasticsearch’s nested type is a block join and its has_child / has_parent / join field is a query-time join — see Elasticsearch joins & relationships.

Choosing an approach

	Query-time join	Block join
Indexing	Independent docs, any order, updatable separately	Parent + children written together, parent last
Update one child	Just re-index that child	Re-index the whole block
Cross-index	Yes (numeric/ordinal overload)	No — one segment
Cost	Per-query ordinal build, scales with field cardinality	Near-free; positional
Analogy	SQL join on a key	A pre-joined, denormalised row group

Query-time join

Block join

Indexing

Independent docs, any order, updatable separately

Parent + children written together, parent last

Update one child

Just re-index that child

Re-index the whole block

Cross-index

Yes (numeric/ordinal overload)

No — one segment

Cost

Per-query ordinal build, scales with field cardinality

Near-free; positional

Analogy

SQL join on a key

A pre-joined, denormalised row group

Neither is a general-purpose relational join: for deeply relational data, model it in a relational database, or denormalise up front as MongoDB data modeling describes and skip the join entirely.

Filtering & faceting — SortedDocValues and the doc-values fields that grouping keys and join fields are built from.
Collectors & concurrent search — CollectorManager and how the grouping passes fit a concurrent search.
Documents & fields — writing a parent/children block with addDocuments.
Elasticsearch joins & relationships — nested (block join) and the join field (query-time join) over the DSL.
SQL relations — foreign keys and real joins, the model these approximate.
MongoDB data modeling — embed vs. reference, the same bounded-vs-churny tradeoff.