The lucene-monitor module

This section documents the current Apache Lucene 10.x line — Lucene 10 requires Java 21 — as published at the Apache Lucene documentation and Javadoc, which is the reference these pages are written and verified against. No specific patch version is pinned; examples target lucene-core 10.x and the companion modules. Some areas (the Panama foreign-memory / Vector API internals, codec file-format internals, and the nightly benchmark harness) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, as Lucene iterates quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

Normal search registers documents and runs a query against them. Reverse search inverts that: you register the queries once and stream documents past them, asking "which of my stored queries does this document match?". lucene-monitor (the module that was formerly the standalone Luwak library) is Lucene’s implementation — the engine behind saved-search alerts, content-routing rules, and real-time classification.

Registering queries in a Monitor

A Monitor holds a set of MonitorQuery records, each a stable id plus a parsed Query (and optional original query string and metadata map). monitor.match takes one or more in-memory documents and returns the ids of every registered query they satisfy.

// https://lucene.apache.org/core/10_0_0/monitor/org/apache/lucene/monitor/Monitor.html
import org.apache.lucene.monitor.Monitor;
import org.apache.lucene.monitor.MonitorQuery;
import org.apache.lucene.monitor.MatchingQueries;
import org.apache.lucene.monitor.QueryMatch;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.TextField;
import org.apache.lucene.document.Field;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.queryparser.classic.QueryParser;
import java.util.Map;

StandardAnalyzer analyzer = new StandardAnalyzer();
QueryParser parser = new QueryParser("text", analyzer);

try (Monitor monitor = new Monitor(analyzer)) {

    // register saved searches -- id, Query, original string, metadata
    monitor.register(
            new MonitorQuery("alert-1", new TermQuery(new Term("text", "lucene"))),
            new MonitorQuery("alert-2", parser.parse("search AND (engine OR library)"),
                    "search AND (engine OR library)", Map.of("team", "platform")));

    // an incoming document (never added to any index)
    Document doc = new Document();
    doc.add(new TextField("text", "the new lucene monitor module does reverse search",
            Field.Store.NO));

    MatchingQueries<QueryMatch> result = monitor.match(doc, QueryMatch.SIMPLE_MATCHER);

    for (QueryMatch m : result.getMatches()) {
        System.out.println("matched " + m.getQueryId());   // alert-1, alert-2
    }
    System.out.println("queries actually run: " + result.getQueriesRun());
}

register re-parses and stores the queries in an internal Lucene index; call it again with the same id to update a query, and monitor.deleteById("alert-1") to remove one. monitor.match(Document[], factory) matches a batch and returns MultiMatchingQueries.

How much detail: the matcher factory

The second argument to match is a MatcherFactory that builds a CandidateMatcher. It decides what each match tells you — and how much work per candidate query:

Factory Each match yields

QueryMatch.SIMPLE_MATCHER

Just the matching query id. Cheapest.

HighlightsMatch.MATCHER

The id plus the hit positions/offsets within the document.

ExplainingMatch.MATCHER

The id plus a Lucene Explanation of why it matched.

ScoringMatch.matchWithSimilarity(sim)

The id plus a relevance score under the given Similarity.

 
// https://lucene.apache.org/core/10_0_0/monitor/org/apache/lucene/monitor/HighlightsMatch.html
import org.apache.lucene.monitor.HighlightsMatch;
import org.apache.lucene.monitor.MatchingQueries;

MatchingQueries<HighlightsMatch> hi = monitor.match(doc, HighlightsMatch.MATCHER);
for (HighlightsMatch m : hi.getMatches()) {
    m.getHits("text").forEach(h -> System.out.println(h.startOffset() + ".." + h.endOffset()));
}

Scaling with a Presearcher

Running every registered query against every document does not scale past a few thousand queries. A Presearcher builds a secondary query from the incoming document’s terms and runs it against the query index first, so only queries that could match are executed. TermFilteredPresearcher (the default) is term-based; MultipassTermFilteredPresearcher trades more index space for fewer false candidates.

// https://lucene.apache.org/core/10_0_0/monitor/org/apache/lucene/monitor/TermFilteredPresearcher.html
import org.apache.lucene.monitor.MonitorConfiguration;
import org.apache.lucene.monitor.TermFilteredPresearcher;
import java.nio.file.Path;

MonitorConfiguration config = new MonitorConfiguration()
        .setIndexPath(Path.of("/var/data/monitor-queries"),  // persist the query set (see below)
                new TermFilteredPresearcher());

try (Monitor monitor = new Monitor(analyzer, config)) {
    // register 100k MonitorQuery objects ...
    MatchingQueries<QueryMatch> result = monitor.match(doc, QueryMatch.SIMPLE_MATCHER);
    // result.getQueriesRun() is now a small fraction of the registered total
}

Compare getQueriesRun() with the registered count to see the presearcher’s selectivity for your workload.

Persisting the query set

By default the Monitor keeps its query index in memory and must be repopulated on restart. Give MonitorConfiguration.setIndexPath a directory and the registered MonitorQuery records are stored on disk and reloaded automatically — the original query strings and metadata are kept so they can be re-parsed. The lucene-monitor module overview documents MonitorConfiguration, the QueryDecomposer that splits disjunctions for better presearcher selectivity, and the QueryCache.

The Maven coordinate is org.apache.lucene:lucene-monitor:

<dependency>
  <groupId>org.apache.lucene</groupId>
  <artifactId>lucene-monitor</artifactId>
  <version>10.0.0</version>
</dependency>

Versus Elasticsearch’s percolator

Elasticsearch’s percolate query is the same reverse-search idea as a managed feature — stored queries indexed in a percolator field type, matched against a document supplied at query time — and it is itself built on this Lucene machinery; see Elasticsearch search extras.

Related pages