Token-filter recipes

This section documents the current Apache Lucene 10.x line — Lucene 10 requires Java 21 — as published at the Apache Lucene documentation and Javadoc, which is the reference these pages are written and verified against. No specific patch version is pinned; examples target lucene-core 10.x and the companion modules. Some areas (the Panama foreign-memory / Vector API internals, codec file-format internals, and the nightly benchmark harness) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, as Lucene iterates quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

Once the analysis pipeline and CustomAnalyzer are in place, most real work is choosing the right token filters and ordering them correctly. This page is a set of recipes — each a self-contained CustomAnalyzer or Analyzer — for the filters in the lucene-analysis-common module that come up again and again. Every class here has SPI-name and factory forms; see the analysis-common package summary for the full catalogue.

Folding: lower-casing and accent removal

LowerCaseFilter normalises case; ASCIIFoldingFilter maps accented Latin-1/Latin-Extended characters to their ASCII base (café → cafe), optionally keeping the original token with preserveOriginal=true so an exact accented match still scores. Fold with the same chain at index and query time.

// https://lucene.apache.org/core/10_0_0/analysis/common/org/apache/lucene/analysis/miscellaneous/ASCIIFoldingFilter.html
Analyzer folding = CustomAnalyzer.builder()
    .withTokenizer("standard")
    .addTokenFilter("lowercase")
    .addTokenFilter("asciiFolding", "preserveOriginal", "false")
    .build();

For non-Latin scripts reach for the ICU filters instead — see Pluggable language modules.

Stop words

StopFilter drops high-frequency, low-signal terms. Pass an explicit CharArraySet rather than relying on a language default when you can, and remember that removing stop words changes phrase positions — a slop-0 "to be or not to be" phrase cannot match an index that dropped every word in it.

// https://lucene.apache.org/core/10_0_0/analysis/common/org/apache/lucene/analysis/core/StopFilter.html
CharArraySet stop = new CharArraySet(Arrays.asList("a", "an", "the", "of", "and"), true);

Analyzer withStop = CustomAnalyzer.builder()
    .withTokenizer("standard")
    .addTokenFilter("lowercase")
    .addTokenFilter("stop", "words", "stopwords.txt", "ignoreCase", "true") // file on the resource path
    .build();
// or programmatically: new StopFilter(tokenStream, stop)

Synonyms: SynonymGraphFilter + FlattenGraphFilter

SynonymGraphFilter emits a token graph so multi-word synonyms (ny <→ new york) keep correct positions. A graph cannot be written to the index as-is, so the index analyzer must append FlattenGraphFilter as the last filter; the query analyzer uses SynonymGraphFilter alone (the query path consumes the graph directly).

flowchart LR subgraph index [Index analyzer] A[tokenizer] --> B[LowerCaseFilter] --> C[SynonymGraphFilter] --> D[FlattenGraphFilter] --> E[(postings)] end subgraph query [Query analyzer] F[tokenizer] --> G[LowerCaseFilter] --> H[SynonymGraphFilter] --> I[Query graph] end
// https://lucene.apache.org/core/10_0_0/analysis/common/org/apache/lucene/analysis/synonym/SynonymGraphFilter.html
SynonymMap.Builder b = new SynonymMap.Builder(true);
b.add(new CharsRef("ny"), new CharsRef("new york"), true);
b.add(new CharsRef("laptop"), new CharsRef("notebook"), true);
final SynonymMap synonyms = b.build();

Analyzer indexAnalyzer = new Analyzer() {
    @Override
    protected TokenStreamComponents createComponents(String field) {
        Tokenizer src = new StandardTokenizer();
        TokenStream tok = new LowerCaseFilter(src);
        tok = new SynonymGraphFilter(tok, synonyms, true);
        tok = new FlattenGraphFilter(tok);            // index-time only
        return new TokenStreamComponents(src, tok);
    }
};
// Query analyzer: identical but WITHOUT FlattenGraphFilter.

Stemming and lemmatization

Four interchangeable reducers, roughly from most aggressive to least:

Filter Notes

PorterStemFilter

Classic Porter algorithm, English only, no options; fast and aggressive (generously → gener).

KStemFilter

Less aggressive English stemmer that stays closer to real words (generously → generous); good default for English relevance.

SnowballFilter

Snowball (Porter2) with a language parameter — English, French, German, Spanish, Russian, …​ — the multi-language choice.

StemmerOverrideFilter

Not a stemmer: a dictionary that pins listed words to a fixed stem and marks them keyword so a later stemmer leaves them alone. Place it before the real stemmer.

 
// https://lucene.apache.org/core/10_0_0/analysis/common/org/apache/lucene/analysis/en/KStemFilter.html
// https://lucene.apache.org/core/10_0_0/analysis/common/org/apache/lucene/analysis/miscellaneous/StemmerOverrideFilter.html
StemmerOverrideFilter.Builder ob = new StemmerOverrideFilter.Builder();
ob.add("mice", "mouse");
ob.add("aging", "age");
final StemmerOverrideFilter.StemmerOverrideMap overrides = ob.build();

Analyzer english = new Analyzer() {
    @Override
    protected TokenStreamComponents createComponents(String field) {
        Tokenizer src = new StandardTokenizer();
        TokenStream tok = new LowerCaseFilter(src);
        tok = new StemmerOverrideFilter(tok, overrides);
        tok = new KStemFilter(tok);                   // swap for PorterStemFilter / SnowballFilter
        return new TokenStreamComponents(src, tok);
    }
};

Autocomplete and partial matching: n-grams

Two shapes:

  • EdgeNGramTokenFilter — prefix grams of each token (search → s, se, sea, sear, searc, search). Apply it in the index analyzer only; the query analyzer feeds the raw user prefix so sea matches the sea gram. This is the standard "search-as-you-type" recipe.

  • NGramTokenizer / NGramTokenFilter — interior grams too, for substring ("infix") matching.

Both multiply term volume: a 20-character field with minGramSize=1 produces up to ~20 terms per token, so the term dictionary and postings grow several-fold. Bound minGramSize (2 or 3, not 1) and maxGramSize, and prefer a dedicated field so the cost is not paid on every query.

// https://lucene.apache.org/core/10_0_0/analysis/common/org/apache/lucene/analysis/ngram/EdgeNGramTokenFilter.html
Analyzer autocompleteIndex = CustomAnalyzer.builder()
    .withTokenizer("standard")
    .addTokenFilter("lowercase")
    .addTokenFilter("asciiFolding")
    .addTokenFilter("edgeNGram", "minGramSize", "2", "maxGramSize", "20", "preserveOriginal", "true")
    .build();

Analyzer autocompleteQuery = CustomAnalyzer.builder()   // NO edgeNGram here
    .withTokenizer("standard")
    .addTokenFilter("lowercase")
    .addTokenFilter("asciiFolding")
    .build();

For a ranked suggester that is not a plain analyzer chain, see Highlighting, suggesters & more.

ShingleFilter: token n-grams for phrase-ish matching

ShingleFilter joins adjacent tokens into "shingles" (the quick fox → the quick, quick fox), which lets a plain TermQuery approximate a phrase match and feeds bigram-boost strategies. Keep outputUnigrams=true unless the field is used only for shingle matching.

// https://lucene.apache.org/core/10_0_0/analysis/common/org/apache/lucene/analysis/shingle/ShingleFilter.html
Analyzer shingles = CustomAnalyzer.builder()
    .withTokenizer("standard")
    .addTokenFilter("lowercase")
    .addTokenFilter("shingle",
        "minShingleSize", "2", "maxShingleSize", "2",
        "outputUnigrams", "true", "tokenSeparator", " ")
    .build();

WordDelimiterGraphFilter: splitting mixed tokens

Splits Wi-Fi, PowerShot, SD500 and iPhone10 on case changes, letter/number boundaries and intra-word delimiters, and can re-concatenate the parts. Like the synonym filter it produces a graph, so the index analyzer ends with FlattenGraphFilter. Run it before LowerCaseFilter when splitOnCaseChange is on, or the case information is already gone.

// https://lucene.apache.org/core/10_0_0/analysis/common/org/apache/lucene/analysis/miscellaneous/WordDelimiterGraphFilter.html
Analyzer partNumbers = new Analyzer() {
    @Override
    protected TokenStreamComponents createComponents(String field) {
        Tokenizer src = new WhitespaceTokenizer();
        int flags = WordDelimiterGraphFilter.GENERATE_WORD_PARTS
                  | WordDelimiterGraphFilter.GENERATE_NUMBER_PARTS
                  | WordDelimiterGraphFilter.CATENATE_WORDS
                  | WordDelimiterGraphFilter.SPLIT_ON_CASE_CHANGE
                  | WordDelimiterGraphFilter.PRESERVE_ORIGINAL;
        TokenStream tok = new WordDelimiterGraphFilter(src, flags, null);
        tok = new LowerCaseFilter(tok);
        tok = new FlattenGraphFilter(tok);           // index-time only
        return new TokenStreamComponents(src, tok);
    }
};

DelimitedPayloadTokenFilter: per-term payloads

Reads a delimiter-suffixed weight off each token (fox|0.9) and stores it as a term payload — a few bytes attached to each posting, later read back by a payload-aware query or similarity for term-level weighting (importance tags, part-of-speech weights, per-occurrence scores).

// https://lucene.apache.org/core/10_0_0/analysis/common/org/apache/lucene/analysis/payloads/DelimitedPayloadTokenFilter.html
Analyzer payloads = new Analyzer() {
    @Override
    protected TokenStreamComponents createComponents(String field) {
        Tokenizer src = new WhitespaceTokenizer();       // input: "quick|0.2 fox|0.9"
        TokenStream tok = new LowerCaseFilter(src);
        tok = new DelimitedPayloadTokenFilter(tok, '|', new FloatEncoder());
        return new TokenStreamComponents(src, tok);
    }
};
// Read them back with PostingsEnum.getPayload(), or score with
// org.apache.lucene.queries.payloads.PayloadScoreQuery.

Related pages