Built-in analyzers & CustomAnalyzer

This section documents the current Apache Lucene 10.x line — Lucene 10 requires Java 21 — as published at the Apache Lucene documentation and Javadoc, which is the reference these pages are written and verified against. No specific patch version is pinned; examples target lucene-core 10.x and the companion modules. Some areas (the Panama foreign-memory / Vector API internals, codec file-format internals, and the nightly benchmark harness) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, as Lucene iterates quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

Most indexes never need a hand-written Analyzer. The lucene-analysis-common module ships a set of ready-made ones, PerFieldAnalyzerWrapper lets different fields use different analyzers, and CustomAnalyzer.builder() assembles a chain from named factories without a subclass. This page tours those options and how to test the result. For how the pipeline itself works, see Analysis pipeline.

The built-in analyzers

All of these are in lucene-analysis-common (package org.apache.lucene.analysis.*), except KeywordAnalyzer which is in lucene-core.

Analyzer Chain and use

Analyzer	Chain and use
`StandardAnalyzer`	`StandardTokenizer` + `LowerCaseFilter` + `StopFilter` (empty stop set by default). Unicode word-boundary segmentation; the sensible default for most text.
`SimpleAnalyzer`	`LetterTokenizer` + `LowerCaseFilter` — splits on any non-letter, so digits are discarded. Rarely what you want for real data.
`WhitespaceAnalyzer`	`WhitespaceTokenizer` only — splits on whitespace, no lowercasing, no punctuation handling. Good for pre-tokenized input or codes.
`KeywordAnalyzer`	`KeywordTokenizer` — the whole input becomes one token, unchanged. Equivalent to indexing a `StringField`; use that instead when you can.
`StopAnalyzer`	`LetterTokenizer` + `LowerCaseFilter` + `StopFilter` — `SimpleAnalyzer` plus stop-word removal.
`EnglishAnalyzer`	`StandardTokenizer` + `EnglishPossessiveFilter` + `LowerCaseFilter` + `StopFilter` + `PorterStemFilter`, with a `KeywordAttribute` exclusion set. The template every language analyzer follows.

StandardAnalyzer

StandardTokenizer + LowerCaseFilter + StopFilter (empty stop set by default). Unicode word-boundary segmentation; the sensible default for most text.

SimpleAnalyzer

LetterTokenizer + LowerCaseFilter — splits on any non-letter, so digits are discarded. Rarely what you want for real data.

WhitespaceAnalyzer

WhitespaceTokenizer only — splits on whitespace, no lowercasing, no punctuation handling. Good for pre-tokenized input or codes.

KeywordAnalyzer

KeywordTokenizer — the whole input becomes one token, unchanged. Equivalent to indexing a StringField; use that instead when you can.

StopAnalyzer

LetterTokenizer + LowerCaseFilter + StopFilter — SimpleAnalyzer plus stop-word removal.

EnglishAnalyzer

StandardTokenizer + EnglishPossessiveFilter + LowerCaseFilter + StopFilter + PorterStemFilter, with a KeywordAttribute exclusion set. The template every language analyzer follows.

lucene-analysis-common carries a matching analyzer for roughly forty languages — FrenchAnalyzer, GermanAnalyzer, RussianAnalyzer, CJKAnalyzer, and so on — each pairing the standard tokenizer with that language’s stop words and stemmer. See Language analysis for choosing and tuning them.

// https://lucene.apache.org/core/10_0_0/analysis/common/org/apache/lucene/analysis/en/EnglishAnalyzer.html
Analyzer std = new StandardAnalyzer();
Analyzer english = new EnglishAnalyzer();          // stemming + English stop words
Analyzer french = new FrenchAnalyzer();

// Provide your own stop words instead of the language default:
Analyzer custom = new EnglishAnalyzer(
        StopFilter.makeStopSet("brand", "sku", "the", "a"));

Maven coordinate

<dependency>
  <groupId>org.apache.lucene</groupId>
  <artifactId>lucene-analysis-common</artifactId>
  <version>10.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.lucene/lucene-analysis-common -->

PerFieldAnalyzerWrapper

A single index usually has fields that want different treatment — an sku that must stay verbatim, a body that wants English stemming, a name that wants only lowercasing. PerFieldAnalyzerWrapper takes a default analyzer plus a per-field map and dispatches by field name; pass it to IndexWriterConfig and to the query parser so both sides analyze consistently.

// https://lucene.apache.org/core/10_0_0/analysis/common/org/apache/lucene/analysis/miscellaneous/PerFieldAnalyzerWrapper.html
var perField = new PerFieldAnalyzerWrapper(
        new StandardAnalyzer(),                 // default for unlisted fields
        Map.of(
            "sku", new KeywordAnalyzer(),
            "body", new EnglishAnalyzer(),
            "name", new WhitespaceAnalyzer()));

var config = new IndexWriterConfig(perField);

CustomAnalyzer.builder(): a chain without a subclass

CustomAnalyzer.builder() builds an analyzer at runtime from the SPI names of the tokenizer and filter factories (the same short names Solr and Elasticsearch use in their schemas — standard, lowercase, stop, asciifolding, edgeNGram, …), each with a map of string parameters. This is the idiomatic way to make analysis config data rather than code, and it reads a resource directory for files such as stop-word or synonym lists.

// https://lucene.apache.org/core/10_0_0/analysis/common/org/apache/lucene/analysis/custom/CustomAnalyzer.html
Analyzer analyzer = CustomAnalyzer.builder(Path.of("conf/analysis"))
        .withTokenizer("standard")
        .addCharFilter("htmlStrip")
        .addTokenFilter("lowercase")
        .addTokenFilter("stop", Map.of(
                "ignoreCase", "true",
                "words", "stopwords.txt",
                "format", "wordset"))
        .addTokenFilter("asciiFolding", Map.of("preserveOriginal", "false"))
        .addTokenFilter("porterStem")
        .build();

Call .builder() with no argument when no factory needs to load a file. Unknown factory names fail fast at build() with the list of available names.

Testing an analyzer

For a quick check, consume the TokenStream and print the terms — the pattern in Analysis pipeline. For real coverage, depend on lucene-test-framework and extend BaseTokenStreamTestCase: its assertAnalyzesTo verifies the exact terms, offsets, positions and types an analyzer produces for an input, and its checkRandomData helper hammers the analyzer with generated strings to surface lifecycle bugs (missing reset(), offsets going backwards).

<dependency>
  <groupId>org.apache.lucene</groupId>
  <artifactId>lucene-test-framework</artifactId>
  <version>10.0.0</version>
  <scope>test</scope>
</dependency>

// https://lucene.apache.org/core/10_0_0/test-framework/org/apache/lucene/tests/analysis/BaseTokenStreamTestCase.html
public class ProductAnalyzerTest extends BaseTokenStreamTestCase {

    public void testStemmingAndStopWords() throws Exception {
        Analyzer analyzer = CustomAnalyzer.builder()
                .withTokenizer("standard")
                .addTokenFilter("lowercase")
                .addTokenFilter("stop")
                .addTokenFilter("porterStem")
                .build();

        assertAnalyzesTo(
                analyzer,
                "The Running Shoes",
                new String[] {"run", "shoe"},   // expected terms
                new int[]    {4, 12},            // start offsets
                new int[]    {11, 17},           // end offsets
                new int[]    {2, 1});            // position increments ("the" removed)

        checkRandomData(random(), analyzer, 200);
        analyzer.close();
    }
}

Analysis pipeline — the Analyzer / TokenStream / attribute model these build on.
Token filters & recipes — the factory names available to CustomAnalyzer.
Language analysis — the per-language analyzer set.
Testing tools & modules — more of lucene-test-framework.
Elasticsearch: text analysis and Solr: text analysis — the same factory names in a schema.
The lucene-analysis-common module Javadoc and CustomAnalyzer Javadoc.