Analysis pipeline: Analyzer, TokenStream & attributes

This section documents the current Apache Lucene 10.x line — Lucene 10 requires Java 21 — as published at the Apache Lucene documentation and Javadoc, which is the reference these pages are written and verified against. No specific patch version is pinned; examples target lucene-core 10.x and the companion modules. Some areas (the Panama foreign-memory / Vector API internals, codec file-format internals, and the nightly benchmark harness) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, as Lucene iterates quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

An Analyzer is the factory that produces the TokenStream Lucene consumes when it indexes a text field and, by default, when it parses a query. The stream is assembled from an optional chain of character filters, exactly one tokenizer, and an optional chain of token filters, and each token it emits carries a bundle of attributes — the term text, its character offsets, its position. This page covers how the pipeline is wired, its lifecycle, and the attribute API.

Analyzer: createComponents and normalize

A concrete Analyzer implements createComponents(String fieldName), returning a TokenStreamComponents that bundles the source Tokenizer with the (possibly wrapped) final TokenStream. Lucene calls this once per field per thread and then reuses the components for subsequent values of that field, resetting the chain each time. normalize(String, TokenStream) is the lighter partner path: it applies only the analyzer’s character-level filters (lowercasing, ASCII folding) and is used for TermQuery, PrefixQuery and sorting, where the input is a single token that must not be split.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/analysis/Analyzer.html
Analyzer analyzer = new Analyzer() {
    @Override
    protected TokenStreamComponents createComponents(String fieldName) {
        Tokenizer source = new StandardTokenizer();
        TokenStream result = new LowerCaseFilter(source);
        result = new StopFilter(result, EnglishAnalyzer.getDefaultStopSet());
        return new TokenStreamComponents(source, result);
    }

    @Override
    protected TokenStream normalize(String fieldName, TokenStream in) {
        return new LowerCaseFilter(in);   // no tokenization: keep it one token
    }
};

The CharFilter → Tokenizer → TokenFilter chain

The three stages always run in this order:

  1. Character filters (CharFilter, zero or more) — rewrite the raw character stream before tokenization while tracking a correction map so final offsets still point into the original text. Example: HTMLStripCharFilter, MappingCharFilter.

  2. Tokenizer (Tokenizer, exactly one) — consume the character Reader and break it into tokens, recording each token’s start/end offset and position. Example: StandardTokenizer, WhitespaceTokenizer, KeywordTokenizer.

  3. Token filters (TokenFilter, zero or more) — add, drop or rewrite tokens in order: LowerCaseFilter, StopFilter, SynonymGraphFilter, a stemmer.

Token filters & recipes and Built-in analyzers & CustomAnalyzer cover the concrete components; the package overview is at the org.apache.lucene.analysis package summary.

An input string flowing through char filters

TokenStream lifecycle and reuse

A TokenStream is a strict state machine. A consumer must call the four methods in order, exactly once per pass:

  1. reset() — rewind to the start before the first incrementToken().

  2. incrementToken() — advance to the next token, populating the shared attributes; returns false at end of stream.

  3. end() — called after the last token to set end-of-stream state such as the final offset.

  4. close() — release the input; the stream can be re-reset() and used again for the next value.

Because Analyzer caches TokenStreamComponents per thread, the same stream objects are reused across every value of a field — which is why reset() and end() exist and why a custom TokenFilter must reset its own state in reset().

The attribute API

Tokens are not objects. A TokenStream extends AttributeSource: attribute instances are created once with addAttribute(…​), shared by the whole chain, and mutated in place on each incrementToken(). The consumer reads whichever attributes it needs after each call.

Attribute Carries

CharTermAttribute

The token’s text (as a mutable char buffer).

OffsetAttribute

Start and end character offsets in the original input — what the UnifiedHighlighter uses to underline a match.

PositionIncrementAttribute

Gap to the previous token: 1 normally, 0 for a synonym stacked on the same position, >1 where a stop word was removed. Phrase and span queries depend on it.

PositionLengthAttribute

How many positions the token spans — >1 for a multi-word synonym in a graph.

TypeAttribute

The tokenizer’s classification, e.g. <ALPHANUM>, <NUM>.

PayloadAttribute

An arbitrary BytesRef stored with the term’s position (per-term weights, part-of-speech tags).

KeywordAttribute

Marks a token as a keyword so downstream stemmers leave it untouched.

Positions and offsets are what make later features possible: PhraseQuery and the interval and span queries match on positions, while offsets let the UnifiedHighlighter map a matched term back to a span of source characters without re-analyzing.

// https://lucene.apache.org/core/10_0_0/core/org/apache/lucene/analysis/tokenattributes/package-summary.html
Analyzer analyzer = new StandardAnalyzer();
try (TokenStream ts = analyzer.tokenStream("body", "The 2 QUICK brown-foxes jumped")) {
    var term   = ts.addAttribute(CharTermAttribute.class);
    var offset = ts.addAttribute(OffsetAttribute.class);
    var posInc = ts.addAttribute(PositionIncrementAttribute.class);

    int position = 0;
    ts.reset();
    while (ts.incrementToken()) {
        position += posInc.getPositionIncrement();
        System.out.printf("term=%-8s pos=%d offset=[%d,%d]%n",
                term, position, offset.startOffset(), offset.endOffset());
    }
    ts.end();
}
// term=the      pos=1 offset=[0,3]
// term=2        pos=2 offset=[4,5]
// term=quick    pos=3 offset=[6,11]
// term=brown    pos=4 offset=[12,17]
// term=foxes    pos=5 offset=[18,23]
// term=jumped   pos=6 offset=[24,30]

Related pages