Pluggable language modules

This section documents the current Apache Lucene 10.x line — Lucene 10 requires Java 21 — as published at the Apache Lucene documentation and Javadoc, which is the reference these pages are written and verified against. No specific patch version is pinned; examples target lucene-core 10.x and the companion modules. Some areas (the Panama foreign-memory / Vector API internals, codec file-format internals, and the nightly benchmark harness) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, as Lucene iterates quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

lucene-analysis-common handles English-like whitespace-and-punctuation text well, but a German compound, an un-spaced Japanese sentence, or an Arabic word with optional diacritics each need a purpose-built tokenizer and normaliser. Lucene ships these as separate modules, each its own Maven/Gradle artifact and SPI bundle, that you add only for the languages you actually index. This page is a map of which module solves which problem and how to wire one in.

The modules at a glance

Module Covers Reach for it when

lucene-analysis-icu

Unicode-correct tokenisation (ICUTokenizer), case/accent folding (ICUFoldingFilter), NFC/NFKC normalisation (ICUNormalizer2Filter), script-aware transforms (ICUTransformFilter)

You index more than one script, or need better-than-ASCIIFoldingFilter folding and Unicode normalisation. A good default base layer for any multilingual index.

lucene-analysis-kuromoji

Japanese morphological analysis — JapaneseTokenizer (dictionary + Viterbi), base-form, part-of-speech stop, reading, katakana-stem filters

You index Japanese. There is no whitespace to split on; you need dictionary segmentation.

lucene-analysis-nori

Korean morphological analysis — KoreanTokenizer, KoreanPartOfSpeechStopFilter, KoreanReadingFormFilter

You index Korean; agglutinative morphology needs decomposition into morphemes.

lucene-analysis-smartcn

Simplified Chinese — HMMChineseTokenizer (hidden-Markov word segmentation) plus a Chinese stop filter

You index Simplified Chinese and want word-level (not bigram) segmentation.

lucene-analysis-stempel

Algorithmic Polish stemming (StempelStemmer) trained on a stemmer table

You index Polish and want a compact algorithmic stemmer.

lucene-analysis-morfologik

Dictionary lemmatisation via Morfologik FSA dictionaries — Polish, plus Ukrainian and others

You index Polish/Ukrainian and want dictionary-accurate lemmas rather than an algorithmic guess.

lucene-analysis-phonetic

Sounds-alike matching — PhoneticFilter (Soundex, RefinedSoundex, Metaphone, Caverphone, Cologne), DoubleMetaphoneFilter, BeiderMorseFilter

You match names/misspellings by pronunciation (people search, deduplication).

lucene-analysis-opennlp

ML-model NLP — sentence detection, tokenisation, POS tagging, chunking, lemmatisation, named-entity types, driven by Apache OpenNLP models you supply

You need linguistic annotation (POS-aware stop removal, NER-typed tokens) and can ship trained models.

Each has a module page under https://lucene.apache.org/core/10_0_0/ — for example analysis-icu, analysis-kuromoji, analysis-nori, analysis-smartcn, analysis-stempel, analysis-morfologik, analysis-phonetic and analysis-opennlp.

Adding a module

Modules are ordinary dependencies; adding one puts its analyzers, tokenizers and filters on the SPI path so CustomAnalyzer can resolve them by name.

<dependency>
  <groupId>org.apache.lucene</groupId>
  <artifactId>lucene-analysis-icu</artifactId>
  <version>${lucene.version}</version>
</dependency>
<dependency>
  <groupId>org.apache.lucene</groupId>
  <artifactId>lucene-analysis-kuromoji</artifactId>
  <version>${lucene.version}</version>
</dependency>

ICU as a base layer

ICUFoldingFilter combines case folding, accent removal and NFKC normalisation in one pass and is a stronger default than LowerCaseFilter + ASCIIFoldingFilter for anything touching non-English text. ICUNormalizer2Filter alone applies a chosen normalisation form when you want folding decisions left to a later language filter.

// https://lucene.apache.org/core/10_0_0/analysis/icu/org/apache/lucene/analysis/icu/ICUFoldingFilter.html
Analyzer icuBase = CustomAnalyzer.builder()
    .withTokenizer("icu")                 // ICUTokenizer: script-boundary aware
    .addTokenFilter("icuFolding")
    .build();

Wiring a non-Latin analyzer (Japanese)

JapaneseTokenizer needs no whitespace; mode=search splits compounds into searchable sub-tokens. The follow-on filters normalise width, strip inflection to the dictionary base form, and drop particle/aux part-of-speech classes.

// https://lucene.apache.org/core/10_0_0/analysis/kuromoji/org/apache/lucene/analysis/ja/JapaneseTokenizer.html
Analyzer japanese = CustomAnalyzer.builder()
    .withTokenizer("japanese", "mode", "search")     // SPI name of JapaneseTokenizerFactory
    .addTokenFilter("cjkWidth")                       // full-width ASCII / half-width kana -> normal
    .addTokenFilter("japaneseBaseForm")              // inflected -> dictionary form
    .addTokenFilter("japanesePartOfSpeechStop", "tags", "stoptags.txt")
    .addTokenFilter("japaneseKatakanaStem", "minimumLength", "4")
    .addTokenFilter("lowercase")                      // for embedded Latin text
    .build();

// The ready-made org.apache.lucene.analysis.ja.JapaneseAnalyzer bundles a very similar chain.

Korean (nori) and Chinese (smartcn) follow the same pattern — a dictionary/HMM tokenizer plus a part-of-speech or stop filter — with .withTokenizer("korean") / .withTokenizer("hmmChinese").

Phonetic matching

PhoneticFilter wraps any Apache Commons Codec Encoder; BeiderMorseFilter is the most accurate for personal names across languages. Add it as a parallel field, not in the main analysis chain, so exact matches still rank above sound-alikes.

// https://lucene.apache.org/core/10_0_0/analysis/phonetic/org/apache/lucene/analysis/phonetic/BeiderMorseFilter.html
Analyzer nameSounds = new Analyzer() {
    @Override
    protected TokenStreamComponents createComponents(String field) {
        Tokenizer src = new StandardTokenizer();
        TokenStream tok = new LowerCaseFilter(src);
        tok = new BeiderMorseFilter(tok,
            new PhoneticEngine(NameType.GENERIC, RuleType.APPROX, true), null);
        return new TokenStreamComponents(src, tok);
    }
};
// Simpler encoders: new PhoneticFilter(tok, new DoubleMetaphone(), false)
//                   new PhoneticFilter(tok, new Soundex(), true)

Related pages