Language analysis: per-language stemming, lemmatization & detection

This section documents the current Solr line (10.0; 9.10.x the maintained 9.x branch), written and verified against the Apache Solr Reference Guide. No specific patch version is pinned. Some capabilities (the Solr Operator on Kubernetes, the package-manager ecosystem, Learning To Rank model training, and expert plugin development) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Solr’s default analysis chains (see Text analysis: analyzers, tokenizers & token filters) are tuned for English-like whitespace-and-punctuation text. Real multilingual content needs more: a German compound word, an Arabic word written with or without diacritics, and a Japanese sentence with no spaces at all each need a different tokenizer and a different notion of "the same word." This page covers Solr’s per-language field types and analyzer chains, the stemming-vs-lemmatization trade-off (including Hunspell dictionary stemming and OpenNLP lemmatization), ICU folding, phonetic matching, and the langid update processor that can detect a document’s language and route it to the right field automatically. See Language Analysis for the full per-language reference this page draws from.

Strategies for multilingual content

Before picking filters, decide how languages map onto the schema. Three approaches are common, and they are not mutually exclusive within one deployment:

Strategy How it works and when to use it

Strategy	How it works and when to use it
Field-per-language	One field per language per logical attribute (`title_en`, `title_de`, `title_fr`, …), each backed by its own `fieldType` and analyzer chain. Indexing writes the document’s text into the field matching its language; querying searches the matching field, or all of them with an edismax `qf` list. This is the default recommendation when the language set is known and bounded — see Schema & fields and Field types for how fields and field types are declared, and Query parsers for multi-field query syntax.
Core-per-language (or collection-per-language)	A separate core or collection per language, each with its own schema and configset, so per-language tuning (stemmers, stopwords, synonyms, caches) is fully isolated and one language’s reindex or schema change cannot affect another’s. Cross-language search then means querying multiple collections and merging results (or fronting them with an aggregator), which costs more operationally. See Collections API, configsets & replica placement and SolrCloud architecture.
Multiple languages in one field	A single field with a generic, language-agnostic chain (Unicode tokenization plus ICU folding and normalization, no language-specific stemming) that degrades gracefully across languages. Simplest to operate, but no field gets language-specific stemming, so recall and ranking quality drop relative to a tuned per-language field. Reasonable when the language mix is unknown per document or too broad to enumerate.

Field-per-language

One field per language per logical attribute (title_en, title_de, title_fr, …), each backed by its own fieldType and analyzer chain. Indexing writes the document’s text into the field matching its language; querying searches the matching field, or all of them with an edismax qf list. This is the default recommendation when the language set is known and bounded — see Schema & fields and Field types for how fields and field types are declared, and Query parsers for multi-field query syntax.

Core-per-language (or collection-per-language)

A separate core or collection per language, each with its own schema and configset, so per-language tuning (stemmers, stopwords, synonyms, caches) is fully isolated and one language’s reindex or schema change cannot affect another’s. Cross-language search then means querying multiple collections and merging results (or fronting them with an aggregator), which costs more operationally. See Collections API, configsets & replica placement and SolrCloud architecture.

Multiple languages in one field

A single field with a generic, language-agnostic chain (Unicode tokenization plus ICU folding and normalization, no language-specific stemming) that degrades gracefully across languages. Simplest to operate, but no field gets language-specific stemming, so recall and ranking quality drop relative to a tuned per-language field. Reasonable when the language mix is unknown per document or too broad to enumerate.

A langid update processor (below) can automate the field-per-language approach by detecting each document’s language at index time and writing (or renaming) into the matching field, so the client does not have to know the language in advance.

Language-specific field types and analyzer chains

Each language ships as a pre-built fieldType in the example managed-schema (text_en, text_de, text_fr, text_ar, text_ja, text_zh, …), each combining a tokenizer with the token filters appropriate for that language’s morphology. Declare or inspect one through the Schema API like any other field type — see Schema & fields for the managed schema and the Schema API in general.

# Inspect the built-in English field type and its analyzer chain.
curl "http://localhost:8983/solr/books/schema/fieldtypes/text_en?wt=json"
# https://solr.apache.org/guide/solr/latest/indexing-guide/language-analysis.html

A representative German field type — a language with productive compounding and case-sensitive capitalization rules — layers a decompounder ahead of a language-specific stemmer:

{
  "add-field-type": {
    "name": "text_de_tuned",
    "class": "solr.TextField",
    "positionIncrementGap": "100",
    "analyzer": {
      "tokenizer": { "class": "solr.StandardTokenizerFactory" },
      "filters": [
        { "class": "solr.LowerCaseFilterFactory" },
        { "class": "solr.GermanNormalizationFilterFactory" },
        { "class": "solr.SnowballPorterFilterFactory", "language": "German2" }
      ]
    }
  }
}

Post that with the Schema API’s add-field-type command against /solr/<collection>/schema, the same endpoint used for any managed-schema change. A handful of languages need more than a tokenizer and a stemmer: Japanese uses solr.JapaneseTokenizerFactory (Kuromoji, with a mode of normal, search, or extended) followed by JapaneseBaseFormFilterFactory and JapanesePartOfSpeechStopFilterFactory; Korean uses solr.KoreanTokenizerFactory (Nori); Chinese uses solr.HMMChineseTokenizerFactory for simplified-Chinese segmentation, or solr.CJKBigramFilterFactory as a segmenter-free fallback across Chinese, Japanese and Korean. Two filters guard stemming from over-eager rewrites regardless of language: solr.KeywordMarkerFilterFactory (with a protected word list) exempts specific terms from every downstream stemmer, and solr.KeywordRepeatFilterFactory emits both the original and the stemmed token at the same position — pair it with solr.RemoveDuplicatesTokenFilterFactory so an unstemmed exact match and a stemmed fuzzy match can both be found from the same field.

Stemming vs. lemmatization

Stemming strips a word down to a root by rule, without knowing the word’s part of speech or whether the result is a real word ("organization" → "organ" is a classic over-stem). Lemmatization instead maps a word to its dictionary base form (a lemma) using either a dictionary or a trained model that is usually part-of-speech aware, so "better" can correctly resolve to "good" rather than being left untouched. Solr offers three tiers, in increasing order of accuracy and cost:

Algorithmic stemming: Snowball

solr.SnowballPorterFilterFactory runs a Snowball/Porter-family algorithm for one language parameter ("English", "French", "German2", "Spanish", "Russian", …). It is fast and needs no external dictionary, at the cost of occasional over- or under-stemming.

<!-- Field type definition in the managed schema, applied at both index and query time. -->
<fieldType name="text_es" class="solr.TextField">
  <analyzer>
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.SnowballPorterFilterFactory" language="Spanish"/>
  </analyzer>
</fieldType>
<!-- https://solr.apache.org/guide/solr/latest/indexing-guide/language-analysis.html -->

Hunspell dictionary stemming

solr.HunspellStemFilterFactory stems against a real Hunspell .dic/.aff dictionary pair (the same format used by many spell-checkers and word processors), which is markedly more accurate than an algorithmic stemmer for languages with irregular morphology, at the cost of loading the dictionary into memory and being slower per token. Dictionaries live under the collection’s conf/ directory alongside the configset:

<filter class="solr.HunspellStemFilterFactory"
        dictionary="hunspell/en_US.dic"
        affix="hunspell/en_US.aff"
        ignoreCase="true"/>
<!-- https://solr.apache.org/guide/solr/latest/indexing-guide/language-analysis.html -->

OpenNLP lemmatization

For true lemmatization, Solr’s OpenNLP integration chains solr.OpenNLPTokenizerFactory with solr.OpenNLPPOSFilterFactory (part-of-speech tagging, needed because a lemma often depends on whether a word is a noun or a verb) and solr.OpenNLPLemmatizerFilterFactory, which can run either a dictionary-based or a statistical-model-based lemmatizer. Each stage needs its own pretrained OpenNLP model file, referenced by path in the analyzer definition:

<analyzer>
  <tokenizer class="solr.OpenNLPTokenizerFactory"
             sentenceModel="opennlp/en-sent.bin"
             tokenizerModel="opennlp/en-tokenizer.bin"/>
  <filter class="solr.OpenNLPPOSFilterFactory" posTaggerModel="opennlp/en-pos-maxent.bin"/>
  <filter class="solr.OpenNLPLemmatizerFilterFactory"
          dictionary="opennlp/en-lemmatizer.dict"
          lemmatizerModel="opennlp/en-lemmatizer.bin"/>
</analyzer>
<!-- https://solr.apache.org/guide/solr/latest/indexing-guide/language-analysis.html -->

Use the Analysis API to compare all three on the same input before committing to one in a schema — this is the closest Solr equivalent to Elasticsearch’s _analyze endpoint:

curl -G "http://localhost:8983/solr/books/analysis/field" \
  --data-urlencode "analysis.fieldtype=text_en" \
  --data-urlencode "analysis.fieldvalue=The runners were running quickly" \
  --data-urlencode "wt=json"
# Returns the token stream produced by each stage of the text_en analyzer chain.

ICU folding and normalization

asciifolding-style accent stripping only works for Latin scripts. solr.ICUFoldingFilterFactory (shipped in the analysis-extras module) applies Unicode NFKC normalization, case folding, and accent removal in one pass, and does so correctly across non-Latin scripts as well — solr.ASCIIFoldingFilterFactory folds café to cafe but has no useful behavior for, say, Cyrillic or Greek input. solr.ICUNormalizer2FilterFactory applies just the Unicode normalization step without folding case or accents, and solr.ICUTransformFilterFactory runs an arbitrary ICU transliteration rule (for example Cyrillic-Latin or Any-Latin; NFC) for script-to-script conversion. For CJK text specifically, solr.CJKWidthFilterFactory normalizes fullwidth and halfwidth character variants that would otherwise index as distinct tokens.

<filter class="solr.ICUFoldingFilterFactory"/>
<!-- https://solr.apache.org/guide/solr/latest/indexing-guide/language-analysis.html -->

Phonetic matching

Phonetic filters index a word alongside a code representing how it sounds, so misspellings and transliteration variants of the same name can still match. solr.PhoneticFilterFactory wraps several general-purpose encoders selected with the encoder parameter: DoubleMetaphone, Metaphone, Soundex, RefinedSoundex, Caverphone2, ColognePhonetic, and Nysiis. Two more specialized factories exist for names: solr.BeiderMorseFilterFactory (Beider-Morse Phonetic Matching, tuned across roughly a dozen languages for personal and place names, with fewer false hits than plain Soundex) and solr.DaitchMokotoffSoundexFilterFactory (a Soundex refinement especially accurate for Slavic and Yiddish surnames, which can emit multiple codes per token). See Phonetic Matching for the full comparison and each encoder’s trade-offs.

Because phonetic encoding is lossy, it is normally indexed into a separate field via copyField rather than replacing the exact-text field, so exact and phonetic queries both remain possible:

<fieldType name="phonetic_dm" class="solr.TextField">
  <analyzer>
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.DoubleMetaphoneFilterFactory" inject="false"/>
  </analyzer>
</fieldType>

<field name="name" type="text_general" indexed="true" stored="true"/>
<field name="name_phonetic" type="phonetic_dm" indexed="true" stored="false"/>
<copyField source="name" dest="name_phonetic"/>
<!-- https://solr.apache.org/guide/solr/latest/indexing-guide/phonetic-matching.html -->

# "Smyth" and "Smith" both encode to the same Double Metaphone code and match.
curl -G "http://localhost:8983/solr/books/select" \
  --data-urlencode "q=name_phonetic:Smyth"

inject="false" replaces the original token with the phonetic code instead of adding it as a second token at the same position; leave it at the default true when the phonetic field should also match exact spellings. See Spell checking, suggestions & similar documents for the complementary, edit-distance-based approach to fuzzy name and term matching.

Automatic language identification with the langid processor

Rather than requiring the client to tag every document with its language, an update request processor can detect it during indexing and act on the result. Two implementations ship with Solr: LangDetectLangIdUpdateProcessorFactory (statistical, based on the language-detection library) and OpenNLPLangDetectUpdateProcessorFactory (uses a pretrained OpenNLP language-detection model). Both are configured the same way in solrconfig.xml’s update request processor chain — `langid.fl names the source field(s) to inspect, and langid.langField is where the detected code is written:

<updateRequestProcessorChain name="langid">
  <processor class="solr.LangDetectLangIdUpdateProcessorFactory">
    <str name="langid.fl">title,body</str>
    <str name="langid.langField">language_s</str>
    <str name="langid.fallback">en</str>
    <bool name="langid.map.lcmap">true</bool>
  </processor>
  <processor class="solr.LogUpdateProcessorFactory"/>
  <processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>
<!-- https://solr.apache.org/guide/solr/latest/indexing-guide/language-detection.html -->

Point an update request at that chain with update.chain, and the detected language lands in language_s on every indexed document:

curl "http://localhost:8983/solr/books/update?update.chain=langid&commit=true" \
  -H 'Content-Type: application/json' \
  -d '[ { "id": "42", "title": "Le Petit Prince", "body": "Il etait une fois..." } ]'

curl -G "http://localhost:8983/solr/books/select" --data-urlencode "q=language_s:fr"

langid can go a step further and drive the field-per-language strategy directly: with langid.map=true, langid.map.fl naming the fields to rewrite, and langid.map.pattern / langid.map.replace giving a rename rule, the processor can rewrite title to title_fr in place based on the detected language, so the document lands straight in the language-specific field defined above without any client-side branching. See Language Detection for the full parameter reference, including langid.langsField for multivalued detection and langid.whitelist to constrain the candidate languages.

Where this fits

Once text is broken into the right per-language terms, Query parsers and Query basics & parameters cover searching across one or several of these fields, and Relevance & scoring covers weighting them against each other. For the non-language-specific half of the analysis chain — tokenizers, character filters, and general-purpose token filters — see Text analysis: analyzers, tokenizers & token filters, and for how a field’s type and analyzer are declared in the first place, see Schema & fields and Field types.