How do we debug full-text relevance failures caused by analyzers rather than scoring math?

Bad relevance is often an analysis bug wearing a scoring costume. Start with one failing query and a tiny document set, inspect the tokens produced at index time and query time, and only then open scoring tools. Elasticsearch’s analysis docs make the core point: full-text search depends on tokenization and normalization at both indexing and search time, so if the wrong terms are produced, BM25 can still score “correctly” on the wrong input (Elasticsearch analysis documentation).

What failure pattern are you actually seeing?

Classify the symptom before changing anything:

• Zero hits: usually token mismatch, wrong field, or incompatible index-time and search-time analyzers.
• Wrong top hit: often a field-design problem, such as broad ngram or synonym-heavy fields competing with exact fields.
• Phrase mismatch: usually positions changed by stopword removal, stemming, or synonym expansion.
• Overbroad recall: often aggressive stemming, folding, or synonym rules.
• Unstable ranking across similar queries: often cross-field query design or analyzer differences between fields.

Lucene’s analysis package is a useful mental model here: char filters modify text before tokenization, tokenizers split text, and token filters then change tokens through stemming, synonym injection, or case folding (Lucene analysis package summary). If the symptom points to one of those stages, ranking math is not your first suspect.

How do you isolate analysis before scoring?

Build a minimal reproduction: one bad query, three to five representative documents, and the exact field mapping involved. Reproduce the configured index and search analyzers on representative text and compare their tokens; this does not recover historical tokens from an indexed document. We recommend inspecting analyzed terms first and using _explain second. Otherwise you risk “fixing” a broken analyzer with boosts.

Worked example:

Documents:

• Doc A: personal computer repair
• Doc B: pc repair
• Doc C: computer desk

Query: personal computer

If your search analyzer uses synonym_graph, Elasticsearch documents that it creates a graph token stream and is designed for search analyzers, especially for multi-word synonyms (Elasticsearch synonym graph token filter). Depending on the synonym rules, the graph can contain both a phrase path and a pc path. If Doc B outranks Doc A, check the expansion, query type, and field design before changing boosts; that ranking alone does not establish a scoring defect.

Which analyzer failures should you check first?

Check these in order:

1. Stopwords removing phrase structure. Phrase queries can drift when connective terms disappear.
2. Stemming collapsing distinctions you cared about, such as product names versus general nouns.
3. ASCII folding or normalization hiding meaningful differences in identifiers or names.
4. Synonym ordering and scope. Elasticsearch explicitly notes token filter order matters, and some multi-token filters can cause issues with synonyms.
5. Ngram or prefix fields overpowering exact-match fields in multi-match queries.

A reliable design is multi-fields with separate intent: exact, stemmed, prefix, and synonym-aware. Then query them deliberately instead of throwing every field into one blended query.

When should you use _explain?

After you trust the analyzed terms. Elasticsearch _explain tells you why a specific document matches or does not match and returns a score explanation for that query-document pair (Elasticsearch explain API). Use it to confirm that scoring is sensible given the produced terms, not to decide whether the analyzer is sane.

Decision checklist:

• Reproduce one bad query on a tiny corpus.
• Inspect index-time tokens.
• Inspect search-time tokens.
• Verify whether synonym, stem, and stopword behavior is intentional.
• Separate exact and broad-recall fields.
• Run _explain on the expected and unexpected top documents.
• Version the analyzer and reindex if index-time analysis changed.
• Add regression tests for both token output and ranking.

Follow-up Q&A?

Should I tune BM25 first if scores look close?

No. Close scores on the wrong tokens are still wrong. Prove the analyzer is producing the intended term stream before touching similarity settings.

Do synonym fixes always require reindexing?

Not always. Synonym-file changes can be reloaded when the filter has updateable: true and is used only in search analyzers, as the synonym graph settings describe. Changing index-time analysis requires reindexing existing documents. Test this boundary before rollout.

Next step: pick one failing query today, capture its field mapping and analyzed token streams, and do not change boosts until those tokens match your intended retrieval model.

Reviewed: 2026-09-05