How to Measure Fuzzy Search Relevance: MRR, NDCG, Recall@K, and Judgment Guidelines
search relevanceNDCGMRRprecision and recallsearch evaluation

How to Measure Fuzzy Search Relevance: MRR, NDCG, Recall@K, and Judgment Guidelines

FFuzzy Direct Editorial Team
2026-11-20
6 min read

Compare fuzzy search changes with MRR, NDCG, MAP, precision, recall, success rate, and practical judgment guidelines.

Measuring fuzzy search relevance requires more than asking whether the first result looks good. This guide explains MRR, MAP, NDCG, precision@K, recall@K, success rate, and zero-result rate, then applies them to a shared example dataset with practical judgment rules and spreadsheet-friendly calculations. Use the framework to compare ranking changes without allowing one aggregate score to hide failures in typo correction, identifier lookup, autocomplete, or exploratory search.

Overview

Offline evaluation compares a search configuration against a labeled set of queries and expected results. It is especially useful for fuzzy search, where a change to typo tolerance, normalization, Levenshtein distance, trigram similarity, or semantic retrieval can improve recall while damaging ranking precision.

Begin with a representative query set. Include misspellings, exact names, product identifiers, natural-language searches, multilingual variants where relevant, and queries that should return no result. For each query, record the results shown by the system and assign a relevance judgment to each result.

A simple three-level judgment scale works well for many teams:

  • 2 — Highly relevant: directly satisfies the query or is the intended result.
  • 1 — Partially relevant: useful or related, but not the best answer.
  • 0 — Not relevant: unrelated, misleading, or too weak to show.

Keep the judgment policy separate from the ranking implementation. Evaluators should judge whether a result helps the user, not whether it has a high text-similarity score. A normalized but incorrect entity can look similar while still being a poor result. For background on building labels, see How to Build a Gold-Standard Dataset for Entity Resolution.

How to compare options

Compare search configurations using the same query sample, candidate pool, result cutoff, normalization pipeline, and judgment set. Run the baseline and the proposed fuzzy-search change against identical inputs. Otherwise, a metric difference may reflect data or sampling changes rather than ranking quality.

MetricWhat it measuresBest useImportant limitation
MRRPosition of the first relevant resultKnown-item lookup and navigational queriesIgnores later relevant results
MAPAverage precision across relevant resultsQueries with several acceptable resultsUsually treats relevance as binary
NDCG@KGraded relevance with a position discountMixed-quality results and exploratory searchNeeds consistent relevance grades
Precision@KShare of the first K results judged relevantVisible result quality and compact result pagesDoes not measure missed relevant items
Recall@KShare of known relevant items found in the first KEntity retrieval, catalog search, and broad discoveryRequires a reasonably complete relevant set
Success rateShare of queries with at least one acceptable result in KFast monitoring of task completionHides result ordering and depth
Zero-result rateShare of queries returning no resultsDetecting coverage and parsing failuresA zero result may be correct for an impossible query

Do not select a single universal winner. A fuzzy-search change that raises recall@10 may lower MRR for exact identifiers. Report metrics by query type and review the failure cases behind each regression. A weighted score can support release decisions, but it should supplement—not replace—the individual metrics.

Feature-by-feature breakdown

MRR and MAP

Mean reciprocal rank is calculated as the average of 1 / rank of the first relevant result. If the first relevant results appear at positions 1, 2, and 1, MRR is (1 + 0.5 + 1) / 3 = 0.833. MRR is clear and useful for an exact product code or company name, where the user generally wants one answer.

Mean average precision gives credit for multiple relevant results and rewards placing them early. With binary judgments, average precision is calculated at each relevant position and averaged. For relevant results at positions 1 and 3 in a three-result list, AP is (1/1 + 2/3) / 2 = 0.833. MAP is the mean AP across queries. MAP is less suitable when “partially relevant” is materially different from “fully relevant.”

NDCG and graded relevance

Normalized discounted cumulative gain is useful when results can be fully, partially, or not relevant. A common form is DCG@K = Σ((2^grade - 1) / log2(rank + 1)). Divide the result’s DCG by the ideal DCG for the same query to obtain NDCG@K. The logarithmic discount means a highly relevant result near the top matters more than the same result lower down.

For example, suppose the judged grades for a query are [2, 1, 0]. The ideal ordering is also [2, 1, 0], so NDCG@3 is 1.00. If the grades are [0, 2, 1], the result is weaker because the best answer is delayed; NDCG@3 is approximately 0.67 using the formula above.

Precision@K, recall@K, and success rate

Precision@K is relevant results in the first K / K. Recall@K is relevant results in the first K / all known relevant results. Precision describes what the user sees; recall describes how much of the expected answer set was recovered.

Success rate uses a deliberately simple rule, such as “at least one result with grade 1 or 2 in the first five.” Define that rule before comparing systems. Otherwise, evaluators may silently change the meaning of success during a release.

Zero-result rate

Track zero-result rate independently from relevance metrics. A query with no matching catalog item may legitimately return nothing, while a misspelled identifier that should have been recovered indicates a coverage problem. Segment zero-result rate by query type, language, and input source so a normalization or indexing failure is not hidden in an overall average.

Shared example and spreadsheet template

Assume three answer-bearing queries are evaluated at K=3, while a fourth query has no eligible catalog match:

Query typeGrades at positions 1–3Known relevant itemsFirst relevant rank
Typo correction: “iphon 15 case”2, 1, 031
Exact identifier: “SKU-4821”0, 2, 122
Exploratory: “wireless charging”1, 2, 141
No-match queryno results0 expectednone

For the first three queries, the example produces MRR of 0.833. Binary MAP is approximately 0.806: the average precisions are 0.833, 0.583, and 1.000. Precision@3 averages 0.778, while recall@3 averages approximately 0.806: the individual recalls are 2/3, 2/2, and 3/4. Using graded judgments, NDCG@3 is approximately 0.851 across the three answer-bearing queries: 1.000, 0.670, and 0.882.

A spreadsheet can use one row per query with columns such as query_id, query_type, result_1_grade, result_2_grade, result_3_grade, known_relevant, relevant_in_k, first_relevant_rank, and zero_result. Useful formulas include:

  • precision_at_k = relevant_in_k / k
  • recall_at_k = relevant_in_k / known_relevant, when known_relevant is greater than zero
  • reciprocal_rank = IF(first_relevant_rank=0, 0, 1/first_relevant_rank)
  • success = IF(relevant_in_k>=1, 1, 0)
  • zero_result = IF(number_of_results=0, 1, 0)

Average each column over the appropriate query set. Exclude queries with no expected answer from recall and NDCG denominators when their ideal gain is zero, but report their zero-result behavior separately. For query-level diagnostics, keep the full ranked list and evaluator notes rather than storing only aggregate scores.

Best fit by scenario

  • Typo correction: prioritize success rate, MRR, and NDCG@5. Check that fuzzy matching recovers the intended result without promoting loosely similar items.
  • Exact identifier lookup: use MRR, precision@1, and exact-match failure rate. Fuzzy fallback should not outrank a valid exact identifier.
  • Autocomplete: use precision@K and success rate at the short result cutoff visible in the interface. Evaluate prefixes separately from complete queries.
  • Exploratory search: use NDCG, recall@K, and query-level review. Several partially relevant results may be useful, so MRR alone is too narrow.
  • Entity matching and deduplication: report precision and recall for match, non-match, and review queues. The cost of a false merge is usually different from the cost of a missed candidate; preserve that distinction in the release decision.

For implementation context, pair relevance evaluation with the retrieval design described in Designing Typo-Tolerant Search Without Sacrificing Relevance and the ranking considerations in PostgreSQL Fuzzy Search with pg_trgm.

When to revisit

Re-run the evaluation whenever the index, analyzer, normalization pipeline, fuzzy threshold, candidate-generation method, ranking model, or result cutoff changes. Also revisit the judgment set when the catalog, entity population, languages, or user query mix changes. A stale gold set can make a new system appear better simply because it no longer represents current search tasks.

For each relevance release, sample fresh queries, preserve a stable regression set, compare baseline and candidate metrics by query type, and inspect the largest individual gains and losses. Record the evaluator version, K value, grade definitions, and calculation method. If MRR rises but recall falls, inspect missed alternatives; if zero-result rate falls but NDCG declines, check whether typo tolerance is admitting noisy candidates. This routine turns fuzzy-search evaluation into an ongoing engineering practice rather than a one-time benchmark.

Related Topics

#search relevance#NDCG#MRR#precision and recall#search evaluation
F

Fuzzy Direct Editorial Team

Senior SEO Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.