Measuring fuzzy search relevance requires more than asking whether the first result looks good. This guide explains MRR, MAP, NDCG, precision@K, recall@K, success rate, and zero-result rate, then applies them to a shared example dataset with practical judgment rules and spreadsheet-friendly calculations. Use the framework to compare ranking changes without allowing one aggregate score to hide failures in typo correction, identifier lookup, autocomplete, or exploratory search.
Overview
Offline evaluation compares a search configuration against a labeled set of queries and expected results. It is especially useful for fuzzy search, where a change to typo tolerance, normalization, Levenshtein distance, trigram similarity, or semantic retrieval can improve recall while damaging ranking precision.
Begin with a representative query set. Include misspellings, exact names, product identifiers, natural-language searches, multilingual variants where relevant, and queries that should return no result. For each query, record the results shown by the system and assign a relevance judgment to each result.
A simple three-level judgment scale works well for many teams:
- 2 — Highly relevant: directly satisfies the query or is the intended result.
- 1 — Partially relevant: useful or related, but not the best answer.
- 0 — Not relevant: unrelated, misleading, or too weak to show.
Keep the judgment policy separate from the ranking implementation. Evaluators should judge whether a result helps the user, not whether it has a high text-similarity score. A normalized but incorrect entity can look similar while still being a poor result. For background on building labels, see How to Build a Gold-Standard Dataset for Entity Resolution.
How to compare options
Compare search configurations using the same query sample, candidate pool, result cutoff, normalization pipeline, and judgment set. Run the baseline and the proposed fuzzy-search change against identical inputs. Otherwise, a metric difference may reflect data or sampling changes rather than ranking quality.
| Metric | What it measures | Best use | Important limitation |
|---|---|---|---|
| MRR | Position of the first relevant result | Known-item lookup and navigational queries | Ignores later relevant results |
| MAP | Average precision across relevant results | Queries with several acceptable results | Usually treats relevance as binary |
| NDCG@K | Graded relevance with a position discount | Mixed-quality results and exploratory search | Needs consistent relevance grades |
| Precision@K | Share of the first K results judged relevant | Visible result quality and compact result pages | Does not measure missed relevant items |
| Recall@K | Share of known relevant items found in the first K | Entity retrieval, catalog search, and broad discovery | Requires a reasonably complete relevant set |
| Success rate | Share of queries with at least one acceptable result in K | Fast monitoring of task completion | Hides result ordering and depth |
| Zero-result rate | Share of queries returning no results | Detecting coverage and parsing failures | A zero result may be correct for an impossible query |
Do not select a single universal winner. A fuzzy-search change that raises recall@10 may lower MRR for exact identifiers. Report metrics by query type and review the failure cases behind each regression. A weighted score can support release decisions, but it should supplement—not replace—the individual metrics.
Feature-by-feature breakdown
MRR and MAP
Mean reciprocal rank is calculated as the average of 1 / rank of the first relevant result. If the first relevant results appear at positions 1, 2, and 1, MRR is (1 + 0.5 + 1) / 3 = 0.833. MRR is clear and useful for an exact product code or company name, where the user generally wants one answer.
Mean average precision gives credit for multiple relevant results and rewards placing them early. With binary judgments, average precision is calculated at each relevant position and averaged. For relevant results at positions 1 and 3 in a three-result list, AP is (1/1 + 2/3) / 2 = 0.833. MAP is the mean AP across queries. MAP is less suitable when “partially relevant” is materially different from “fully relevant.”
NDCG and graded relevance
Normalized discounted cumulative gain is useful when results can be fully, partially, or not relevant. A common form is DCG@K = Σ((2^grade - 1) / log2(rank + 1)). Divide the result’s DCG by the ideal DCG for the same query to obtain NDCG@K. The logarithmic discount means a highly relevant result near the top matters more than the same result lower down.
For example, suppose the judged grades for a query are [2, 1, 0]. The ideal ordering is also [2, 1, 0], so NDCG@3 is 1.00. If the grades are [0, 2, 1], the result is weaker because the best answer is delayed; NDCG@3 is approximately 0.67 using the formula above.
Precision@K, recall@K, and success rate
Precision@K is relevant results in the first K / K. Recall@K is relevant results in the first K / all known relevant results. Precision describes what the user sees; recall describes how much of the expected answer set was recovered.
Success rate uses a deliberately simple rule, such as “at least one result with grade 1 or 2 in the first five.” Define that rule before comparing systems. Otherwise, evaluators may silently change the meaning of success during a release.
Zero-result rate
Track zero-result rate independently from relevance metrics. A query with no matching catalog item may legitimately return nothing, while a misspelled identifier that should have been recovered indicates a coverage problem. Segment zero-result rate by query type, language, and input source so a normalization or indexing failure is not hidden in an overall average.
Shared example and spreadsheet template
Assume three answer-bearing queries are evaluated at K=3, while a fourth query has no eligible catalog match:
| Query type | Grades at positions 1–3 | Known relevant items | First relevant rank |
|---|---|---|---|
| Typo correction: “iphon 15 case” | 2, 1, 0 | 3 | 1 |
| Exact identifier: “SKU-4821” | 0, 2, 1 | 2 | 2 |
| Exploratory: “wireless charging” | 1, 2, 1 | 4 | 1 |
| No-match query | no results | 0 expected | none |
For the first three queries, the example produces MRR of 0.833. Binary MAP is approximately 0.806: the average precisions are 0.833, 0.583, and 1.000. Precision@3 averages 0.778, while recall@3 averages approximately 0.806: the individual recalls are 2/3, 2/2, and 3/4. Using graded judgments, NDCG@3 is approximately 0.851 across the three answer-bearing queries: 1.000, 0.670, and 0.882.
A spreadsheet can use one row per query with columns such as query_id, query_type, result_1_grade, result_2_grade, result_3_grade, known_relevant, relevant_in_k, first_relevant_rank, and zero_result. Useful formulas include:
precision_at_k = relevant_in_k / krecall_at_k = relevant_in_k / known_relevant, when known_relevant is greater than zeroreciprocal_rank = IF(first_relevant_rank=0, 0, 1/first_relevant_rank)success = IF(relevant_in_k>=1, 1, 0)zero_result = IF(number_of_results=0, 1, 0)
Average each column over the appropriate query set. Exclude queries with no expected answer from recall and NDCG denominators when their ideal gain is zero, but report their zero-result behavior separately. For query-level diagnostics, keep the full ranked list and evaluator notes rather than storing only aggregate scores.
Best fit by scenario
- Typo correction: prioritize success rate, MRR, and NDCG@5. Check that fuzzy matching recovers the intended result without promoting loosely similar items.
- Exact identifier lookup: use MRR, precision@1, and exact-match failure rate. Fuzzy fallback should not outrank a valid exact identifier.
- Autocomplete: use precision@K and success rate at the short result cutoff visible in the interface. Evaluate prefixes separately from complete queries.
- Exploratory search: use NDCG, recall@K, and query-level review. Several partially relevant results may be useful, so MRR alone is too narrow.
- Entity matching and deduplication: report precision and recall for match, non-match, and review queues. The cost of a false merge is usually different from the cost of a missed candidate; preserve that distinction in the release decision.
For implementation context, pair relevance evaluation with the retrieval design described in Designing Typo-Tolerant Search Without Sacrificing Relevance and the ranking considerations in PostgreSQL Fuzzy Search with pg_trgm.
When to revisit
Re-run the evaluation whenever the index, analyzer, normalization pipeline, fuzzy threshold, candidate-generation method, ranking model, or result cutoff changes. Also revisit the judgment set when the catalog, entity population, languages, or user query mix changes. A stale gold set can make a new system appear better simply because it no longer represents current search tasks.
For each relevance release, sample fresh queries, preserve a stable regression set, compare baseline and candidate metrics by query type, and inspect the largest individual gains and losses. Record the evaluator version, K value, grade definitions, and calculation method. If MRR rises but recall falls, inspect missed alternatives; if zero-result rate falls but NDCG declines, check whether typo tolerance is admitting noisy candidates. This routine turns fuzzy-search evaluation into an ongoing engineering practice rather than a one-time benchmark.