Duplicate records rarely appear in one predictable form: a customer may change email addresses, a vendor may use an abbreviated legal name, or two product feeds may describe the same item differently. This guide presents a repeatable entity resolution workflow for generating candidates, scoring matches, routing uncertain cases to review, merging records safely, and monitoring the process over time.
Overview
Entity resolution is the process of deciding which records refer to the same real-world entity. In a deduplication process, the goal is not simply to find similar strings. It is to combine evidence from several fields, account for missing or inconsistent data, and make a controlled decision about whether records should remain separate, be reviewed, or be merged.
A reliable workflow has five stages:
- Normalize: Convert values into comparable forms without discarding useful distinctions.
- Generate candidates: Retrieve plausible pairs or groups instead of comparing every record with every other record.
- Score: Combine field-level similarity and business rules into an entity matching confidence score.
- Review: Send ambiguous or high-impact cases to a person or a carefully designed secondary process.
- Merge and monitor: Apply reversible merge rules, preserve history, and measure results after each run.
Separating candidate generation from final matching is important. A fast approximate string matching method can identify possible pairs, while a more cautious scoring model decides whether the evidence is strong enough to act. For algorithm choices, see the comparison of Levenshtein, Jaro-Winkler, and trigram similarity.
What to track
Input quality and normalization
Track how often important fields are present, how values are formatted, and how frequently normalization changes them. Useful fields might include name, email, phone number, postal address, account identifier, tax identifier, SKU, or manufacturer part number. The right fields depend on the entity type and the consequences of a false match.
Document transformations such as case folding, whitespace cleanup, punctuation handling, tokenization, Unicode normalization, and country-specific address rules. Keep the original values alongside normalized values. This makes decisions explainable and allows reviewers to see whether a match came from real evidence or an overly aggressive transformation. A normalization pipeline for fuzzy matching should be treated as a versioned component, not an invisible preprocessing step.
Candidate generation
Measure the number of candidates produced per source record, the proportion of records with no candidates, and the share of candidates generated by each blocking key. Blocking keys can include email domain and local part, phone suffix, postal code, normalized surname, product brand, or a stable external identifier. Use more than one key when a single field is incomplete or unreliable.
Track candidate recall using a labeled sample: how often does the candidate stage include the correct match when one exists? An efficient system that fails to generate the true pair cannot recover it with a better scoring model.
Scoring and decisions
Record field-level similarities as well as the final score. Levenshtein distance may be useful for short identifiers, Jaro-Winkler can help with names that share prefixes, and trigram similarity can handle token and substring variation. Phonetic matching can add evidence for names that sound alike, but it should rarely be the sole reason to merge records.
Use three decision bands rather than one universal threshold:
- Automatic match: Evidence is strong, the fields agree, and the cost of a false positive is acceptable.
- Human review: Evidence is mixed, fields conflict, or the record has meaningful business impact.
- Separate: Evidence is weak or a contradiction makes a match implausible.
Track the volume and percentage in each band, plus the eventual outcome of reviewed cases. Thresholds should be validated against representative examples, not selected only because they produce a convenient number of matches. The guide to fuzzy matching thresholds provides a useful framework for that work.
Review, merge, and rollback activity
For the review queue, track age, backlog, reviewer agreement, override reasons, and time to resolution. Capture why a reviewer rejected a candidate: shared household address, common name, reused phone number, brand variation, or insufficient evidence. These reasons reveal where the model needs better features or where the workflow needs clearer rules.
For merges, preserve source record IDs, the selected surviving record, field-level provenance, the matching model version, reviewer or job identifier, timestamp, and merge reason. A merge should be reversible. Prefer a merge ledger or relationship table over destructive deletion so that an incorrect decision can be undone without reconstructing lost data.
Cadence and checkpoints
Run operational checks at the same cadence as the data refresh that feeds the process. A monthly or quarterly review is often a practical starting point, but high-volume systems may need daily monitoring for failures and a separate monthly quality review.
Per run
- Confirm that input files, streams, or tables arrived with expected schemas.
- Check record counts, null rates, encoding errors, and normalization failures.
- Compare candidate and decision volumes with recent runs.
- Verify that automatic merges stayed within configured limits.
- Sample accepted, rejected, and reviewed matches.
Monthly or quarterly
- Refresh a labeled evaluation set with new edge cases and confirmed outcomes.
- Review precision and recall separately for important entity segments.
- Inspect changes by source system, country, language, product category, or customer type.
- Audit a sample of merged records and verify that downstream systems received the intended identity.
- Review unresolved clusters and decide whether new blocking keys or field rules are needed.
Keep a change log for threshold updates, feature changes, normalization rules, source-system migrations, and schema changes. Without this history, a shift in match rates is difficult to interpret.
How to interpret changes
A rise in automatic matches is not automatically an improvement. It may indicate better source data, but it can also result from an overly permissive threshold or a blocking key that creates many weak candidates. Review precision and false-positive examples before treating increased coverage as success.
A growing review queue can mean the model is uncertain, the queue lacks enough reviewer capacity, or a new source has introduced unfamiliar formats. Separate these possibilities by examining queue age, source-level distributions, and reviewer outcomes. If reviewers often select “insufficient evidence,” improve candidate context and data collection rather than simply lowering the threshold.
A drop in candidate volume may reflect a healthy reduction in duplicates, but it may also signal a broken ingestion job, changed formatting, or an index that was not refreshed. Compare upstream record counts and field completeness before changing the matching logic.
Interpret results by segment. A threshold that works for structured product identifiers may be unsafe for common personal names. Address matching may require country-specific normalization, while multilingual name matching may benefit from transliteration and alternate-script fields. Aggregate metrics can conceal these differences.
When to revisit
Revisit the workflow at least monthly or quarterly, and immediately after any event that changes the data or the consequences of a false match. Triggers include a new source system, a field format change, an acquisition or catalog expansion, a change in language or geography, a new identifier policy, a spike in customer complaints, or an unexpected shift in match and review volumes.
Use each review to answer four practical questions:
- Are the records being compared still representative of the entities the system is meant to resolve?
- Are automatic decisions accurate enough for their downstream use?
- Can reviewers understand and reproduce the evidence behind each decision?
- Can every merge be rolled back without losing the original records or their provenance?
Start with a small labeled sample, inspect errors by category, and change one part of the workflow at a time. Re-run the evaluation set before promoting a new threshold or matching rule. For a broader implementation plan, use the entity resolution pipeline guide for names, addresses, and customer records, and keep a standing review calendar so duplicate detection does not become a one-time cleanup project.
