How to Build a Gold-Standard Dataset for Entity Resolution
entity resolutionrecord linkagedata labelingprecision and recallevaluation datasetsdata quality

How to Build a Gold-Standard Dataset for Entity Resolution

FFuzzy Direct Editorial Team
2026-11-13
8 min read

Build a reliable entity-matching benchmark with sampling, labeling rules, agreement checks, precision and recall, and version control.

A reliable entity-resolution system starts with reliable labeled data. This guide shows how to build a gold-standard matching dataset: define the task, sample useful candidate pairs, write repeatable labeling rules, handle uncertainty, measure precision and recall, record annotator agreement, and maintain a versioned benchmark as your data and matching workflow change.

Overview

Entity resolution, also called record linkage or duplicate detection, attempts to decide whether two records refer to the same real-world entity. The entities might be customers, companies, suppliers, addresses, products, or accounts. A system may use exact identifiers, normalized fields, fuzzy matching, semantic similarity, or a combination of methods. Whatever the implementation, evaluation depends on a trusted set of examples with known outcomes.

A gold-standard dataset is not simply a random export with a match column. It is a documented evaluation set in which each pair has a clear labeling rule, a traceable review history, and enough coverage to expose both false matches and missed matches. It should represent the difficult cases your production system will encounter, not only the easy pairs that confirm an algorithm is working.

Keep the evaluation task separate from the training or tuning data when possible. If the same records are repeatedly used to choose thresholds, select features, and report results, the benchmark can become overly familiar. A practical setup usually includes a development set for experimentation, a review or tuning set for threshold decisions, and a locked test set for final comparisons.

Before labeling, document the entity definition. For example, two company records may match when they identify the same legal entity, but not when they represent a parent and subsidiary. Two address records may match at the building level while differing at the unit level, or they may need to match at the complete deliverable-address level. Your definition determines the labels, the acceptable evidence, and the cost of an error.

For field preparation, document every transformation applied before comparison. This may include case folding, whitespace cleanup, punctuation handling, accent treatment, transliteration, abbreviation expansion, or phone-number formatting. A field-safe normalization pipeline can improve consistency, but it can also remove distinctions that matter. Preserve the original values alongside normalized values so reviewers can inspect what changed.

Checklist by scenario

1. Define the labeling task

  • Write one sentence describing what counts as a match.
  • Specify the entity level: person, household, location, legal company, account, product, or another unit.
  • Decide whether labels are binary, such as match and non-match, or include an explicit uncertain class.
  • List the fields reviewers may use and identify fields that must not influence the decision.
  • Define how historical changes, shared addresses, subsidiaries, household members, and reused identifiers should be treated.

For company matching, legal names, domains, registration identifiers, and addresses may carry different evidentiary weight. For name matching, spelling variation and transliteration may be expected, while a shared phone number may be ambiguous. For address matching, unit numbers can determine whether two records refer to the same destination. The company record matching workflow and address matching guidance illustrate why the entity definition must come first.

2. Sample candidate pairs deliberately

Do not build the dataset only from pairs already identified by your matcher. That approach can hide missed matches and make the evaluation look better than the real workflow. Draw pairs from several sources:

  • Confirmed matches: records connected by a trusted identifier, an existing merge decision, or a carefully reviewed relationship.
  • Likely non-matches: records from the same blocking group that appear similar but represent different entities.
  • Hard negatives: pairs sharing a common name, address, domain, phone number, or product description but requiring different outcomes.
  • Model-boundary cases: pairs whose similarity scores sit near the proposed decision threshold.
  • Missed-match candidates: plausible pairs found through an alternate blocking rule, a manual search, or a second matching method.
  • Coverage samples: examples separated by source system, language, geography, time period, field completeness, and formatting style.

Record how each pair entered the dataset. Sampling provenance helps you detect selection bias and recreate the benchmark later. Stratified sampling is often more useful than a purely random sample when rare but costly errors matter.

3. Write operational labeling rules

Rules should tell a reviewer what evidence is sufficient, what evidence is insufficient, and what to do when fields conflict. Avoid instructions such as use your best judgment without examples. Instead, create a decision table with positive, negative, and uncertain examples.

For every scenario, answer questions such as:

  • Does an exact, trusted identifier override conflicting names?
  • When is a shared address evidence of a match, and when is it only a blocking signal?
  • Are abbreviations, transliterations, nicknames, or reordered tokens accepted?
  • How should missing values be treated? Missing should generally mean unknown, not disagreement.
  • What evidence is required before merging records with different identifiers?
  • Should a parent company and subsidiary receive separate labels?

Include a short rationale field for difficult decisions. The rationale is useful for adjudication, future rule changes, and auditing a model that produces an unexpected result.

4. Review, adjudicate, and measure agreement

Have two independent reviewers label at least a meaningful subset of pairs, and preferably all high-risk or uncertain pairs. Hide previous labels during independent review. Then compare disagreements before an adjudicator assigns the final reference label.

Track raw agreement as well as a chance-adjusted agreement measure when it suits the label design. The exact statistic is less important than documenting the method and interpreting it alongside examples. Low agreement usually signals unclear entity definitions, insufficient examples, ambiguous fields, or reviewer training problems. It should not be solved by silently forcing reviewers into consensus.

5. Measure the matching system

For a binary match decision, calculate:

  • Precision: the proportion of predicted matches that are true matches.
  • Recall: the proportion of true matches that the system finds.
  • F-score: a combined measure that can summarize the balance between precision and recall when that balance is appropriate.

Also report the confusion matrix: true positives, false positives, true negatives, and false negatives. Review results by segment rather than relying only on one overall score. A system can perform well overall while failing on a particular language, source, address format, or entity type. For threshold-based fuzzy matching, compare several thresholds and record the operational consequence of each choice. The algorithm and threshold-testing guide provides a useful framework for that comparison.

6. Use a practical dataset schema

A reusable schema might include these fields:

  • pair_id: stable identifier for the comparison.
  • source_a and source_b: origin of each record.
  • record_id_a and record_id_b: source identifiers.
  • entity_type: the entity definition being evaluated.
  • raw_fields_a and raw_fields_b: values shown to reviewers, subject to privacy controls.
  • normalized_fields_a and normalized_fields_b: comparison-ready values.
  • sampling_reason: confirmed match, hard negative, boundary case, and so on.
  • label_reviewer_1, label_reviewer_2, and final_label.
  • confidence and rationale: reviewer context.
  • adjudication_status and adjudication_notes.
  • dataset_version, labeling_guideline_version, and created_at.
  • model_score, predicted_label, and model_version, added only for evaluation runs.

What to double-check

Privacy and access: Use the least sensitive fields necessary for labeling. Restrict access, record who viewed or changed labels, and consider masked or tokenized identifiers where they preserve the review task. Set retention and deletion procedures appropriate to your organization and the data involved. A gold-standard dataset should not become an uncontrolled copy of production data.

Label leakage: Check whether a field reveals the answer because it was created by a previous matching or merge process. Also check whether model scores, cluster IDs, or post-merge identifiers are visible to reviewers when they should not be.

Pair direction and duplicates: Decide whether the pair is treated as unordered. Prevent the same logical pair from appearing as both A-to-B and B-to-A unless that repetition is intentional. If records belong to clusters, document whether labels are pairwise or cluster-level.

Segment coverage: Create a small coverage report showing counts by source, time period, language, geography, missingness, and entity type. Add targeted examples for segments where the system has higher risk or where the business impact of a false match is substantial.

Evaluation integrity: Keep the final test set locked. Store the code or query used to generate metrics, the normalization configuration, the candidate-generation rules, and the model version. If you change any of these, record a new evaluation run rather than overwriting the old result.

Common mistakes

  • Using only easy positives: Confirmed exact-ID matches do not test fuzzy matching or duplicate detection quality.
  • Treating similarity as truth: A high Levenshtein, Jaro-Winkler, or trigram score is evidence for review, not an automatic label. Similarity can be misleading for common names, short strings, and shared addresses.
  • Forcing uncertain pairs into match or non-match: Preserve uncertainty and adjudicate it. If the business process requires a decision, record both the decision and the confidence.
  • Ignoring negatives near the threshold: These examples are essential for choosing a safe operating point.
  • Changing rules without versioning: A label means little if the definition of a match changes invisibly.
  • Reporting one aggregate score: Segment-level errors often matter more than the average.
  • Evaluating only candidate pairs: If candidate generation misses a true pair, a scorer cannot recover it. Measure blocking and retrieval separately from final classification.
  • Using the benchmark as a permanent fixture: A dataset that reflects last year’s sources may not represent today’s records.

When to revisit

Re-run the same benchmark before seasonal planning cycles and whenever workflows or tools change. More specifically, revisit the dataset when you add a source system, entity type, language, geography, identifier, normalization rule, candidate-generation method, matching algorithm, or review policy. Also revisit it after a large data-format change, a merge-policy change, or a production incident involving a false match or missed match.

Use a two-part update process. First, preserve the existing version so historical model comparisons remain valid. Second, add a refresh set containing recent records and newly observed edge cases. Label the refresh set using the current rules, compare performance on both versions, and investigate any divergence. This separates genuine model improvement from a change in the evaluation population.

Before closing each review cycle, use this action checklist:

  1. Confirm the entity definition and labeling guideline version.
  2. Inspect sampling coverage and add hard positives, hard negatives, and missed-match candidates.
  3. Run independent labeling and adjudicate disagreements.
  4. Check privacy, access, leakage, duplicate pairs, and schema completeness.
  5. Evaluate precision, recall, confusion counts, agreement, and segment-level results.
  6. Lock the test version, archive the run configuration, and document known limitations.
  7. Schedule the next review trigger rather than waiting for the benchmark to become outdated.

A gold-standard dataset is an operating asset, not a one-time labeling project. Treat it as versioned test infrastructure for entity resolution, and it will make threshold decisions, fuzzy matching changes, and deduplication releases easier to compare and safer to deploy.

Related Topics

#entity resolution#record linkage#data labeling#precision and recall#evaluation datasets#data quality
F

Fuzzy Direct Editorial Team

Editorial team

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.