Entity Resolution & Record Linkage
Deduplicating and matching records at scale — blocking, probabilistic matching, clustering and evaluation
Learn to design and defend an entity resolution system end to end, with numbers you can reproduce: problem variants (deduplication, cross-source linkage, identity resolution, company and product matching) and why 'same entity' is a business definition; the pipeline (standardization, blocking, comparison, classification, clustering, survivorship) and blocking quality measured as pair completeness, reduction ratio and pair quality, with the arithmetic that 110,538 records already mean 6.1 billion candidate pairs and 1 million records mean 500 billion; similarity features (Jaro-Winkler, Levenshtein and Soundex hand-checked on martha/marhta, nickname tables, date and numeric tolerances, missing-value levels); models (hand rules, Fellegi-Sunter m and u probabilities fitted by EM, with a measured failure when u is taken from random pairs, a per-field match-weight breakdown for one pair that adds up to 7.220 bits and a posterior of 0.9933, and a gradient-boosted pair classifier); clustering (the chaining problem with the a-times-b false-pair arithmetic, transitive closure versus pivot versus average-linkage clustering measured on B-cubed); evaluation (pairwise versus B-cubed and CEAF, recall estimation by clerical sampling and by capture-recapture, and why capture-recapture overestimated recall here); scale (n(n-1)/2, MinHash and LSH banding tuned against an empirical S-curve, Spark patterns from memory); privacy and governance (PII, Bloom-filter privacy-preserving linkage as a toy, auditability, review queues); an assumption-based LLM-matcher cost and benefit estimate; and a worked example on a synthetic 110,538-record customer table with fixed seeds, honest about which results are weak or surprising.
Practice questions (6)
-
View →
Design Blocking for 20 Million Customer Records
Intermediate · Free -
View →
Compute a Fellegi-Sunter Match Weight by Hand
Intermediate · Free -
View →
Choose a Clustering Strategy After Closure Merges Families
Advanced -
View →
Estimate Recall When You Have No Complete Ground Truth
Advanced -
View →
Is an LLM Matcher Worth It? A Back-of-Envelope
Advanced