Practice — Entity Resolution & Record Linkage (6 questions)

A retailer has 20 million customer records from a web shop, a loyalty programme and a call centre. Fields: first name, last name, date of birth (often missing), street address, zip code, phone and email (both often missing). Names and addresses are typed by hand, so typos, nicknames and abbreviations are common. You have about 5,000 labelled true duplicate pairs from past manual merges.

1. Compute the number of pairs a naive comparison needs and explain why blocking is needed.
2. Propose a multi-pass blocking plan: name the keys, say what each one is good and bad at, and say how you would add a fuzzy pass.
3. State the metrics you would use to choose between plans and the numbers you would set as a budget.
4. Name two ways a block can become dangerously large and what you do about them.

A Fellegi-Sunter model has a prior match share \lambda = 0.01 among candidate pairs and the following parameters, given for this exercise (they are not measured values):

For one candidate pair: last name agrees exactly, first names are "william" and "bill", dates of birth are equal, zips differ, one phone is missing.

1. Compute the total match weight in bits and the posterior match probability.
2. What contribution does the missing phone make, and why?
3. The surname is the rarer "ostrowski" and 0.68 percent of records have that surname. Recompute the surname contribution with a term-frequency adjustment and say what changes.
4. Name one reason the posterior from (1) is likely too confident.