Compute a Fellegi-Sunter Match Weight by Hand
A Fellegi-Sunter model has a prior match share \lambda = 0.01 among candidate pairs and the following parameters, given for this exercise (they are not measured values):
| Field | Level | m | u |
|---|---|---|---|
| last name | exact | 0.95 | 0.01 |
| first name | nickname or initial | 0.30 | 0.04 |
| date of birth | exact | 0.80 | 0.001 |
| zip | different | 0.20 | 0.98 |
| phone | missing | - | - |
For one candidate pair: last name agrees exactly, first names are "william" and "bill", dates of birth are equal, zips differ, one phone is missing.
- Compute the total match weight in bits and the posterior match probability.
- What contribution does the missing phone make, and why?
- The surname is the rarer "ostrowski" and 0.68 percent of records have that surname. Recompute the surname contribution with a term-frequency adjustment and say what changes.
- Name one reason the posterior from (1) is likely too confident.
1. Total weight. Each term is \log_2(m/u), plus the prior term \log_2(\lambda/(1-\lambda)):
| Term | Computation | Bits |
|---|---|---|
| prior | \log_2(0.01/0.99) | -6.629 |
| last name exact | \log_2(0.95/0.01) = \log_2 95 | +6.570 |
| first name nickname | \log_2(0.30/0.04) = \log_2 7.5 | +2.907 |
| dob exact | \log_2(0.80/0.001) = \log_2 800 | +9.644 |
| zip different | \log_2(0.20/0.98) | -2.293 |
| phone missing | no evidence | 0 |
The sum is W = 10.198 bits (computed). Odds are 2^{10.198} \approx 1175 to 1, so the posterior is 1175/1176 = 0.99915. Note the disagreeing zip reduces the weight by 2.3 bits but does not rescue the non-match hypothesis, because the date of birth and surname together carry about 16 bits.
2. The missing phone contributes zero. Treat "missing" as its own level whose m and u are equal (or omit the term): a missing value is not evidence either way. Scoring it as disagreement would punish every pair from sparse sources, which are exactly the records most in need of linking. (In the lesson's data 52 percent of phones were missing.)
3. Term-frequency adjustment. With u = 0.0068 for this surname, the contribution is \log_2(0.95/0.0068) = 7.126 bits instead of 6.570, so the total rises to 10.755 bits. A surname rarer than average earns a bonus (0.68 percent is less than the global 1 percent), and a very common one would be penalized; the adjustment matters most at the extremes: in the lesson's data an exact surname match was worth 3.62 bits for "smith" (6.8 percent of records) and 10.44 bits for a surname held by 0.06 percent of records, against 6.56 bits with one global u.
4. Overconfidence. Fellegi-Sunter assumes fields are independent given match status; zip, street and phone are correlated, so their weights are counted more than once. A posterior of 0.999 is better read as a ranking score and calibrated (for example with isotonic regression on labelled pairs) before it is used as a probability. Another valid reason: \lambda and u were estimated on a different blocked population than the one being scored.
Share this question