Find Mislabeled Examples in an 8,000-Item Training Set

A classifier trained on 8,000 labeled messages performs worse than expected, and you suspect a share of the labels are wrong. Describe how you would find likely errors. Use these measurements from a synthetic experiment with ten classes: with 20 percent of labels flipped uniformly (true flip rate 0.199) confident learning flagged 2,153 items with precision 0.518 and recall 0.702, and estimated the noise rate at 0.373; with 20 percent class-conditional flips (always to the confusable partner class) it flagged 2,442 items with precision 0.391 and recall 0.596.

1. Describe the procedure, including why probabilities must be out-of-fold.
2. Compute how many true errors each review queue contains, and the review cost at USD 1.20 per item.
3. Explain why the estimated noise rate is much larger than the injected one and why class-conditional noise is harder.
4. Say what you do with the flagged items, and what results to expect.