Designing a Dialect Fairness Audit That Detects the Real Problem

Your cascade (cheap text classifier, then a stronger tier on a routed fraction, then humans) auto-removes 16,130 posts per day. A community group says posts written in a particular dialect ("group B", about 12 percent of posts, roughly 600,000 per day) are removed too often. Your tier-1 audit at the flag threshold shows a false-positive-rate ratio of 1.21 between group B and the reference group A, which a colleague calls "within tolerance".

Simulation results at the enforcement point: false auto-removals per benign post are 0.00014 for group A and 0.00045 for group B (62 and 27 false removals in the test sample, with 437,799 and 59,626 benign posts). Precision of auto-removal is 0.956 for A and 0.864 for B.

1. Why can both statements be true, and what do you audit instead?
2. How many labels does an audit of the false-removal rate need compared with an audit of precision among removed items?
3. How would you find the root cause?
4. What are the remedies and their trade-offs?