Advanced
Open
Pro
Evaluate an anomaly detector when you have no labels
Part of the ML System Design Interview path →
Part of the Advanced Data Science & Experimentation path →
You have built an Isolation Forest on 5 features of card transactions (20,000 a day) and are about to switch it on. There are no labels, the true anomaly rate is believed to be about 0.1 percent, and the reviewing team can handle 10 alerts a day. A colleague proposes "we will inject fake anomalies, measure recall, and report ROC-AUC".
Write the evaluation plan. Say what each method can and cannot tell you, how many reviews you need, and which metric you will report.
Share this question