Intermediate
Open
Pro
Twelve Metrics, Three Stars
An experiment dashboard shows 12 metrics; three come out with p < 0.05: 0.004, 0.021 and 0.047. The remaining nine are 0.09, 0.15, 0.22, 0.31, 0.38, 0.44, 0.60, 0.71, 0.88.
- If all 12 nulls were true, what is the probability of seeing at least one p < 0.05? What does that imply about the three stars?
- Apply Bonferroni at \alpha = 0.05 and Benjamini–Hochberg at q = 0.05. Which metrics survive each?
- Explain what FWER and FDR each control, and when you would prefer each in a product-analytics setting.
Share this question