Paths Subjects Questions Quizzes Pricing Search
Intermediate Open Pro

Twelve Metrics, Three Stars

An experiment dashboard shows 12 metrics; three come out with p < 0.05: 0.004, 0.021 and 0.047. The remaining nine are 0.09, 0.15, 0.22, 0.31, 0.38, 0.44, 0.60, 0.71, 0.88.

  1. If all 12 nulls were true, what is the probability of seeing at least one p < 0.05? What does that imply about the three stars?
  2. Apply Bonferroni at \alpha = 0.05 and Benjamini–Hochberg at q = 0.05. Which metrics survive each?
  3. Explain what FWER and FDR each control, and when you would prefer each in a product-analytics setting.

Share this question

← Back to Hypothesis Testing & Statistical Inference practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.