Why the 5-User Rule Doesn't Apply to an A/B Test
A team runs an A/B test with 5 users, reasoning that "5 users finds 85% of problems, per Nielsen's rule, so this should be enough."
What's wrong with applying that rule to an A/B test's required sample size?
The correct answer is "A/B tests measure effect size with statistical power, not discover binary problems."
This subject is explicit that the 5-user rule doesn't transfer to quantitative research: qualitative usability testing is answering a discovery question ("does this problem exist or not"), which has fast diminishing returns per participant. An A/B test is answering a measurement question ("how big is this effect, reliably"), where the required sample size is set by the baseline conversion rate, the minimum effect size worth detecting, and tolerance for false positives/negatives — often thousands of users, not five.
The distractors misapply or invert the rule: the 5-user rule specifically does not apply equally to quantitative research — that's the entire point being tested; A/B tests need more participants than five to reach statistical validity, not fewer; and the 85% figure isn't scoped only to heuristic evaluation — it's Nielsen's qualitative usability-testing curve, which is exactly why it doesn't generalize to a statistically powered quantitative test either.
Share this question