Intermediate
Open
Pro
Auditing a Usability Study Report for Hidden Bias
A colleague shares a study report: "We tested the new dashboard with 5 power users from our top-tier enterprise accounts, all of whom use the product multiple times a day. All 5 completed every task with zero errors. We're confident this design is ready to ship to our full user base, including new signups."
- Identify the specific bias in this study and explain concretely why it undermines the "ready to ship to our full user base" conclusion.
- The report treats "5 for 5, zero errors" as strong evidence. Is the sample size itself a problem here, separate from who was recruited? Explain why or why not, referencing what a 5-user qualitative round is actually good for.
- Propose a specific follow-up study design that would address the gap you identified, including who you'd recruit and what you'd change about the task design, if anything.
Share this question