Practice — Usability Testing and Evaluation (5 questions)

You're validating a redesigned multi-step signup flow. Launch is in 10 days. The redesign combines what used to be three separate screens (account details, plan selection, payment) into a single scrolling page, and your team has no prior data on how users react to that kind of consolidation. You have budget for either 5 moderated sessions this week, or 20 unmoderated sessions that could run overnight.

1. Which approach would you choose, and what specifically about this scenario drives that choice?
2. What is the single biggest risk of choosing the other approach instead, given this specific redesign?
3. Is there a way to use both within the 10-day window? Describe it.

You're given a two-week window and a fixed budget to evaluate a redesigned settings page before it ships. You can afford either: (a) a heuristic evaluation by 4 evaluators plus 5 moderated usability sessions, spread across the two weeks, or (b) 15 moderated usability sessions and no heuristic evaluation at all, using the full budget on user sessions.

1. Which option would you recommend, and what specifically makes it the stronger use of a fixed budget?
2. What kind of problem would option (b)'s extra 10 sessions likely find that option (a)'s 5 sessions would miss — and is that gap worth the trade-off?
3. What kind of problem would option (a)'s heuristic evaluation catch that no amount of additional user sessions would catch faster or cheaper?

A usability study on a new onboarding flow returns these results across 8 participants: completion rate 50%, average time-on-task 4 minutes (well above the 90-second target), average 3.1 errors per session, and a SUS score of 79 from a post-task questionnaire.

1. Explain how a design can score well above the 68 SUS benchmark average while also having a 50% completion rate. What does this combination most likely mean?
2. Which of the two results — the SUS score or the completion rate — should carry more weight in a go/no-go launch decision, and why?
3. What additional data would you want before finalizing a recommendation, and why would think-aloud data specifically help here?

A product manager comes to you: "Our free-trial-to-paid conversion rate is 8%, well below the 15% benchmark for similar products. Let's A/B test three different pricing page layouts to see which one converts best — we have plenty of trial-signup traffic to make it statistically valid quickly."

1. What's the risk in going straight to a 3-variant A/B test here, even though there's enough traffic to run one validly?
2. What would you propose doing first, and what specific output would you want from it before designing any A/B test variants?
3. Once that first step is done, how would you decide whether A/B testing is even still the right next move, versus another round of qualitative work?

A colleague shares a study report: "We tested the new dashboard with 5 power users from our top-tier enterprise accounts, all of whom use the product multiple times a day. All 5 completed every task with zero errors. We're confident this design is ready to ship to our full user base, including new signups."

1. Identify the specific bias in this study and explain concretely why it undermines the "ready to ship to our full user base" conclusion.
2. The report treats "5 for 5, zero errors" as strong evidence. Is the sample size itself a problem here, separate from who was recruited? Explain why or why not, referencing what a 5-user qualitative round is actually good for.
3. Propose a specific follow-up study design that would address the gap you identified, including who you'd recruit and what you'd change about the task design, if anything.