A Survey Was the Wrong Tool for This Question
Your PM ran a survey asking existing users, "On a scale of 1-5, how easy is it to set up a new project in our tool?" The average score came back 4.2 out of 5 — reassuringly high. Two months later, support tickets show new signups are abandoning setup at a high rate, and a quick usability test with 5 new users shows three of them getting stuck on the exact same step.
- Using the NN/g qualitative/quantitative × attitudinal/behavioral framework, explain specifically why the survey result and the usability test result don't actually contradict each other, even though they look like they do.
- What was wrong with using this survey, sent to existing users, to answer a question about setup difficulty in the first place?
- If you only had one week and had to choose one method to re-investigate the setup problem properly, which would you pick and why?
1. Why the two results don't actually contradict each other
They're measuring different things for a different population. The survey is quantitative and attitudinal — a stated opinion, at scale, from people who already successfully set up a project (they had to have gotten through setup to be an active, surveyable existing user). The usability test is qualitative and behavioral — observed actual struggle, from new users encountering the step for the first time. A 4.2 average from people who already cleared the hurdle tells you almost nothing about whether new users clear it; it's survivorship bias wearing the disguise of a reassuring number. There is no real contradiction, because the two studies never measured the same population doing the same thing.
2. What was wrong with the survey population and framing
Sending an attitudinal, self-report survey to existing users structurally excludes anyone who found setup too hard and left before becoming an "existing user" to survey — the exact population the question was actually about never appears in the sample. Even among respondents, "how easy is it" as a 1-5 recall question asks people to rate a memory of a task they may have done months ago and have since become expert at, not their actual real-time experience during first setup. The question needed behavioral, in-the-moment data from new users, not attitudinal recall from people who've already survived the step being asked about.
3. Best single method with one week
A moderated usability test with 5-8 new users attempting real setup from scratch — qualitative and behavioral, evaluative of the current flow. It directly observes where and why people get stuck (which the support tickets and the 5-user test already hinted at), it's achievable within a week with a small recruited sample, and per Nielsen's sample-size reasoning, 5-8 participants should surface most of the concrete usability problems in the step, which is a discovery question, not a measurement one. A new survey wouldn't fix the core issue (wrong population, wrong data type); a full A/B test isn't achievable credibly in a week without enough signup volume to reach significance.
Share this question