Simpson's Paradox in a Product Comparison
Two onboarding tutorials were live for a month. Completion counts:
| Segment | Tutorial A | Tutorial B |
|---|---|---|
| New users | 630 completed / 3,000 shown | 96 completed / 400 shown |
| Returning users | 240 completed / 500 shown | 1,750 completed / 3,600 shown |
- Compute the completion rate for each tutorial within each segment, and overall. What do you observe?
- Explain mechanically why the pooled comparison disagrees with the segment comparisons. What is the confounder?
- Which tutorial would you recommend, and what would you say to a stakeholder who insists "the overall number is what matters"? What would you check in the data before finalising the recommendation?
1. Rates
| Segment | A | B |
|---|---|---|
| New | 630/3000 = 21.0% | 96/400 = 24.0% |
| Returning | 240/500 = 48.0% | 1750/3600 = 48.6% |
| Overall | 870/3500 = 24.9% | 1846/4000 = 46.2% |
Tutorial B is slightly better in both segments (24.0 vs 21.0, 48.6 vs 48.0), and dramatically better overall (46.2 vs 24.9). Here the direction happens to agree, but the magnitude of the pooled gap (21 percentage points) is wildly misleading compared with the within-segment gaps (3 points and 0.6 points). Change the numbers a little and the direction flips — that is Simpson's paradox in its full form; this is its milder and more common cousin, where the pooled effect size is a mix artefact.
2. Mechanism and confounder
The pooled rate for each tutorial is a weighted average of its segment rates, weighted by that tutorial's own segment mix:
- A: 0.857 \times 21.0\% + 0.143 \times 48.0\% \approx 24.9\% (86% new users)
- B: 0.10 \times 24.0\% + 0.90 \times 48.6\% \approx 46.2\% (90% returning users)
Returning users complete tutorials at roughly double the rate of new users regardless of tutorial. B was shown overwhelmingly to returning users; A to new users. The pooled comparison is therefore mostly measuring "which tutorial got the easier audience", not "which tutorial is better". User type is the confounder: it influences both assignment (which tutorial a user saw) and outcome (completion).
3. Recommendation and the conversation
Recommend B, but on the strength of the within-segment evidence — a modest edge among new users, near parity among returning — not the pooled 21-point gap. To the stakeholder: "The overall number mixes two very different populations in different proportions. If we rolled B out to everyone, we would not see completion jump from 25% to 46%; we would see roughly a 1–3 point improvement, because the user mix would be the same for both. The pooled figure answers 'who got more returning users?', which we already know."
Before finalising: (a) confirm why assignment was unbalanced — if tutorial B was deliberately shown to returning users, there may be further hidden confounders (time of month, platform); (b) check the new-user difference is not noise (400 users on B is a small sample; a 3-point gap at n = 400 is roughly one standard error — the Hypothesis Testing subject formalises this); (c) look for other segmenting variables (device, region) that might reverse the within-segment picture again; and (d) propose a properly randomised test so the segment mix is balanced by design (the A/B Testing subject).
Share this question