Intermediate
Open
Pro
Campaign A Converts Better Overall, but B Wins in Every Segment: Who Is Better?
Part of the Data Science Foundations path →
Part of the Advanced Data Science & Experimentation path →
Marketing compares two ad campaigns, each shown to 10,000 visitors.
| Campaign | Audience | Visitors | Conversions |
|---|---|---|---|
| A | Returning | 8,000 | 320 |
| A | New | 2,000 | 20 |
| B | Returning | 2,000 | 90 |
| B | New | 8,000 | 120 |
The dashboard shows overall conversion A = 3.4 percent versus B = 2.1 percent, and the head of marketing wants to cut B.
- Compute segment-level and overall conversion for both campaigns.
- Explain what is happening and compute a mix-standardized comparison.
- Can you conclude B is the better campaign? What additional information or design would you need, and what is the difference from an A/B test?
- Give two other places in a product analytics workflow where this paradox bites.
Share this question