Advanced
Open
Pro
Detect and Handle Interference in a Social-Feed Experiment
A social app tests a one-tap "reshare with comment" feature, randomized 50/50 by user. After two weeks, treated users post 8 percent more than control. But product analytics shows that control users' posting is also up 3 percent versus their own pre-period, and the effect is larger among treated users with many treated friends. The PM wants to ship because "+8 percent is significant".
- What does this pattern suggest and why does the +8 percent not answer "what happens if we ship to everyone"?
- List three analyses that diagnose the interference, and what each would show.
- Propose a design that estimates the global effect, including the cost.
- A neighbour regression gives a direct effect of 1.0 and a spillover coefficient of 0.6. State the implied global effect and the bias of the original A/B.
Share this question