Plan a Long-Term Holdout Program for a Growth Team

A growth team ships roughly 40 winning experiments per half-year. The sum of the individual measured lifts in 28-day retention is +9 percent relative, yet the company's overall retention curve looks nearly flat. The VP asks you to set up a universal holdout: keep 2 percent of users out of all of the team's launches for six months. The product has 10 million active users and 28-day retention of about 30 percent, and the app has social features.

1. Why might the individual wins not add up? Give at least four reasons.
2. How large an effect can a 2 percent holdout detect?
3. Describe two ways the holdout can be contaminated and how to prevent each.
4. Specify the program: assignment, duration and refresh, what is measured, who is eligible, and how results are used.