Critique This Online Test Plan for a Targeting Policy

A teammate proposes the following plan to validate the new uplift policy.

Send the coupon to the 15 percent of customers the uplift model ranks highest. Compare their conversion over the next 7 days with the conversion of all customers we did not contact. If it is higher and significant, ship. Two arms: new policy and "everyone else". We need 3 percent of the base for a few days.

Known facts from your offline work: treat-all loses EUR 0.40 per user; the incumbent response policy loses EUR 0.58 per user; the uplift policy gains EUR 0.55 per user; profit SD per user is EUR 11.84; the randomized log has propensity 0.05.

1. Identify at least five flaws.
2. Propose a redesign with arms, the ghost-treatment logging, the primary and guardrail metrics and the analysis populations.
3. Compute the required sample per arm for uplift versus holdout on the untriggered profit metric, with Bonferroni correction across three comparisons (alpha 0.05/3, power 0.8). Then explain why the triggered analysis needs far less traffic.