Conversion Rose in the Test but Revenue Did Not

Your three-arm test ran for two weeks. The uplift arm's conversion rate is 2.1 points higher than the holdout (p = 0.01), but revenue per eligible user is not significantly different from the holdout, and profit per eligible user is slightly negative.

The pre-registered primary metric was profit per eligible user (margin minus discount minus contact cost). Guardrails were unsubscribes per contact and margin per order.

1. List at least four distinct explanations, ordered by how you would check them, and name the data each needs.
2. How would the ghost-treatment logs let you separate "the policy picked the wrong people" from "the policy picked the right people but the offer is unprofitable"?
3. What would you tell the stakeholder who says "conversion went up, so ship it"?