Monitoring Effect Drift and Funding Exploration

Six months after launch, the targeting dashboard looks fine: the conversion rate of contacted users has sat near 25 percent for four cycles. Finance, however, says promo spend is up and the program's profit "feels" lower. The model is frozen and a 5 percent persistent ghost-scored holdout has been kept.

The holdout analysis for the selected users shows:

(Plus-minus values are standard errors.)

1. Why was the dashboard blind to this, and what is the metric that would have caught it?
2. Propose an alert rule with a threshold and say what the first alert should have been on this data.
3. What is the likely mechanism, and how would you test it?
4. After retraining you want to keep the model fresh. A product manager objects to a 50/50 randomized slice because it "wastes coupons". Quantify the cost and propose cheaper alternatives.