Investigating Training–Serving Skew

A fraud model shows offline recall of 0.78 at the chosen threshold but only 0.55 in shadow mode in production. Offline features are computed in Spark SQL from the warehouse; online features are computed by a Java service reading a Redis store populated by a stream job. The feature list includes txn_count_10min, amount_over_user_mean_30d, merchant_country, and hour_of_day.

1. List four concrete ways these features could differ between training and serving.
2. Design a skew test that would catch these differences before the model is promoted, and say what you would alert on.
3. Which architectural change removes most of this class of bugs, and what does it cost?