Choosing the Right Rung on the Model Ladder
Three teams are each starting an ML system. For each, pick the first model you would ship and the next rung you would try, and justify both in terms of data volume, features, and latency budget.
- A B2B SaaS company with 40,000 customers wants to predict which accounts will churn next quarter. They have 3 years of monthly usage snapshots and a sales team that will act on a list of 200 accounts per month.
- A marketplace with 50 M listings wants "similar items" on every product page, served in under 30 ms, with 10 M pageviews per day.
- A payments company wants to block fraudulent card transactions in real time; 0.2 % of 100 M monthly transactions are fraudulent, and chargebacks arrive 30–90 days later.
1. B2B churn (small tabular data, offline batch)
Data: 40 k accounts × 36 months ≈ 1.4 M rows at most, tabular. First model: a heuristic baseline (usage dropped > 50 % in 60 days) to get a number, then logistic regression on engineered features — it is interpretable, which sales needs, and there is too little data for deep models. Next rung: a GBM (LightGBM/XGBoost) with the same features plus SHAP explanations. Latency is irrelevant (monthly batch); the metric is precision@200 because sales capacity is 200 accounts. Deep learning is not justified.
2. Similar items at scale (retrieval, tight latency)
Scoring 50 M items per pageview is impossible in 30 ms; this is a retrieval problem. First model: co-visitation / item-item collaborative-filtering counts precomputed offline into a top-50 list per item, served from a key–value store — near-zero latency, strong baseline. Next rung: a two-tower embedding model (content + behaviour) with an ANN index, which handles cold-start items and long-tail queries; embeddings are precomputed for all 50 M items and refreshed daily. A light re-ranker over the ~200 ANN candidates can come later.
3. Real-time fraud (imbalanced, delayed labels)
Data is large (10⁸/month) but positives are ~200 k/month, and labels are delayed so the training window lags. First model: rules + logistic regression with velocity features (transactions per card in last 10 min, amount vs. historical mean) — millisecond latency and easy to audit. Next rung: GBM on hundreds of tabular and streaming aggregate features, evaluated by PR-AUC and precision at the manual review capacity, with a time-based split that respects label maturity. Deep sequence models over transaction histories are a third rung once the feature pipeline is stable. In all three cases the interviewer is scoring the justification, not the final architecture.
Share this question