Baselines Before the Deep Model
A team is building a "recommended for you" module for a mid-sized e-commerce site (2 M monthly users, 300 k products). Their first plan is to train a two-tower neural network with user- and item-history encoders and ship it if offline recall@50 exceeds 0.30.
- Which baselines should they build first, and what does each one tell them?
- The two-tower model reaches recall@50 = 0.31. Popularity ranking reaches 0.27. How would you use these numbers in a go/no-go discussion?
- Name one operational reason, unrelated to accuracy, why the baseline should still be maintained after the neural model ships.
1. Baselines and what they reveal
- Random / constant: recall@50 for random ranking over 300 k products is about 50 / 300 000 ≈ 0.02 %. If the evaluation code reports anything else for random, the metric or the join is broken.
- Popularity (global, or per category / country): the strongest cheap baseline for recommendation, especially for new users. It establishes what "no personalisation" is worth.
- Heuristic: "same category as last purchase", "recently viewed" — product-intuitive rules that also serve as cold-start logic.
- Logistic regression / small GBDT on hand-built features (user category affinities, item popularity, price band): the first learned model; interpretable coefficients tell the team which features carry signal.
Together the ladder calibrates the deep model's number and debugs the pipeline on models that train in seconds.
2. Reading 0.31 vs 0.27
The two-tower model adds +0.04 absolute recall over popularity, a ~15 % relative gain. That is a real but modest lift, and it is an offline proxy on logged data with selection bias. Questions to raise: is the gain concentrated in returning users (where personalisation is possible) or spread evenly? What is the marginal cost — training infrastructure, embedding refresh, serving latency? Would a GBDT on features close most of the gap at a fraction of the cost? A go decision is reasonable if the lift holds on the returning-user slice and the serving cost is acceptable; then confirm online with an A/B, because a +15 % offline recall gain does not guarantee a +15 % engagement gain.
3. Operational reason to keep the baseline
The baseline is the fallback: when the model server times out, the embedding index is stale, or a new user has no history, the system degrades to popularity rather than an empty module. It also serves as the control arm in future A/B tests and as a sanity check when the pipeline changes.
Share this question