Allocating a Latency Budget Across the Funnel
A search product has an end-to-end p99 budget of 250 ms for the results page. Network, page assembly and non-ML services consume 120 ms, leaving 130 ms for the ML path. The corpus is 200 M documents.
- Sketch a candidate-generation → ranking → re-ranking funnel and assign a rough latency budget and candidate count to each stage.
- A colleague proposes a cross-encoder model that jointly encodes (query, document) at ~4 ms per pair on the available hardware. Where, if anywhere, can it fit?
- Name two things you would monitor for this funnel that are specific to ML rather than generic service health.
1. Funnel with budgets
Query understanding (spell, intent, embedding) ~10 ms
Candidate generation ~30 ms 200 M → ~1,000
- inverted index / BM25 top-500
- ANN over document embeddings top-500 (parallel)
Ranking (GBM or small NN, rich features) ~50 ms 1,000 → 100
Re-ranking (diversity, freshness, business) ~10 ms 100 → 10–20
Slack for tail latency / retries ~30 ms
Candidate generation must be sub-linear in corpus size (index lookups, ANN), and its two sources run in parallel. The ranker sees only ~1,000 items, so a per-item cost of ~50 µs (batched) is fine.
2. Where the cross-encoder fits
Not over 1,000 candidates: 1{,}000 \times 4 ms = 4 s sequentially, and even with 32-way batching on accelerators it blows the budget. It can fit as a second-stage re-ranker over the top 20–30 from the first ranker: 25 \times 4 = 100 ms sequential, or ~10–20 ms batched on a GPU — inside budget if the earlier stages are trimmed. Alternative placements: run it offline to produce distillation labels for the cheaper ranker, or apply it only for high-value / low-QPS query segments.
3. ML-specific monitoring
- Recall of the retrieval stage: fraction of clicked/purchased documents in the logs that were present in the ~1,000 candidates. If it drops, the ranker cannot recover the loss no matter how good it is.
- Ranker score / feature distribution drift by query segment, and the calibration of P(\text{click}) against realised CTR — a shift may indicate a broken feature pipeline or genuine drift.
- Also reasonable: position-bias-corrected CTR@k over time, and the share of queries falling back to the heuristic path when a model times out.
Share this question