Choose a Sequence Model for 10,000-Event User Histories
Your ranker must use each user's last 10,000 events (item, action, timestamp). Latency budget for the ranker is 40 ms for 1,000 candidates. One engineer wants a SASRec-style Transformer over all 10,000 events; another wants DIN-style target attention with an MLP scorer over all 10,000 events for each candidate; a third says "just average the embeddings". Embedding dimension is 64 and there are 500M users.
- Estimate the multiply-accumulates per request for (a) dot-product target attention over 10,000 events for 1,000 candidates, (b) one full self-attention layer over the 10,000 events, and (c) a DIN-style MLP scorer of about 20,600 MAC per pair. Which of them fits a 40 ms budget on one accelerator slice, and what would you do about the rest?
- Estimate the storage for the histories as IDs (8 bytes) and as fp16 vectors, for 500M users.
- Explain a retrieve-then-attend design (SIM or ETA style) and give the new cost if 200 events are retrieved.
- When would you choose a SASRec-style user encoder over target attention, and how does BERT4Rec differ at serving time?
- List three training or evaluation pitfalls specific to sequence models.
1. Cost. (a) Dot-product target attention: per candidate, scoring costs L\cdot d and the weighted sum another L\cdot d, so 1000\cdot10{,}000\cdot64\cdot2 = 1.28\times10^{9} MAC = 1.28 GMAC. (b) One self-attention layer over the history costs about 2L^2d = 2\cdot10^{8}\cdot64 = 12.8 GMAC (before projections and the feed-forward part), though it is done once per user, not per candidate, and can be cached between requests. (c) The MLP scorer: 1000\cdot10{,}000\cdot20{,}600 \approx 206 GMAC. On an accelerator sustaining a few tens of TMAC/s per slice, 1.28 GMAC is well within budget, 12.8 GMAC is feasible if cached, and 206 GMAC is not (the practical throughput of small irregular kernels is far below peak). So: (a) is fine, (b) must be amortized, (c) must be avoided or applied only to a retrieved subset.
2. Storage. IDs: 10{,}000\cdot8 = 80 KB per user, \times 5\times10^{8} = 40 TB. fp16 vectors of dimension 64: 10{,}000\cdot64\cdot2 = 1.28 MB per user, = 640 TB. So store IDs and look up embeddings from the (sharded) table when needed, not precomputed vectors.
3. Retrieve then attend. First stage: for each candidate, select the few hundred history events most related to it, by category match (hard search, SIM-style) or embedding similarity (soft search via an ANN or via locality-sensitive hashing in ETA-style designs). Second stage: ordinary target attention over those. With 200 retrieved events the cost is 1000\cdot200\cdot64\cdot2 = 25.6 MMAC, 50 times less than 1.28 GMAC, and the MLP scorer drops from 206 GMAC to about 4.1 GMAC. Keep a short recent window (for example 50 events) through a full Transformer, since recency matters regardless of the candidate. Caveat: the retrieval step can miss relevant events, so monitor it against exhaustive attention on a sample.
4. SASRec vs target attention vs BERT4Rec. A SASRec-style user encoder computes one user state with a causal mask (every position predicts the next item) and then scores any number of candidates by a dot product: good for retrieval and for cheap ranking, but the single vector cannot specialize to the candidate. Target attention makes the user representation depend on the candidate, which suits ranking with rich candidate features, at a cost proportional to candidates times history. BERT4Rec trains with masked items using both directions, but at serving the future does not exist, so a mask token is appended and the model must predict it: a train-inference mismatch and a heavier training procedure. Replication studies (from memory; verify) reported that its advantage over SASRec disappeared under equal careful training.
5. Pitfalls. (a) Random splits leak the future: in the lesson's experiment hit@10 was 0.249 on a random split and 0.168 on a time-ordered split, mostly because random splits hide items born after the cutoff. (b) Sampled metrics (ranking the positive among a few random negatives) can reorder models relative to full-catalogue metrics. (c) Timestamps and gaps: positional embeddings alone ignore that a week-long gap differs from a one-minute gap; features at training must be the as-of values serving will have. Also mention negatives: use sampled softmax with a logQ correction.
Share this question