Stating and Re-deciding a Trade-off
During a news feed design you said: "I'll precompute each user's feed on write (fan-out on write)."
- Restate that decision using the four-part trade-off template (choice, alternative, deciding requirement, accepted cost/mitigation), referencing numbers from a Twitter-like estimate (150 M DAU, 300 M posts/day, ~100:1 read:write, average 200 followers).
- The interviewer changes a requirement: "Assume this is a professional network where posts are rare (0.05/user/day) and most reads come from a handful of hugely followed publishers." Re-decide, using the same template.
- Name two failure modes of your original design and how you would detect and handle each.
1. Original decision in template form
"I'd choose fan-out on write over fan-out on read because reads outnumber writes ~100:1 (350k timeline reads/s vs 3.5k posts/s) and we want feed reads under ~100 ms, which a precomputed per-user list in cache delivers in a single lookup. The cost is write amplification: 3.5k posts/s × 200 followers ≈ 700k timeline-cache writes/s average, and a user with 10 M followers would generate 10 M writes for one post. We mitigate by processing fan-out asynchronously through a queue and by treating accounts above a follower threshold as 'celebrities' whose posts are merged in at read time (a hybrid)."
2. Re-decision under the new requirement
"With posts at 0.05/user/day, writes fall to ~7.5 M/day ≈ 90/s, and most reads target a few publishers with enormous audiences. Now I'd choose fan-out on read (or fan-out on write only for the long tail) over full fan-out on write, because pushing each publisher post into millions of per-user lists is pure amplification for content that everyone reads anyway. Instead, cache each publisher's recent posts once and merge the followed publishers' lists at read time. The cost is more work per read (merge k sorted lists), acceptable because k is small and the publisher lists are hot in cache; we mitigate tail latency by capping k and precomputing merged pages for the most common combinations."
3. Failure modes of the original design
- Fan-out queue backlog (a burst of posts or slow workers): feeds go stale for minutes. Detect via queue depth and consumer lag alarms; handle by autoscaling workers, prioritising recent posts, and showing a "new posts" indicator that triggers a read-time merge as fallback.
- Timeline cache node loss / cold cache: reads for those users miss and fall through to reconstructing feeds from the database, a thundering-herd risk. Detect via cache hit-ratio drop and DB QPS spike; handle with cache replication, request coalescing on rebuild, and rate-limiting rebuilds per user.
Share this question