Building a Latency Budget from the Reference Numbers
A product page must render in under 200 ms at p99 (server side). The request path is: load balancer → API service → (a) product cache lookup, on miss a database read, → (b) a call to a pricing service in the same datacenter, → (c) a call to a recommendations service hosted in another region.
- Using the standard latency reference numbers, estimate the cost of each hop and decide whether the 200 ms budget is achievable.
- Propose two changes, justified with numbers, that would make the budget comfortable.
- If the pricing service is 99.9% available and the API service is 99.95% available, what availability can the product page promise if it fails whenever either fails?
1. Rough budget
- Load balancer + intra-DC hop to the API service: ~0.5–1 ms.
- Product cache lookup (Redis, same DC): ~0.5 ms round trip + ~100 ns memory access → ~1 ms. On a miss, an indexed database read is a few ms (SSD reads ~100 µs each, plus query overhead) → ~5–10 ms.
- Pricing service, same DC: ~0.5 ms network + processing → ~5 ms.
- Recommendations service, cross-region: ~150 ms round trip before the service does any work, and cross-region tail latency is worse than the mean.
Sum ≈ 1 + 10 + 5 + 150 ≈ 165 ms typical, but at p99 the cross-region call alone can exceed 200 ms. The budget is not safely achievable as drawn; the design's problem is one hop that costs 100× every other hop.
2. Two changes
- Move recommendations off the critical path or into the region. Either replicate the recommendations service (or a precomputed per-user recommendation list in a cache) into the serving region, turning ~150 ms into ~1–5 ms, or fetch recommendations asynchronously after the page renders. Either change removes ~150 ms.
- Call independent dependencies in parallel. Product data and pricing do not depend on each other; issuing them concurrently makes the wall-clock cost max(10, 5) rather than 10 + 5. This saves only ~5 ms here, but stops the budget scaling with the number of dependencies.
With both, p99 becomes ~15–25 ms with large margin.
3. Availability of serial dependencies
Availability multiplies for components that must all succeed: 0.999 × 0.9995 ≈ 0.9985 → 99.85%, i.e. roughly 13 hours of downtime per year, worse than either dependency alone. To promise 99.9% or better the page must tolerate pricing failures (show a cached price, or hide the price with a fallback) rather than fail closed.
Share this question