Caching Strategies

Almost every system design interview ends up with a box labelled "cache" between the application servers and the database. The box is easy to draw and hard to defend. The questions that follow — what happens when it is cold? how does it learn that a price changed? what if one key gets a million reads a second? can a user ever see a stale value after their own write? — are where candidates separate. Caching is also the single highest-leverage performance tool in production systems: a 95% hit ratio in front of a database is a 20× reduction in database load, which is often the difference between one primary and a sharded fleet.

This subject treats the cache as a component with a lifecycle: where it sits, how data gets in (read and write patterns), how data gets out (TTL, invalidation, eviction), how it fails (stampedes, hot keys, stale reads), and how big it needs to be. It ends with a worked design of the caching layer for a product page serving 50,000 requests per second, and the traps interviewers set around it.

Distributing keys across cache nodes with minimal reshuffling is its own topic — see the Consistent Hashing subject; here we only point to where it plugs in.

Why Cache: Latency and Load

Two independent reasons, and you should name both.

Latency. A cache is faster storage placed closer to the reader. Reference numbers: a main-memory read is ~100 ns; a random SSD read is ~100 µs; a round trip to a cache node in the same datacenter is ~0.5 ms; a typical indexed database query, including network, parsing, buffer-pool lookups and result serialisation, is ~1–10 ms; a call to a downstream service or third-party API can be 50–500 ms. An in-process cache turns milliseconds into nanoseconds; a distributed cache turns a 5 ms query into a 0.5 ms lookup and, more importantly, gives a predictable latency that does not degrade when the database is under pressure.

Load. The database or origin has finite capacity — on the order of 10^4 simple reads per second for a single relational primary. If a cache serves a fraction h of requests (the hit ratio), the origin sees only (1-h) of the traffic:

At 50,000 QPS, h = 0.9 leaves 5,000 QPS on the origin; h = 0.99 leaves 500. Note the non-linearity: going from 90% to 99% halves nothing on the cache side but cuts origin load by 10×. This is why interviewers care about the hit ratio, not the cache's existence.

Effective latency follows the same shape:

With L_{\text{cache}} = 0.5 ms and L_{\text{origin}} = 10 ms, h = 0.9 gives 1.5 ms average — but the p99 is still ~10 ms, because 1% of requests is a lot more than 1% of the misses. Caches improve the average far more than the tail unless the hit ratio is very high.