Rate Limiting
"Design a rate limiter" is one of the most common short system-design prompts, and rate limiting shows up as a sub-component in nearly every larger design — an API gateway, a public API, a login endpoint, a notification pipeline. Interviewers like it because it is small enough to finish in 30 minutes yet touches algorithms, data structures, distributed state, failure handling and API ergonomics all at once.
The bar is higher than "use a token bucket". You are expected to explain how each algorithm behaves at a window boundary and under a burst, pick one and defend it with numbers, decide where the counters live when there are 200 gateway instances, say what the client sees when it is limited, and — the part most candidates skip — say what happens when the rate limiter's own datastore is unavailable.
This subject covers all of that. It stays inside the limiter itself: load balancing in front of the gateway is the Load Balancing subject; circuit breakers, retries and load shedding as reliability tools are in Reliability & Observability Patterns (this subject only draws the boundary between them and rate limiting); Redis internals and cache placement are in Caching Strategies.
Why Rate Limit, and Where
Four distinct motivations — say which one you are solving, because it changes the design:
| Goal | Typical key | Typical limit shape |
|---|---|---|
| Abuse / security — brute-force logins, scraping, credential stuffing | IP, account, device fingerprint | Low limits on sensitive endpoints, e.g. 5 login attempts / 15 min |
| Cost control — a paid downstream (LLM tokens, SMS, maps API) | Tenant / API key | Quotas per day/month plus a per-second cap |
| Fairness — one noisy tenant must not starve others | Tenant / user | Per-tenant share of capacity, tiered by plan |
| Protecting downstream — the DB can do 5k writes/s, full stop | Global / per endpoint | A hard aggregate ceiling, often combined with load shedding |
Where to enforce (usually more than one layer):
client SDK ──► CDN / edge ──► API gateway ──► service ──► downstream (DB, 3rd party)
(self-throttle, (per-IP, (per-user / (per-tenant (outbound limiter:
backoff+jitter) DDoS-ish per-key, the business rules, respect *their*
volumetric) main policy) per-endpoint) limits)
- Client-side limits are a courtesy, not a control — never trust them.
- Edge / CDN handles cheap, coarse limits (per-IP) before traffic reaches you.
- Gateway is the canonical place for per-user / per-API-key policy: one place to configure, one place to emit headers.
- Service-level limits capture rules the gateway cannot know ("max 3 concurrent exports per tenant").
- Outbound limiters protect you from tripping a third party's limits.