Reliability, Resilience & Observability Patterns
Every non-trivial system is built from parts that fail. Disks die, a dependency deploys a bad build, a network partition drops 2% of packets, a payment provider's p99 quietly climbs from 300 ms to 9 s. Reliability is the discipline of delivering the promised behaviour despite that. Resilience is the set of mechanisms that let a service degrade gracefully instead of collapsing when a dependency misbehaves. Observability is how you find out — from the outside — what the system is doing right now and why.
In a system design interview these three topics are what separates a "boxes and arrows" answer from a senior one. After you have drawn the load balancers, caches, and databases, the interviewer will ask: "What happens when the payment provider is slow? How would you know? What is your availability target and how would you hit it?" Candidates who can quantify availability, explain why naive retries make outages worse, and describe what they would alert on get hired for staff-level roles. This subject gives you that vocabulary and the reasoning behind it.
The rate-limiting mechanics themselves (token bucket, sliding window) are covered in the Rate Limiting subject; here we only reference rate limiting as a resilience tool. Similarly, replication and failover of the database tier are detailed in Database Replication and Sharding — this subject covers redundancy at the service and region level.
Availability Math
Availability is the fraction of time (or of requests) a service is working:
Engineers talk in "nines". Memorise this table — interviewers ask for it directly:
| Availability | Nines | Downtime / year | Downtime / month (30 d) | Downtime / day |
|---|---|---|---|---|
| 99% | two | 3.65 days | 7.2 hours | 14.4 min |
| 99.9% | three | 8.76 hours | 43.8 min | 1.44 min |
| 99.95% | — | 4.38 hours | 21.9 min | 43 s |
| 99.99% | four | 52.6 min | 4.38 min | 8.6 s |
| 99.999% | five | 5.26 min | 26.3 s | 0.86 s |
Note the practical implication: at four nines you have under an hour a year. That is less time than a human takes to notice, page, and fix a problem, so four nines requires automated failover — no manual step can be on the critical path.
Serial vs parallel components
Serial dependency (request needs A and B): availabilities multiply.
Parallel redundancy (request needs A or B — any one suffices): unavailabilities multiply.
Worked calculation
A request path is: load balancer (99.99%) → API service (99.9%) → database (99.95%) → payment provider (99.5%).
That is roughly 2.4 days of downtime per year, even though every individual component sounds respectable. The chain is only as strong as its weakest link, and the links compound. The lesson: more serial dependencies always lowers availability; every extra synchronous call in the request path is a tax.
Now put two independent payment providers in parallel with failover:
Overall availability becomes 0.9999 \times 0.999 \times 0.9995 \times 0.999975 \approx 99.84\% — cutting yearly downtime from ~58 hours to ~14 hours. Two nines of an unreliable part, doubled, gave more improvement than anything else on the path. That is the quantitative case for redundancy — with the caveat that the formula assumes independent failures. Two providers that both depend on the same upstream card network are not independent.