The Readiness Check That Took Down Everything

A Kubernetes-hosted API has 12 pods behind a load balancer. Its readiness probe calls /ready, which returns 200 only if the pod can execute SELECT 1 on the primary database and get a 200 from the recommendations service. During a 90-second recommendations outage, all 12 pods were marked unready simultaneously and the API returned 503 for every endpoint, including ones that never use recommendations. Ten minutes later a database failover caused the same thing, and additionally the pods were restarted repeatedly.

1. Explain what went wrong in each incident and the distinction between liveness and readiness that was violated.
2. Redesign the probes. What should each check, and how should the recommendations dependency be handled instead?
3. During the DB failover, why were the pods restarted, and why did that make things worse?