Diagnosing a Retry Storm

A mobile app calls an API gateway, which calls a search service, which calls an Elasticsearch cluster. Each layer is configured to retry failed calls up to 3 times immediately (no backoff). Normal search QPS at the cluster is 2,000. One morning the cluster's p99 latency rises from 80 ms to 2 s because of a hot shard, and within a minute it is completely unresponsive. Search error rate at the mobile app goes to 100%.

1. Explain quantitatively how a modest slowdown became a total outage.
2. Redesign the retry policy across the three layers. Be specific about backoff, jitter, budgets, and which layer retries.
3. Which two additional resilience patterns would you add so that a slow cluster no longer takes down the search feature, and what should the user experience be in that mode?