Diagnosing and Preventing a Runaway Warehouse Cost Spike

1. How the two settings combined

A one-hour AUTO_SUSPEND means the warehouse only suspends after a full 60 minutes with zero queries. If literally anything — a scheduled dashboard refresh, an analyst's ad hoc query, a lingering connection from the backfill session, even a low-frequency monitoring query — touches that warehouse at least once an hour, the idle timer never actually elapses and the warehouse never suspends. An X-LARGE warehouse billing continuously for 11 days at that size is exactly what you'd expect if the warehouse simply never went idle for a full hour at a stretch, which on a shared analytics warehouse touched by multiple scheduled jobs and analysts is entirely plausible. The resize to X-LARGE compounded this: the same "never actually suspends" behavior now bills at a much higher per-second rate, so the same operational mistake (a too-long auto-suspend) costs several times more per day than it would have at a smaller size. Neither setting alone is unusual in isolation (a demo not wanting cold starts, a backfill wanting more compute) — the incident is the combination, left in place after the circumstances that motivated each setting had passed.

2. Concrete fixes

• Set a short AUTO_SUSPEND (60-300 seconds) as the enforced default for every warehouse, via a policy/template rather than a per-warehouse manual setting an individual can silently override — e.g., a Terraform module or account-level default that provisions new warehouses with a short auto-suspend baked in, so "someone forgot to change it back" isn't possible in the first place.
• Separate warehouses by workload and size, not one shared warehouse resized ad hoc. A dedicated, appropriately-sized warehouse for one-off backfills that is provisioned, used, and explicitly dropped or resized back down as part of the backfill's own runbook/script — not a shared analytics warehouse manually resized up and, implicitly, expected to be resized back down by someone remembering to.
• Require resize-up changes to be time-boxed or scripted, e.g., a backfill script that resizes the warehouse up, runs the job, and resizes it back down (or suspends it) as its last step regardless of success or failure — so the "make it finish faster" instinct doesn't leave a manual, easy-to-forget change in place indefinitely.
• Budget/spend alerts independent of root cause — an absolute or percentage-based daily spend threshold per warehouse that pages someone the day spend crosses it, which catches this failure mode (and any other cause of anomalous spend) without needing anyone to have predicted "auto-suspend interacting with a resize" specifically.

3. Faster detection

Track warehouse-active-time and daily cost per warehouse as a metric with a same-day or next-day alert threshold — e.g., "any warehouse billing more than N hours in a 24-hour period" or "any warehouse's daily cost more than 2x its trailing 30-day average" — computed from WAREHOUSE_METERING_HISTORY on a scheduled query that runs daily (or more often) and posts to a monitoring channel or pages on-call. The key property is that this doesn't require predicting the specific failure mode (auto-suspend + resize) in advance — it's a generic anomaly signal on "is this warehouse behaving like its historical baseline," the same posture as alerting on response-length or refusal-rate shifts for a production LLM prompt: track a metric broad enough to catch classes of problems you haven't seen yet, sliced by the resource (here, warehouse name) so the alert points directly at the actionable cause instead of only a top-line monthly bill.