A 3am Pipeline Failure With No Usable Runbook
An on-call engineer, new to the data platform team, gets paged at 3am
because the nightly orders_fact dbt model failed its not_null test
on customer_id. The DAG's only documentation is a one-line
description: "builds the orders fact table." The engineer has no
context on who depends on this table, whether it's safe to just rerun
it, or whether the null values are a new data quality problem or a
known, tolerated edge case. They spend two hours investigating from
scratch before determining it's safe to rerun and the underlying cause
was a known late-arriving source feed. The table was 5 hours stale by
the time it was fixed, and a downstream exec dashboard showed
incomplete numbers for the morning stand-up.
- List, specifically, what should have been in the runbook that would have taken this from a two-hour cold investigation to a short, confident fix.
- The underlying cause was a "known, tolerated edge case" — a late-arriving source feed. If this happens regularly, is a better runbook actually the right long-term fix? What would you build instead, and why?
- Propose one change to the DAG or the test itself that would prevent this specific alert from paging anyone at all in the future, while still catching genuinely new problems.
Share this question