Paths Subjects Questions Quizzes Pricing Search
Advanced Open Pro

A 3am Pipeline Failure With No Usable Runbook

An on-call engineer, new to the data platform team, gets paged at 3am because the nightly orders_fact dbt model failed its not_null test on customer_id. The DAG's only documentation is a one-line description: "builds the orders fact table." The engineer has no context on who depends on this table, whether it's safe to just rerun it, or whether the null values are a new data quality problem or a known, tolerated edge case. They spend two hours investigating from scratch before determining it's safe to rerun and the underlying cause was a known late-arriving source feed. The table was 5 hours stale by the time it was fixed, and a downstream exec dashboard showed incomplete numbers for the morning stand-up.

  1. List, specifically, what should have been in the runbook that would have taken this from a two-hour cold investigation to a short, confident fix.
  2. The underlying cause was a "known, tolerated edge case" — a late-arriving source feed. If this happens regularly, is a better runbook actually the right long-term fix? What would you build instead, and why?
  3. Propose one change to the DAG or the test itself that would prevent this specific alert from paging anyone at all in the future, while still catching genuinely new problems.

Share this question

← Back to Data Engineering in Production practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.