Paths Subjects Questions Quizzes Pricing Search
Advanced Open Pro

The Airflow Scheduler Goes Down for Two Hours During the Nightly Run

It's 2:15am. The nightly pipeline is mid-run — CDC data has landed, staging models have completed, and the intermediate sessionization model is executing — when the Airflow scheduler crashes and stays down for two hours due to an infrastructure issue unrelated to the pipeline itself. The 7:00am SLA still applies.

  1. Walk through exactly what happens to the in-flight and not-yet-started tasks during those two hours, and what state the warehouse is left in.
  2. Once the scheduler recovers, what needs to happen for the pipeline to still hit (or safely miss with clear communication) the 7:00am SLA?
  3. What would you change about this design, in advance, so that a two-hour scheduler outage is a survivable event rather than a crisis each time it happens?

Share this question

← Back to Case Study: Design a Batch Analytics Platform practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.