Advanced
Open
Pro
The Airflow Scheduler Goes Down for Two Hours During the Nightly Run
It's 2:15am. The nightly pipeline is mid-run — CDC data has landed, staging models have completed, and the intermediate sessionization model is executing — when the Airflow scheduler crashes and stays down for two hours due to an infrastructure issue unrelated to the pipeline itself. The 7:00am SLA still applies.
- Walk through exactly what happens to the in-flight and not-yet-started tasks during those two hours, and what state the warehouse is left in.
- Once the scheduler recovers, what needs to happen for the pipeline to still hit (or safely miss with clear communication) the 7:00am SLA?
- What would you change about this design, in advance, so that a two-hour scheduler outage is a survivable event rather than a crisis each time it happens?
Share this question