Airflow & Workflow Orchestration
DAGs, the scheduling model, sensors vs deferrable operators, retries and SLAs, dynamic task mapping, XComs, and idempotent task design from the orchestrator's point of view
A practitioner's tour of Airflow as a data engineering interview topic: how DAGs, tasks, and operators fit together; why the scheduling model's catchup default silently reprocesses history; why classic sensors starve the worker pool and what deferrable operators fix; how retries, SLAs, and alerting are actually wired up; dynamic task mapping for runtime-sized fan-out; why large payloads must never flow through XComs; idempotent task design as the property that makes retries and backfills safe; and a short comparison of Airflow's task-centric model to Dagster and Prefect's asset-centric one.
Practice questions (6)
-
View →
A New DAG Silently Reprocessed a Year of History
Intermediate · Free -
View →
A Sensor Is Starving the Entire Worker Pool
Intermediate -
View →
The Airflow UI Is Getting Slower Every Week — And So Is the Scheduler
Intermediate -
View →
Retries Are Duplicating Rows in a Metrics Table
Intermediate -
View →
A Two-Hour-Late Dashboard and No One Was Told
Intermediate