Airflow & Workflow Orchestration

Interviewers ask about orchestration because it is where a pipeline's correctness actually lives, not where its business logic lives. The SQL in a transformation step can be perfect and the pipeline can still be wrong — because a task ran twice and duplicated rows, because a backfill silently reprocessed six months of history the moment a DAG was turned on, because a sensor held a worker slot hostage for four hours and starved every other job in the queue, or because a retry re-ran a task whose side effects weren't safe to repeat. Airflow (and orchestrators like it) is not primarily a scheduler in the cron sense; it is a system for making "this graph of work happens, in order, exactly once per logical unit of time, and recovers correctly when something fails" an operational guarantee instead of a hope.

The gap between a weak and a strong answer on this topic is almost never about DAG syntax. Anyone can write task_a >> task_b. The gap is in the parts that only show up once something has gone wrong in production: understanding that Airflow's default scheduling behavior (catchup=True) will happily fire off months of historical runs the instant you deploy a new DAG with a past start_date; knowing that a classic sensor sitting in a poke loop is occupying a scarce worker slot the entire time it waits, and what to do instead; knowing that a task's return value in the TaskFlow API becomes a database row via XCom, and that "just return the DataFrame" is a way to bring down the metadata database; and being able to state, precisely, what makes a task safe to retry or backfill — the same idempotency concern covered from the pipeline-design side in etl-vs-elt-and-pipeline-design, but here from the orchestrator's side: what does Airflow guarantee about task execution, and what is left for the task's own code to guarantee?

This subject works through DAG structure, the scheduling model, sensors and deferrable operators, retries/SLAs/alerting, dynamic task mapping, XComs, idempotent task design, and a brief comparison to Dagster and Prefect's asset-centric model — the axis interviewers use to check whether you understand why Airflow is shaped the way it is, not just how to use it.