A Sensor Is Starving the Entire Worker Pool

Your Airflow deployment runs with 16 worker slots total, shared across roughly 40 DAGs. Every day around 9am, on-call gets paged because dozens of unrelated tasks across many different DAGs are stuck in "queued" state for hours, even though nothing appears to be failing.

Investigation turns up this task, present in 12 different DAGs (each team copy-pasted it from an internal wiki example years ago):

from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor

wait_for_upstream = S3KeySensor(
    task_id="wait_for_upstream_export",
    bucket_name="partner-exports",
    bucket_key="{{ ds }}/export_complete.flag",
    poke_interval=120,
    timeout=60 * 60 * 8,   # 8 hours
)

The partner's export process is known to be unreliable and sometimes doesn't finish until early afternoon.

1. Explain precisely why 12 copies of this task, none of which are failing, can still cause unrelated tasks in other DAGs to sit queued for hours.
2. Propose two different fixes, at two different levels (a task-config-level fix and a deployment-level fix), and explain the trade-off between them.
3. A teammate asks: "why not just remove the sensor and instead schedule the downstream DAGs to start at 2pm, after the partner export is usually done?" Explain what this "fix" trades away.