The Airflow UI Is Getting Slower Every Week — And So Is the Scheduler

Over the past two months, the Airflow webserver has become noticeably slower to load the "Task Instances" and "XCom" views, and the scheduler's heartbeat latency has been creeping up. Nobody changed the infrastructure. A junior engineer on the team recently added this task to a DAG that runs every 15 minutes:

@task
def fetch_and_dedupe_events() -> pd.DataFrame:
    raw = _pull_events_from_kafka_batch()      # ~50k rows, ~40MB in memory
    deduped = _dedupe(raw)
    return deduped                              # returned for the next task

@task
def load_events(deduped_events: pd.DataFrame) -> None:
    _bulk_insert(deduped_events, table="events.deduped")

load_events(fetch_and_dedupe_events())

Nothing about this task is failing — it "works."

1. Explain the specific mechanism connecting this task to the creeping UI and scheduler slowness. Be precise about what's actually happening under the hood, not just "XCom is bad for big data."
2. Propose a fix, including what changes about where the data lives and what (if anything) should still flow through XCom.
3. Why did this problem take two months to notice instead of failing immediately?