A Join Stage Is Spilling Tens of GB to Disk Per Task

A job joins two large fact tables (orders, ~600GB, and shipments, ~400GB) on order_id, both too large to broadcast. The job used to take 40 minutes; after orders roughly doubled in size over the past quarter, it now takes 3.5 hours. spark.sql.shuffle. partitions is set to 200 (a value copied from an older, smaller job template and never revisited). In the Spark UI, the join stage's task table shows: median task duration 4 minutes, "Spill (Memory)" around 18GB per task, "Spill (Disk)" around 6GB per task, fairly evenly across almost all 200 tasks (not a small skewed subset). GC time is also elevated, averaging 25% of task duration.

1. Explain why this is not a data-skew problem, using the evidence given, and identify the actual root cause.
2. Propose a concrete fix with numbers, showing your reasoning for the new value you'd choose.
3. Explain why raising spark.executor.memory alone, without touching spark.sql.shuffle.partitions, would be a weaker fix than what you propose in part 2.