Raising the Broadcast Threshold Caused Executor OOMs

To speed up several joins against a product_catalog table, an engineer globally raises the broadcast threshold:

spark.conf.set("spark.sql.autoBroadcastJoinThreshold", str(500 * 1024 * 1024))  # 500MB

Most joins against product_catalog (normally ~80MB) now broadcast and run faster, as intended. But a separate, unrelated job that joins a promotions table (normally small, but occasionally balloons to 600-900MB during major sale events due to a wide denormalized promotional-text column) starts failing with executor out-of-memory errors during exactly those sale-event windows — the same job ran fine the rest of the year.

1. Explain precisely why raising the global threshold caused a previously-unrelated job to start failing, connecting it to what a broadcast join actually does on each executor.
2. Propose a fix that keeps the product_catalog speedup but removes the risk to the promotions job, and explain the trade-off between two possible approaches.
3. What would you check in the Spark UI to confirm the fix worked without waiting for the next sale event to test it live?