Spark Performance Tuning
Shuffles, data skew, broadcast joins, partition sizing, caching, memory management, and reading the Spark UI to diagnose a slow job
A practitioner's guide to Spark performance tuning as a data engineering interview topic: what a shuffle actually costs and what triggers one, diagnosing and fixing data skew with salting and adaptive query execution, when a broadcast join is a free win and when the threshold needs tuning, sizing partitions against the target-file-size heuristic, choosing a persistence storage level (and knowing when caching makes things worse), the execution-vs-storage memory split and what spill to disk looks like, reading the Spark UI's stage and task views to find the actual bottleneck, and a fully worked diagnose-and-fix walkthrough that ties all of it together.
Practice questions (5)
-
View →
A GroupBy Aggregation Hangs at 799/800 Tasks
Advanced · Free -
View →
A Join Stage Is Spilling Tens of GB to Disk Per Task
Advanced -
View →
Adding .cache() Made the Pipeline Slower, Not Faster
Advanced -
View →
Yesterday's Output Created 40,000 Tiny Files and Today's Read Job Is Slow
Advanced -
View →
Raising the Broadcast Threshold Caused Executor OOMs
Advanced