Spark Performance Tuning

Interviewers ask about Spark performance tuning because it is the fastest way to tell whether a candidate has actually run jobs against real data at real scale, or only ever run df.show() against a laptop-sized sample. Every senior data engineer has a story about a job that took six hours and, after one change, took twenty minutes — and the story is never "we added more machines." It is almost always one of a small number of root causes: an unnecessary shuffle, a handful of keys carrying the whole dataset's skew, a join Spark should have broadcast and didn't, partitions so small the scheduler drowns in overhead or so large that a single task blows through executor memory, a cache that cost more than it saved, or a job quietly spilling gigabytes to disk on every stage while nobody looked at the Spark UI to notice.

What separates a strong answer from a weak one in this topic is not knowing more configuration flags. It's the ability to reason from mechanism: why does a shuffle require writing to disk before the network transfer even starts, why does salting fix skew but not for every kind of aggregation, why does a storage level with off-heap deserialized objects behave differently under memory pressure than one with serialized on-heap bytes. A weak answer recites spark.sql.shuffle.partitions = 200 as a magic number. A strong answer explains what that number controls, how to compute a better one from data volume and target file size, and how to verify the fix actually worked by reading the Spark UI before and after. This subject builds that mechanistic understanding, one failure mode at a time, and ends with a full diagnose-and-fix walkthrough that exercises all of it together. It assumes the execution model covered in spark-architecture-and-execution-model — jobs, stages, tasks, the DAG scheduler, and the driver/executor split — as background; this subject is specifically about what makes that execution slow and how to make it fast.