Spark Architecture & Execution Model
Driver, executors, and cluster managers; RDDs vs DataFrames; lazy evaluation and the transformation/action split; the Catalyst optimizer and Adaptive Query Execution; partitions and parallelism; and knowing when Spark is overkill
A practitioner's tour of Spark as a data-engineering interview topic: how the driver, executors, and cluster manager divide work across a cluster; why DataFrames replaced RDDs as the default API; what lazy evaluation actually buys you and the exact moment a chain of transformations becomes real work; how the Catalyst optimizer turns a DataFrame plan into an executed physical plan and how Adaptive Query Execution re-plans that physical plan mid-flight using runtime statistics; how partition count and executor cores determine real parallelism; and the decision framework for recognizing when a single-node warehouse query or an engine like DuckDB would answer the question faster and cheaper than spinning up a cluster.
Practice questions (5)
-
View →
Why Does Nothing Happen Until `.count()` Runs?
Intermediate · Free -
View →
A Job That Ran Fine Across the Cluster Crashes on the Last Line
Intermediate -
View →
Diagnosing a 40-Minute Straggler Task in an Otherwise 2-Minute Job
Intermediate -
View →
Should This Nightly Report Run on Spark or a Single Machine?
Intermediate -
View →
Reading an `explain()` Plan to Explain a Slow Join
Intermediate