Spark Architecture & Execution Model

Interviewers ask about Spark architecture because it is the fastest way to tell whether a candidate has actually operated a distributed engine or has only called .show() on a notebook a few times. A weak answer describes Spark as "a faster way to do pandas on big data." A strong answer can explain why a chain of ten .filter() and .withColumn() calls does nothing until an action is called, what physically happens on the cluster between that action and the result landing in the driver, why the shuffle is almost always where the time and the cost go, and — just as important — when the honest answer is "I wouldn't use Spark for this at all." That last point is the one most candidates miss: Spark is a tool for data that does not fit comfortably on one machine, not a default choice, and knowing where that line sits is itself a signal of production experience.

This subject builds the mental model from the ground up: the driver/executor/cluster-manager split that defines what "distributed" means concretely in Spark; the RDD and DataFrame APIs and why DataFrames are the default today; lazy evaluation and the precise transformation-vs-action distinction that trips up nearly every new Spark engineer at least once; the Catalyst optimizer and how a DataFrame plan becomes an executed physical plan; Adaptive Query Execution and why it exists; and partitions, the unit that ties all of the above to actual CPU cores doing actual work. It closes with the decision an interviewer is really testing for: Spark or something smaller. Deeper tuning of the concepts introduced here — shuffle configuration, join strategy selection, memory management, spill behavior — is covered in spark-performance-tuning; this subject is the architectural foundation that tuning advice only makes sense on top of.