Paths Subjects Questions Quizzes Pricing Search
Intermediate Open Pro

Diagnosing a 40-Minute Straggler Task in an Otherwise 2-Minute Job

You're paged because a nightly Spark job that normally finishes in about 12 minutes has been taking over 45 minutes for the past three nights, with no code changes and no meaningful growth in total input data volume. Looking at the Spark UI's stage detail for the groupBy aggregation stage, you see 199 tasks complete within 30-90 seconds of each other, and one single task still running after 40 minutes.

  1. What is the most likely root cause, and what in the data would you check to confirm it?
  2. Adaptive Query Execution is enabled in this cluster's configuration (the Spark version is 3.3). Why might it not have automatically fixed this, or what would you check to see if it did try to?
  3. Propose a concrete fix, and explain why simply adding more executors to the cluster would not resolve this specific symptom.

Share this question

← Back to Spark Architecture & Execution Model practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.