Paths Subjects Questions Quizzes Pricing Search
Intermediate Open Pro

Reading an `explain()` Plan to Explain a Slow Join

A colleague asks you to look at why a join is slow. They run result_df.explain() and share this physical plan:

== Physical Plan ==
*(5) Project [order_id#12, customer_name#45, total#20]
+- *(5) SortMergeJoin [customer_id#13], [customer_id#44], Inner
   :- *(2) Sort [customer_id#13 ASC NULLS FIRST], false, 0
   :  +- Exchange hashpartitioning(customer_id#13, 200)
   :     +- *(1) Filter isnotnull(customer_id#13)
   :        +- *(1) FileScan parquet [order_id,customer_id,total]
   +- *(4) Sort [customer_id#44 ASC NULLS FIRST], false, 0
      +- Exchange hashpartitioning(customer_id#44, 200)
         +- *(3) Filter isnotnull(customer_id#44)
            +- *(3) FileScan parquet [customer_id,customer_name]

They mention the customers table (the right side of the join, scanned in stage 3) is small — only about 2,000 rows, a few hundred KB — while orders (left side) is hundreds of millions of rows.

  1. Identify the join strategy Catalyst chose, and explain, given the table sizes described, why this is very likely the wrong choice.
  2. Explain what specifically is expensive about this plan in terms of the physical work it causes across the cluster.
  3. Propose a fix, and explain the mechanism by which it would avoid the expensive part of this plan. If AQE were enabled here, would you expect it to have already fixed this automatically — why or why not?

Share this question

← Back to Spark Architecture & Execution Model practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.