Intermediate
Open
Pro
Reading an `explain()` Plan to Explain a Slow Join
A colleague asks you to look at why a join is slow. They run
result_df.explain() and share this physical plan:
== Physical Plan ==
*(5) Project [order_id#12, customer_name#45, total#20]
+- *(5) SortMergeJoin [customer_id#13], [customer_id#44], Inner
:- *(2) Sort [customer_id#13 ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(customer_id#13, 200)
: +- *(1) Filter isnotnull(customer_id#13)
: +- *(1) FileScan parquet [order_id,customer_id,total]
+- *(4) Sort [customer_id#44 ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(customer_id#44, 200)
+- *(3) Filter isnotnull(customer_id#44)
+- *(3) FileScan parquet [customer_id,customer_name]
They mention the customers table (the right side of the join,
scanned in stage 3) is small — only about 2,000 rows, a few hundred KB
— while orders (left side) is hundreds of millions of rows.
- Identify the join strategy Catalyst chose, and explain, given the table sizes described, why this is very likely the wrong choice.
- Explain what specifically is expensive about this plan in terms of the physical work it causes across the cluster.
- Propose a fix, and explain the mechanism by which it would avoid the expensive part of this plan. If AQE were enabled here, would you expect it to have already fixed this automatically — why or why not?
Share this question