Adding .cache() Made the Pipeline Slower, Not Faster

An engineer notices a multi-step ETL pipeline recomputes an expensive filtered-and-joined intermediate DataFrame (enriched_orders, built from a shuffle join and a filter) three times across three downstream branches, and adds .cache() right after it's created, expecting a speedup:

enriched_orders = (
    orders.join(customers, "customer_id")   # shuffle join
          .filter(F.col("status") == "completed")
)
enriched_orders.cache()

branch_a = enriched_orders.groupBy("region").agg(F.sum("amount"))
branch_b = enriched_orders.groupBy("product_id").agg(F.count("*"))
branch_c = enriched_orders.write.parquet(output_path)

After deploying, the job's total runtime increased by about 20%. The Spark UI's Storage tab shows enriched_orders is only 40% cached in memory, with the remaining 60% spilled to disk, and other stages running concurrently with the cache now show new, previously-absent spill in their own Spill (Memory)/(Disk) columns.

1. Explain the mechanism by which caching this DataFrame made the overall job slower, connecting it to Spark's unified memory model.
2. Identify one concrete change to what is being cached (not whether to cache at all) that would reduce the problem.
3. Propose how you'd decide, with evidence rather than intuition, whether caching enriched_orders is net-positive after your fix.