Intermediate
Open
Pro
Should This Nightly Report Run on Spark or a Single Machine?
Your company's finance team needs a nightly report: join a 40GB table of transactions against an 800MB table of account metadata, filter to the last 90 days, aggregate revenue by region and product line, and write the result (a few thousand rows) to a dashboard-backing table. The engineer proposing the design defaults to "we'll build this in Spark since that's our standard for data pipelines," and wants to provision a small permanent Spark cluster (4 executors, 8 cores each) to run it nightly.
- Evaluate whether Spark is the right choice here, reasoning from the actual data volumes involved rather than "Spark is our standard."
- Propose a specific alternative architecture if you think a single-node engine is more appropriate, including what it would read from and write to.
- Under what future change to this workload would you revisit the decision and consider moving it to Spark (or keeping it there, if it's already there)?
Share this question