Yesterday's Output Created 40,000 Tiny Files and Today's Read Job Is Slow

A daily job writes roughly 6GB of aggregated results:

result.repartition(2000).write.parquet(output_path)

The repartition(2000) was copied from a much larger upstream job months ago and never adjusted for this smaller output. The output directory now contains around 40,000 tiny Parquet files (many well under 1MB — more files than partitions because the source data itself wasn't perfectly evenly distributed across all 2000 partitions before the repartition shuffle). Separately, a downstream job that reads this output every morning has gotten progressively slower over the past few months as more days of output accumulated, even though the total data volume per day hasn't grown.

1. Explain the two distinct costs this small-file problem is imposing — one on the write side, one on the read side — and why they're different mechanisms.
2. Propose a fix for the write path, with the specific number you'd target and why.
3. The downstream reader has gotten slower over months even though each day's data volume is constant. Explain why, and state whether fixing today's write job alone resolves the existing problem.