Designing a Compaction Strategy for a Streaming Landing Zone

A Kafka-fed streaming job writes micro-batches into a lake table every 90 seconds to keep a downstream dashboard near-real-time. Each micro-batch writes one small Parquet file per output partition (partitioned by event_date). After a few weeks in production, a single day's partition has accumulated around 900 files averaging 3 MB each, and analysts running ad hoc queries against yesterday's data report queries that used to take 5 seconds now taking over a minute.

1. Confirm this is a small-files problem and explain the mechanism by which 900 × 3 MB files are slower to query than, say, 15 × 180 MB files covering the same data.
2. You cannot slow down the 90-second micro-batch commits — the near-real-time dashboard depends on that cadence. Design a compaction approach that fixes the analyst-facing query performance without touching the ingestion cadence.
3. What would you monitor to know compaction is keeping up, and what would trigger you to compact more aggressively?