Intermediate
Open
Pro
Predicate Pushdown Isn't Helping — Diagnose the Layout
A payments table is stored as Parquet, partitioned by
payment_date, with healthy file sizes (roughly 300 MB each). A
dashboard query filters WHERE payment_date = '2026-08-10' AND merchant_id = 'M-88421'. The payment_date partition pruning works
correctly — the engine only scans that day's files — but within those
files, the query still reads nearly every row group, and profiling
shows almost no row groups get skipped by the merchant_id predicate
even though merchant_id has embedded min/max statistics in every row
group.
- Give the most likely explanation for why
merchant_idpushdown isn't skipping row groups, even though the statistics exist. - Propose a concrete fix, and explain what specifically changes about the file layout that makes it work.
- A colleague suggests instead partitioning by
merchant_idin addition topayment_date, since "partitioning always prunes better than statistics." Evaluate that suggestion given what you know aboutmerchant_id's likely cardinality (tens of thousands of merchants).
Share this question