File Formats, Partitioning & Storage Layout

Interviewers probe this topic because it is where "I can write correct SQL" and "I can run that SQL against a petabyte of data for a reasonable price" diverge. A query that scans a well-partitioned, well-compressed Parquet dataset with tight column statistics can finish in seconds by touching a few megabytes; the identical query against the same logical data, stored as row-oriented, unpartitioned, gzip-compressed CSV split into fifty thousand tiny files, can take an hour and cost real money — and no amount of SQL cleverness fixes that, because the bottleneck is physical layout, not the query plan. Weak candidates treat file format and partitioning as an operational afterthought ("we use Parquet because everyone does"). Strong candidates can explain, mechanically, why a specific layout decision changes how many bytes an engine has to read off disk, and can diagnose a slow query by asking "what does the storage layout look like" before "what does the query look like."

This subject covers the physical storage layer that sits underneath everything else in this track: how sql-query-optimization-and-indexing reasons about indexes and access paths inside a database, this subject reasons about files and directories inside a data lake or lakehouse, which data-warehouses-and-lakehouses covers architecturally. The two are the same discipline pointed at different targets — minimizing the bytes an engine has to touch to answer a query — and the same intuitions (selectivity, pruning, avoiding full scans) transfer directly.