ETL vs ELT & Pipeline Design
Why cheap warehouse compute flipped transform-then-load into load-then-transform, and the idempotency, incremental-load, CDC, backfill, and retry semantics that separate a pipeline that survives production from one that quietly corrupts a table
A practitioner's tour of batch pipeline design as a data engineering interview topic: the real history behind ETL vs ELT and why elastic cloud-warehouse compute made load-then-transform the default, idempotency as the non-negotiable property of any job that will ever be re-run, watermark-based incremental loading and merge/upsert patterns, log-based CDC (Debezium) vs query-based timestamp CDC and their tradeoffs, why backfilling a year of history is a structurally different problem than the daily job, and how at-least-once delivery plus idempotent writes gets you effectively-once processing without a distributed transaction.
Practice questions (5)
-
View →
A Retried Job Double-Counted Revenue — Diagnose and Redesign
Intermediate · Free -
View →
Rows Silently Stopped Appearing — A Watermark Design Bug
Intermediate -
View →
A Year-Long Backfill Overwhelmed the Source Database
Intermediate -
View →
Choosing Between Query-Based and Log-Based CDC
Intermediate -
View →
Design Review: Does This Streaming-to-Warehouse Pipeline Achieve Effectively-Once?
Intermediate