Subjects

Data Engineering in Production

The practitioner-level layer that separates a data engineer who has built pipelines in a notebook from one who has run them in production: testing dbt models and DAGs before merge with slim CI, dev/staging/prod environment strategy and promotion, secrets management for pipeline credentials, cost optimization in cloud warehouses (clustering, auto-suspend, per-query attribution, the runaway-warehouse story), on-call and runbooks for pipeline failures, how sources, orchestration, storage, transformation, and BI actually compose end to end with real ownership boundaries, and the documentation practices that keep a pipeline alive after its author leaves.

Case Study: Design a Streaming Event Pipeline

Model interview answer for designing a clickstream/event-analytics pipeline that ingests billions of events per day through Kafka, computes real-time aggregates with a stream processor, and lands the same events in a lakehouse for historical analytics: the arithmetic behind peak throughput and partition counts, why partition key choice is the single decision that determines correctness downstream, the windowed-aggregation design and engine choice, why kappa beats lambda here and what it doesn't solve, exactly-once semantics traced through every hop from producer to sink, watermarking and late-data handling, a schema-evolution strategy that survives years of producer changes, cost and storage math at scale, and the monitoring signals that catch pipeline failure before the business notices.

Case Study: The SQL Interview Gauntlet

A worked answer key for the SQL problems that show up over and over in data engineering interviews, structured as seven realistic prompts with sample data and full solutions: funnel conversion analysis with drop-off rates, N-day retention by signup cohort, sessionizing a raw event log with a gap threshold, deduplication strategies compared (ROW_NUMBER, DELETE, DISTINCT ON), efficient running/cumulative metrics without a self-join, the classic Nth-highest-value problem generalized to Nth-highest-per-group with DENSE_RANK, and a this-year-vs-last-year cohort comparison in a single query. Each problem is reasoned from the execution model rather than pattern-matched from memory.

Case Study: Design a Batch Analytics Platform

Model interview answer for designing the batch analytics platform behind a mid-size company's BI and reporting layer, from OLTP sources through CDC/ELT ingestion, lakehouse/warehouse landing, dbt transformation layers, and Airflow orchestration to dashboards: concrete row-count and byte-volume math that drives every downstream decision, the CDC-vs-batch-extract call made per source rather than uniformly, warehouse-vs-lakehouse and file-format/partitioning choices, staging/intermediate/marts layering in dbt, an Airflow DAG shaped around data-aware dependencies rather than fixed clock time, an SLA built with deliberate slack instead of run at the theoretical minimum, an idempotent partition-scoped backfill strategy, cost control through partition pruning and materialization rather than bigger warehouses, data-quality gates that quarantine rather than silently drop or block, and a team/ownership model that treats the staging layer as a reviewed public interface.

Spark Performance Tuning

A practitioner's guide to Spark performance tuning as a data engineering interview topic: what a shuffle actually costs and what triggers one, diagnosing and fixing data skew with salting and adaptive query execution, when a broadcast join is a free win and when the threshold needs tuning, sizing partitions against the target-file-size heuristic, choosing a persistence storage level (and knowing when caching makes things worse), the execution-vs-storage memory split and what spill to disk looks like, reading the Spark UI's stage and task views to find the actual bottleneck, and a fully worked diagnose-and-fix walkthrough that ties all of it together.