Case Study: Design a Streaming Event Pipeline
A full data-engineering interview answer for a clickstream analytics pipeline at scale: event volume math, Kafka producer/partition strategy, windowed stream aggregation, the lambda-vs-kappa decision, end-to-end exactly-once semantics, watermarking for late data, schema evolution across years of events, real-time serving plus lakehouse cost math, and monitoring the pipeline itself
Model interview answer for designing a clickstream/event-analytics pipeline that ingests billions of events per day through Kafka, computes real-time aggregates with a stream processor, and lands the same events in a lakehouse for historical analytics: the arithmetic behind peak throughput and partition counts, why partition key choice is the single decision that determines correctness downstream, the windowed-aggregation design and engine choice, why kappa beats lambda here and what it doesn't solve, exactly-once semantics traced through every hop from producer to sink, watermarking and late-data handling, a schema-evolution strategy that survives years of producer changes, cost and storage math at scale, and the monitoring signals that catch pipeline failure before the business notices.
Practice questions (5)
-
View →
A Consumer Group Is Falling Behind Under Load
Advanced · Free -
View →
Adding a New Downstream Consumer Without Breaking Existing Ones
Advanced -
View →
Revenue Numbers Doubled After a Flink Job Restart
Advanced -
View →
Redesigning for a Sustained 10x Traffic Spike
Advanced -
View →
Real-Time and Lakehouse Revenue Numbers Disagree
Advanced