Case Study: Design a Streaming Event Pipeline

"Design the pipeline that powers our real-time analytics dashboards" sounds like a smaller problem than it is, because the phrase "real-time" hides most of the actual difficulty. A weak answer jumps straight to "we'll use Kafka and Flink" and spends the rest of the interview describing a topology diagram — producers, a topic, a processor, a database — without ever confronting the questions that determine whether that topology is correct: what happens when a consumer falls behind, what happens when an event shows up eleven minutes after the window it belongs to already closed, what happens when a checkpoint fails halfway through writing to two different sinks, what happens when a mobile team ships a new event field and half the fleet is still running the old app version. A strong answer treats "real-time" as a spectrum of guarantees — freshness, ordering, completeness, exactly-once — and makes an explicit, justified decision about how much of each guarantee each part of the system actually needs, because those guarantees trade directly against latency, cost, and complexity, and asking for all of them everywhere is not a design, it's a wish list.

This case study designs a clickstream/event-analytics pipeline for a consumer product: web and mobile clients emit events (page views, clicks, add-to-cart, purchases, errors), the pipeline computes near-real-time aggregates for operational dashboards (active users per minute, revenue per minute, funnel conversion), and it also lands every event durably for historical analytics, ad hoc SQL, and ML feature backfills. That dual mandate — a live number an on-call engineer stares at, and an immutable historical record an analyst queries eighteen months later — is what makes this a genuine system-design problem rather than a "pick a framework" question. streaming-fundamentals-and-kafka is the deep dive behind the ingestion mechanics in Step 2; this case study is the model answer that assembles ingestion, stream processing, storage, and operability into one coherent, defensible system, the way an interview expects.

The single sentence to keep coming back to across every step: match the guarantee to what's actually at stake, and keep exactly one place that remembers the truth. Every decision below — partitioning, windowing, exactly-once scope, the kappa choice, schema rules — is in service of that sentence.