Paths Subjects Questions Quizzes Pricing Search
Intermediate Open Pro

Design Review: Does This Streaming-to-Warehouse Pipeline Achieve Effectively-Once?

A teammate proposes the following design for landing Kafka events (each event has a unique event_id) into a warehouse table, and asks you to review it before it ships:

"Our consumer reads events from Kafka and commits its offset back to Kafka immediately after reading each batch, before processing it. Then it writes the batch to the warehouse with a plain INSERT. If the warehouse write fails, we just log the error and move on to the next batch — we don't want a stuck consumer blocking the whole pipeline. Since every event has a unique event_id, and Kafka guarantees each event is delivered, I think we're safe."

  1. Identify every place this design deviates from "at-least-once delivery + idempotent writes = effectively-once," and explain the concrete failure mode each deviation causes.
  2. Redesign the consumer's commit and write logic to actually achieve effectively-once processing.
  3. Your teammate points out that event_id is unique per event and asks why that alone doesn't already make the plain INSERT safe. Answer precisely.

Share this question

← Back to ETL vs ELT & Pipeline Design practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.