Intermediate
Open
Pro
Design Review: Does This Streaming-to-Warehouse Pipeline Achieve Effectively-Once?
A teammate proposes the following design for landing Kafka events
(each event has a unique event_id) into a warehouse table, and asks
you to review it before it ships:
"Our consumer reads events from Kafka and commits its offset back to Kafka immediately after reading each batch, before processing it. Then it writes the batch to the warehouse with a plain
INSERT. If the warehouse write fails, we just log the error and move on to the next batch — we don't want a stuck consumer blocking the whole pipeline. Since every event has a uniqueevent_id, and Kafka guarantees each event is delivered, I think we're safe."
- Identify every place this design deviates from "at-least-once delivery + idempotent writes = effectively-once," and explain the concrete failure mode each deviation causes.
- Redesign the consumer's commit and write logic to actually achieve effectively-once processing.
- Your teammate points out that
event_idis unique per event and asks why that alone doesn't already make the plainINSERTsafe. Answer precisely.
Share this question