Debugging Duplicate Events Despite an 'Exactly-Once' Pipeline
Your team runs a Kafka-based order pipeline that reads from an
orders topic, computes a per-order total, and writes the result to a
Postgres order_totals table via a plain JDBC INSERT. Kafka
transactions are enabled end to end (idempotent producer,
transactional.id set, consumers read with read_committed), and the
team's documentation confidently describes the system as
"exactly-once." Finance reports that a small number of orders show up
twice in order_totals with identical totals, always following
brief periods of consumer-group rebalancing (deploys, pod restarts).
- Explain precisely why Kafka's transactional guarantees do not prevent this duplication, given what's described above.
- Identify the specific moment in the consume-process-write sequence where a rebalance-triggered crash produces a duplicate row.
- Propose a concrete fix, and explain why it closes the gap that Kafka's own exactly-once mechanisms don't cover.
1. Why Kafka's guarantees don't prevent this
Kafka's exactly-once mechanisms — the idempotent producer and
transactions — guarantee exactly-once behavior only within Kafka's own
transactional boundary: they ensure no duplicate writes land in a
Kafka partition on producer retry, and that a set of Kafka writes plus
a consumer offset commit are atomic and only visible to
read_committed consumers once fully committed. None of that extends
to what happens after the consumer reads a committed record and takes
an external action — here, a plain JDBC INSERT into Postgres. The
INSERT has no unique constraint or idempotency key, so nothing
prevents the same order from being inserted twice if the consuming
application processes it more than once. The team's "exactly-once"
label describes the Kafka-internal hop correctly but is false for the
pipeline as a whole, which is only as strong as its weakest,
non-idempotent link.
2. Where the duplicate is actually produced
The failure window is between finishing the external side effect
(the INSERT succeeding) and the consumer offset commit being
finalized. A consumer-group rebalance (triggered by a deploy or pod
restart) can revoke a partition from a consumer instance mid-flight:
if the instance has already executed the INSERT for a record but the
rebalance happens before its offset commit for that record is
finalized, the partition gets reassigned to another consumer instance,
which resumes from the last committed offset — which is still
before the record that was just inserted. That instance reprocesses
the same record and issues a second INSERT for the same order. This
is a textbook at-least-once failure mode: the side effect happened,
but the bookkeeping that would have prevented reprocessing it didn't
get durably recorded before the reassignment.
3. The fix
Make the external write idempotent, since Kafka's own guarantees
cannot be extended to cover it: add a unique constraint on the
business key that identifies an order (e.g., order_id, or better, a
stable per-message identifier if order_id could legitimately repeat
for other reasons), and change the write from a plain INSERT to an
upsert (INSERT ... ON CONFLICT (order_id) DO UPDATE in Postgres, or
equivalent). With that in place, reprocessing the same order after a
rebalance-triggered redelivery still executes a write, but it's a
no-op relative to the existing row rather than a duplicate — the
effect becomes exactly-once even though the underlying delivery is
still at-least-once, which is the only realistic way to make an
external, non-transactional sink safe under Kafka's guarantees. A
secondary improvement worth naming: only commit the Kafka offset
after the database write succeeds (not before), so a crash before
the write completes results in reprocessing (safe, given the upsert)
rather than silently losing the record — reinforcing at-least-once
delivery paired with an idempotent sink, which together produce the
effectively-once behavior the team's documentation should actually
claim.
Share this question