Advanced
Open
Pro
Adding a New Downstream Consumer Without Breaking Existing Ones
A fraud-detection team wants to subscribe to events.commerce to build
a near-real-time model that flags suspicious purchase patterns. They
ask for two things: (a) access to the raw event stream, and (b) the
ability to occasionally replay the last 30 days of events to
retrain/backtest their model, which is longer than this pipeline's
7-day Kafka retention.
- Can this new consumer be added safely without any changes to the existing pipeline? If changes are needed, what are they and why?
- Their request for 30-day replay exceeds Kafka's retention. Propose a way to satisfy it that doesn't mean extending Kafka's retention to 30 days for everyone.
- The fraud team mentions they may eventually want to write a
fraud_flagevent back onto the stream so other systems can react to it. What does this request imply that the current design does not yet have, and how would you extend the design to support it safely?
Share this question