Choosing Between a Queue and a Log
Your team is adding asynchronous processing to two different features:
- Feature A: image uploads. Each upload must be resized into three sizes by a pool of workers. Nobody else cares about the event, and once the resize is done the message has no further value. Some resizes take 30 s; occasionally a corrupt file crashes the worker.
- Feature B:
OrderStatusChangedevents. Fulfilment, email, fraud scoring and a data-warehouse loader all need every event, events for one order must be seen in order, and the warehouse team wants to reload the last 30 days after a bug fix.
- Which messaging model (consume-and-delete queue vs log-based stream) fits each feature, and why?
- For Feature A, what mechanism prevents a slow-but-healthy 30 s resize from being handed to a second worker, and what mechanism handles the corrupt-file case?
- For Feature B, what would you use as the partition key and what ordering guarantee does that give?
1. Model choice
Feature A is a classic work queue: one logical consumer group, no replay, no fan-out, and per-message features (long processing, failure quarantine) matter. A consume-and-delete queue (SQS, RabbitMQ) fits. Workers are interchangeable and scale without any partition constraint.
Feature B has four independent consumers of the same events, a per-entity ordering requirement, and an explicit replay requirement. That is the log-based streaming profile (Kafka/Kinesis-style): each consumer group tracks its own offsets over the same retained data, order is preserved within a partition, and the warehouse loader can reset its offsets 30 days back. Doing this on a queue would require a fan-out to four separate queues, and replay would be impossible once messages were acked.
2. Feature A mechanisms
Slow-but-healthy work: the queue's visibility timeout (SQS) /
unacked-redelivery window. Set it comfortably above the p99 processing
time — e.g. 120 s for a 30 s job — or extend it from the worker while the
job is still running (SQS ChangeMessageVisibility). If it is left at
the default 30 s, a healthy worker's message is redelivered to a second
worker and the resize is done twice.
Corrupt file (poison message): a redrive policy with a max receive count (e.g. 3) that moves the message to a dead-letter queue. Retry a couple of times in case the crash was transient, then quarantine it so it stops crashing workers and stops occupying the queue. Alert on DLQ depth. Since a corrupt file is a permanent failure, keep the retry count low.
3. Feature B partition key
Key on order_id. All events for one order hash to the same partition
and are consumed in append order, so a consumer never sees Shipped
before Paid for the same order. There is no ordering across different
orders — and none is needed. order_id is high-cardinality, so load
spreads evenly; keying on something like warehouse_id would give a
handful of hot partitions and no benefit, since ordering across orders
is not a requirement.
Share this question