Poison Messages, Retry Topics and DLQ on a Log

Your notification service consumes user-events (Kafka, 16 partitions, key = user_id) and calls an SMS provider. Two problems appear:

• a message with a malformed phone number fails deserialisation every time and the whole partition behind it stops moving;
• the SMS provider returns HTTP 503 for about two minutes several times a day, and the naive "retry immediately, in a loop" logic hammers it.

1. Why does one bad message stop an entire partition on a log, when it would not stop a queue?
2. Design a retry + dead-letter scheme that (a) unblocks the partition, (b) retries transient provider errors with increasing delays, and (c) does not retry the malformed message at all. Show the topic layout and the delay schedule.
3. What ordering guarantee do you lose with this design, and does it matter for SMS notifications?