Streaming Fundamentals & Kafka

Interviewers reach for streaming questions because it's one of the fastest ways to tell whether a candidate has actually operated a stateful, always-on system or has only read about one. Batch pipelines fail loudly and recover simply — rerun the job. Streaming pipelines fail in ways that are subtle, stateful, and often invisible until a downstream number is quietly wrong: a consumer that silently falls behind, a window that closes one second too early and drops real data, a producer retry that writes the same record twice into a system with no idea what "twice" means. A weak answer describes Kafka as "a queue" and exactly-once as something you turn on with a config flag. A strong answer can explain, mechanically, why a partition is the unit of both parallelism and ordering, why "exactly-once" is really "effectively-once" achieved through two specific, nameable mechanisms, and why almost every hard bug in a streaming system traces back to conflating when an event happened with when the system noticed it.

This subject also covers the question that should come before all the mechanics: whether streaming is the right tool at all. A team that reaches for Kafka and a stateful stream processor to solve a problem that a 15-minute batch job would have solved just as well has taken on real, ongoing operational cost for no product benefit — and knowing when not to build a streaming system is as much a signal of seniority as knowing how to build one correctly. etl-vs-elt-and-pipeline-design covers pipeline design more broadly; this subject is the deep dive on the specific case where "the pipeline never stops running."