A Consumer Group Is Falling Behind Under Load

1. Diagnosing the root cause

Several plausible, non-exclusive causes, and each has a distinct signature to check for before assuming which one applies:

• Insufficient parallelism headroom at the spike. Job parallelism is already 200, matching partition count — at 120,000 events/sec during the spike, that's ~600 events/sec/partition, close to or past what a single subtask's per-key aggregation logic (parsing, state updates, checkpointing overhead) can sustain if the per-event processing cost is non-trivial. Confirm by checking per-subtask CPU utilization during the spike — if most subtasks are pegged near 100%, this is compute-bound, not I/O-bound.
• Checkpoint overhead growing under load. More events per interval means more state churn per checkpoint; if checkpoint duration grew during the spike (a metric this pipeline already tracks per Step 9), the job spends a larger fraction of wall-clock time checkpointing rather than processing, which shows up as lag even if raw compute isn't saturated. Confirm by correlating the lag curve against the checkpoint-duration metric's timeline — if they move together, this is the driver.
• Backpressure from a slow sink. If the revenue sink (the upsert-based real-time store) can't absorb writes fast enough at elevated throughput, Flink's backpressure mechanism throttles upstream operators to match, which shows up as reading lag even though the bottleneck is downstream. Confirm via Flink's backpressure metrics per operator — if backpressure is concentrated at the sink operator specifically, this is the driver, not insufficient source parallelism.

A single symptom (lag) with three plausible causes is exactly why this needs metrics correlated across the timeline, not a guess — fixing the wrong one (e.g., adding parallelism when the real bottleneck is a slow sink) burns effort and doesn't fix the SLA miss.

2. Fixes and their trade-offs

• If compute-bound: increasing partition count (and therefore max parallelism) is the structural fix, but it's disruptive — it reshuffles the key-to-partition mapping for every producer and consumer and can't be done casually mid-incident. The faster, safer lever is provisioning task-manager parallelism up to the existing 200-partition ceiling for known spike windows (campaign calendars are usually known in advance) rather than running at steady-state sizing year-round — trading idle compute cost during normal periods for headroom during predictable spikes.
• If checkpoint-bound: shortening the checkpoint interval doesn't help (more frequent checkpoints under load makes it worse); the fix is usually incremental checkpointing tuning (already in the base design) or checking whether state size has grown unexpectedly (a state TTL not being applied correctly would show up here) — this is a state-hygiene fix, not a scaling fix.
• If sink-bound: scale the sink's write capacity (more partitions/shards on the real-time store, or batching upserts) — this is the fix that a naive "just add more Flink parallelism" response would completely miss, since adding upstream parallelism into a throttled sink just moves the backpressure point, it doesn't remove it.

Each fix trades cost or complexity for headroom in a different place; picking the wrong one for the actual bottleneck wastes both.

3. Was sizing partitions off sustained peak, not campaign spike, wrong?

Not wrong, but it does mean campaign spikes are a known, accepted trade-off rather than an oversight, and this incident is exactly what that trade-off predicted. Sizing 200 partitions for the rarer 120,000 events/sec spike (rather than the 70,000 events/sec sustained peak) would mean carrying that headroom — and its cost — every single day for a burst that happens occasionally. The better fix isn't re-sizing partitions for the worst case that occurs rarely; it's making sure task-manager parallelism (which can scale up to, but not past, the partition ceiling) is provisioned dynamically for known spike windows, which gets the headroom exactly when needed without paying for it year-round. If spikes were frequent and unpredictable rather than occasional and campaign-scheduled, that calculus would shift toward provisioning more partitions up front — the right answer depends on how predictable and how frequent the spike pattern actually is, not on a fixed rule.