Debug p99 TTFT Spikes on an LLM Service: Queueing, Long Prefills, Cache Misses or Preemption

A chat service on an 8B model (single-H100 replicas, continuous batching, chunked prefill disabled) shows p50 TTFT 180 ms and p99 TTFT 6 s. Every few minutes TPOT p99 also jumps to over a second for all users on the replica. Since last week a new enterprise customer sends document-QA requests with prompts up to 100k tokens. Prefix-cache hit rate fell from 62 percent to 31 percent. Preemption counters are rising. Replicas autoscale on GPU utilization.

1. Compute how long a 32k-token and a 100k-token prefill block a replica (assume 8B on one H100 at 50 percent of 989 TFLOPs, including the quadratic attention term for a 32-layer model with hidden size 4,096), and what that does to co-running decoders.
2. List the hypotheses (at least five) and the metric that separates each.
3. Which fixes address which hypotheses? Give a chunk size using a 60 ms inter-token gap SLO and a decode step of 14 ms.
4. How do you stop one customer's long documents from degrading everyone?
5. What do you add to monitoring so the next spike is diagnosed in minutes?