Design Autoscaling for an LLM Service: Signals, Thresholds and Cold Starts

An 8B model (16 GB in fp16) runs on single-H100 replicas of an open-source engine behind a gateway. Traffic is bursty: it can grow 10 percent per minute during a morning ramp, with an average prompt of about 2,000 tokens. A replica prefills about 30,800 tokens/s (assumed, 50 percent of peak bf16). A cold replica takes about 120 s from the scale-up decision to serving its first request (image pull, weight load, CUDA initialization, graph capture and warm-up). The team currently scales on average GPU utilization with a threshold of 70 percent and complains about TTFT spikes at 9 am.

1. Why is GPU utilization a bad signal for an LLM engine?
2. Propose the scaling signals and concrete thresholds, including a waiting-token budget derived from the prefill rate.
3. Account for the cold-start time and the traffic growth rate.
4. How do you scale down safely, and how do you route traffic so the scaling actually helps (prefix cache, long requests)?
5. What do you load-test and which dashboards prove it works?