Capacity Plan for a Traffic Profile: GPUs for 40 Requests per Second

A support product serves an 8B GQA model (32 layers, 8 KV heads, head_dim 128, fp16 weights 16.06 GB) on H100 80 GB GPUs. Per request: 1,200 prompt tokens, of which 800 are a shared system prompt that the engine can cache, and 300 output tokens. Peak: 40 requests per second. SLO: TPOT at most 40 ms. GPU price assumed USD 2.50 per hour. Use these ASSUMPTION parameters: 3.35 TB/s HBM at 70 percent efficiency, 989 TFLOPs bf16 at 50 percent efficiency, 90 percent memory utilization, 3 GB overhead, 1 ms of per-step engine overhead, a 15 percent KV safety margin and 70 percent target utilization at peak.

1. Compute the KV pool, the concurrency the pool supports at the average live context, and the decode step time as a function of the batch.
2. Pick the operating batch, show whether the KV pool or the TPOT SLO binds, and compute TPOT.
3. Compute node-seconds per request with and without the cached system prompt, and the requests per second per GPU.
4. How many GPUs do you need at peak, with one spare? What changes if the prefix hit rate is only 50 percent?
5. Compute the cost per 1M output tokens at 100 percent and 50 percent utilization, and list what you would validate with a real load test.