Practice — LLM Inference Engineering (6 questions)

You serve an 8B-class GQA model (32 layers, 32 query heads, 8 KV heads, head_dim 128, 8.03B parameters) in fp16 on a single 80 GB GPU with an engine that may use 90 percent of GPU memory. Assume about 3 GB of activation workspace and CUDA overhead. The product allows 32k-token contexts, but the median conversation is only about 2,500 tokens long. An engineer says "we get 12 concurrent users, so we need more GPUs".

1. Derive the KV cache size per token and per 32k request, and the weight memory.
2. How big is the KV pool, and how many tokens does it hold?
3. How many concurrent requests fit if the engine reserves the maximum length per request, and how many fit with paged allocation at the real average live context of 2,500 tokens? Why is the second number an upper bound rather than a promise?
4. What would the numbers be with full multi-head attention (32 KV heads), and with an fp8 KV cache?
5. At what batch-times-context product does reading the KV cache cost more HBM time per decode step than reading the weights?

You must serve a 405B-parameter dense model (126 layers, 8 KV heads, head_dim 128, hidden size 16,384) behind an interactive chat product with a TPOT target of about 40 ms. Hardware: nodes of 8 H100 80 GB GPUs with NVLink inside a node, slower InfiniBand between nodes. Use 3.35 TB/s HBM, 70 percent achievable bandwidth, 90 percent memory utilization and 4 GB of overhead per GPU (so 68 GB usable). All numbers are ASSUMPTION-labeled estimates.

1. Compute weight memory in fp16 and fp8, the KV bytes per token, and the KV size of an 8k request.
2. Evaluate three plans: fp8 with TP=8 on one node; fp16 with TP=8 and PP=2 on two nodes; fp16 with TP=16 across two nodes. For each give weights per GPU, the KV pool and the concurrency at 8k context.
3. Estimate the weight-read time per decode step and discuss what pipeline parallelism does and does not do for latency.
4. Explain the problem with TP=16 here.
5. Make a recommendation, including what you would check before committing to fp8.