Choose TP and PP to Serve a 405B Dense Model: One Node in fp8 or Two Nodes in fp16
You must serve a 405B-parameter dense model (126 layers, 8 KV heads, head_dim 128, hidden size 16,384) behind an interactive chat product with a TPOT target of about 40 ms. Hardware: nodes of 8 H100 80 GB GPUs with NVLink inside a node, slower InfiniBand between nodes. Use 3.35 TB/s HBM, 70 percent achievable bandwidth, 90 percent memory utilization and 4 GB of overhead per GPU (so 68 GB usable). All numbers are ASSUMPTION-labeled estimates.
- Compute weight memory in fp16 and fp8, the KV bytes per token, and the KV size of an 8k request.
- Evaluate three plans: fp8 with TP=8 on one node; fp16 with TP=8 and PP=2 on two nodes; fp16 with TP=16 across two nodes. For each give weights per GPU, the KV pool and the concurrency at 8k context.
- Estimate the weight-read time per decode step and discuss what pipeline parallelism does and does not do for latency.
- Explain the problem with TP=16 here.
- Make a recommendation, including what you would check before committing to fp8.
1. Memory. Weights: 405B x 2 B = 810 GB in fp16 and 405 GB in fp8. KV per token = 2 \times 126 \times 8 \times 128 \times 2 = 516{,}096 B (504 KiB); one 8k request is 4.23 GB. Neither model fits on one GPU in either precision, and 810 GB does not fit on one 8 x 80 GB node (640 GB total), so fp16 needs two nodes.
2. The three plans (usable 68 GB per GPU):
| Plan | GPUs | Weights/GPU | Free for KV/GPU | KV pool (tokens) | Concurrent 8k |
|---|---|---|---|---|---|
| fp8, TP=8 | 8 | 50.6 GB | 17.4 GB | 269,000 | about 33 |
| fp16, TP=8 x PP=2 | 16 | 50.6 GB | 17.4 GB | 539,000 | about 66 |
| fp16, TP=16 | 16 | 50.6 GB | 17.4 GB | 539,000 | about 66 (before KV replication, see 4) |
Each TP=8 GPU stores 1/8 of every layer and, with 8 KV heads, exactly one KV head of the cache. All plans leave only 17.4 GB per GPU for KV because the weights dominate: the fp8 plan has the least comfortable pool, and 33 concurrent 8k requests is a small batch, which limits throughput per node.
3. Latency. Each decode step must read each GPU's weights: 50.6 / (3350 \times 0.7) = 21.6 ms per stage (ideal at nominal bandwidth: 15.1 ms). TP=8 inside a node therefore gives a floor of about 21.6 ms plus communication: with two all-reduces per layer and 126 layers there are 252 collectives per token; at an assumed 20 microseconds each over NVLink that is about 5 ms, so TPOT is roughly 27 ms plus KV reads, inside the 40 ms target. Pipeline parallelism does not reduce the latency of one sequence: stages run in series, so PP=2 doubles the time a token spends traversing the weights to about 43 ms plus communication, which already breaks the 40 ms target at batch 1. PP helps capacity and throughput (twice the memory, more micro-batches in flight) and tolerates the slower link, because it sends only an activation of 32 KB per token across stages.
4. TP=16. All-reduces would run across the InfiniBand link, 252 times per token. If a cross-node collective costs about 80 microseconds (assumed), that is about 20 ms of pure communication per token on top of the 21.6 ms weight read, and it is on the critical path. TP=16 also does not divide 8 KV heads, so each KV head has to be replicated onto two GPUs, which halves the effective KV capacity gain. TP across nodes is the classic wrong turn; keep TP inside the NVLink domain.
5. Recommendation. If quality holds, serve fp8 with TP=8 on one node: lowest latency (about 27 ms plus KV), no cross-node traffic, half the GPUs. Scale throughput with data-parallel replicas, not by stretching one replica across nodes. Before committing, evaluate the fp8 model on your own tasks and long-context prompts and compare against fp16 (Model Compression, Quantization & On-Device ML), and consider an fp8 KV cache to double the 33-request pool. If fp16 is mandatory use TP=8 x PP=2 and accept about 43 ms plus comm at low load, which means the 40 ms target has to be renegotiated; PP is then a capacity decision, not a speed one. If throughput per dollar matters more than latency, run the batch pool in PP and the interactive pool on the fp8 TP=8 path.
Key points.
- Fit first: weights fp8 405 GB or fp16 810 GB; KV 504 KiB per token.
- TP inside NVLink, divides weights and KV reads, needs TP to divide the KV heads; TP across nodes puts all-reduces on a slow link.
- PP adds capacity, not single-request speed (stages are serial).
- Weight-read floor per step: 21.6 ms for 50.6 GB per GPU.
- Scale out with replicas; validate fp8 quality before choosing it.
Share this question