How Many 32k-Context Chats Fit on One 80 GB GPU? KV Cache Sizing for an 8B Model
You serve an 8B-class GQA model (32 layers, 32 query heads, 8 KV heads, head_dim 128, 8.03B parameters) in fp16 on a single 80 GB GPU with an engine that may use 90 percent of GPU memory. Assume about 3 GB of activation workspace and CUDA overhead. The product allows 32k-token contexts, but the median conversation is only about 2,500 tokens long. An engineer says "we get 12 concurrent users, so we need more GPUs".
- Derive the KV cache size per token and per 32k request, and the weight memory.
- How big is the KV pool, and how many tokens does it hold?
- How many concurrent requests fit if the engine reserves the maximum length per request, and how many fit with paged allocation at the real average live context of 2,500 tokens? Why is the second number an upper bound rather than a promise?
- What would the numbers be with full multi-head attention (32 KV heads), and with an fp8 KV cache?
- At what batch-times-context product does reading the KV cache cost more HBM time per decode step than reading the weights?
1. Sizes. KV bytes per token = 2 \times L \times H_{kv} \times d_{head} \times \text{bytes} = 2 \times 32 \times 8 \times 128 \times 2 = 131{,}072 B (128 KiB). One 32k request is 131{,}072 \times 32{,}768 = 4.29 GB. Weights in fp16: 8.03 \times 10^9 \times 2 = 16.06 GB. (The factor 2 is keys plus values, H_{kv} counts KV heads and not query heads, and the final 2 is bytes per fp16 element.)
2. Pool. Usable memory is 0.9 \times 80 = 72 GB. Subtract weights and workspace: 72 - 16.06 - 3 = 52.94 GB of KV pool. Divided by 128 KiB per token that holds about 403,900 tokens. (These come from a back-of-envelope script with ASSUMPTION-labeled overheads; a real engine reports its own pool size at startup, which you should read from the log.)
3. Concurrency.
| Allocation | Tokens reserved per request | Concurrent requests |
|---|---|---|
| Reserve 32k (max length) | 32,768 | 403{,}900 / 32{,}768 = 12.3, so 12 |
| Paged, real average live context 2,500 | 2,500 | 403{,}900 / 2{,}500 = 161.6, about 161 |
| Paged at the 1,500-token live context of mid-conversation | 1,500 | about 269 |
The "12 concurrent users" is exactly the max-length reservation result: the engineer sized for the worst case on every request. Paging (PagedAttention) allocates 16-token blocks on demand, so the pool holds live tokens instead of reserved tails. The 161 is an upper bound because (a) it assumes no fragmentation beyond the last partial block (each sequence wastes up to 15 tokens, negligible here), (b) live length keeps growing as sequences decode, so the engine must leave headroom or preempt a sequence when the pool is full (recompute or swap, which shows up as a tail-latency spike), and (c) a few 32k conversations arriving together can still exhaust the pool: 12 of them take all of it. The practical plan is a 10 to 15 percent margin plus an admission limit on very long contexts, so about 140 concurrent sequences, not 161.
4. Counterfactuals. Full MHA with 32 KV heads has 4x the bytes per token (524,288 B, 512 KiB): the pool holds about 101,000 tokens, which is 12.3 concurrent requests at 8k instead of 49. GQA with 8 KV heads is the reason a modern model fits at all. An fp8 KV cache halves the bytes per token to 64 KiB: 807,800 tokens in the pool, or 98.6 concurrent 8k requests instead of 49.3. The cost is a possible accuracy loss that must be evaluated on your own long-context tasks (LLM Observability and Evaluation), not assumed to be zero.
5. When KV reads rival weight reads. Each decode step reads the weights once (16.06 GB) and every sequence's KV cache. The KV read equals the weight read when b \times c \times 131{,}072 = 16.06 \times 10^9, i.e. b \times c = 122{,}500 tokens in flight: for example 64 sequences at 1,915 tokens, or 15 sequences at 8k. Beyond that point the step time grows linearly with batch times context, so a bigger batch is no longer almost free, and long contexts are expensive even when they fit. This is also why GQA speeds decode and not only memory.
Key points.
- KV per token = 2 L H_{kv} d_{head} \times bytes; use KV heads.
- Pool = usable memory minus weights minus workspace; concurrency = pool tokens divided by tokens per request.
- Max-length reservation gives 12; paged allocation gives about 160 (140 with margin); the gap is fragmentation and reservation waste.
- GQA (4x here) and KV quantization (2x) multiply the capacity.
- Weight read and KV read are comparable at about 122k tokens in flight.
Share this question