GPU Memory Needed to Train an LLM
GPT-2 XL has 1.5 B parameters. Its weights in FP16 take ~3 GB on disk.
Roughly how much GPU memory is needed to train it (full fine-tuning, standard mixed-precision Adam) on a single GPU?
Show the per-parameter memory breakdown that gets you to your answer, and name two techniques that would bring the requirement down.
D) 40 GB — and the reason is that weights are the smallest part of training memory.
Inference needs only the weights: 1.5 B × 2 bytes (FP16) ≈ 3 GB. Training with standard mixed-precision Adam holds, per parameter:
| Tensor | Precision | Bytes/param |
|---|---|---|
| Weights (working copy) | FP16 | 2 |
| Gradients | FP16 | 2 |
| Master weights | FP32 | 4 |
| Adam first moment (m) | FP32 | 4 |
| Adam second moment (v) | FP32 | 4 |
| Total (model states) | 16 |
1.5 B × 16 bytes ≈ 24 GB just for model states — 8× the FP16 weight size — before a single activation is stored. Activations for the backward pass scale with batch × sequence length × layers × hidden size and easily add several more GB (more without activation checkpointing), plus CUDA context and allocator fragmentation. That lands in the ~30–40 GB range, so D) 40 GB is the only option that is actually sufficient; A/B/C can't even hold the optimizer states.
The rule of thumb to say out loud: ~16 bytes per parameter for mixed-precision Adam training (≈ 8× the FP16 weights), plus activations. Pure FP32 training without mixed precision is similar (4 + 4 + 4 + 4 = 16 bytes) — the FP16 working copies save compute, not much memory.
Two ways to bring it down:
- Parameter-efficient fine-tuning (LoRA / QLoRA) — freeze the base weights so gradients and optimizer states exist only for the small adapter matrices; QLoRA additionally quantises the frozen base to 4-bit. Training GPT-2 XL then fits comfortably in < 10 GB.
- 8-bit optimizers (e.g. bitsandbytes Adam8bit) — cut the two Adam states from 8 bytes/param to ~2, taking model states from 16 → ~10 bytes/param.
- Others worth naming: gradient/activation checkpointing (trade recompute for activation memory), smaller micro-batches with gradient accumulation, and ZeRO/FSDP sharding of optimizer states across GPUs when more than one GPU is available.
Share this question