Will a 13B Full Fine-Tune Fit? Memory Estimate Across Strategies
A team wants to fully fine-tune a 13B-parameter decoder model with
AdamW in bf16 mixed precision (fp32 master weights and fp32 Adam
moments). The only hardware available is a node with 4 x 80 GB GPUs
(NVLink). An engineer launches plain DistributedDataParallel and
reports "CUDA out of memory before the first step finishes". Another
suggests "just buy 8-bit Adam".
- Compute the static training state (weights, gradients, master weights, optimizer moments) per parameter and in total for 13B. Which part dominates?
- Compute the per-GPU static memory for DDP, ZeRO-1, ZeRO-2 and ZeRO-3/FSDP on 4 GPUs. Which strategies leave room for activations on an 80 GB card?
- What does 8-bit optimizer state buy, and does it change the conclusion? What does LoRA change instead?
- After sharding, activations still do not fit at the desired micro-batch. List the levers in the order you would apply them and the cost of each.
- State the communication consequence of your chosen strategy relative to DDP.
1. Static state. Mixed-precision Adam costs 2 + 2 + 4 + 4 + 4 = 16 bytes per parameter: bf16 weights (2), bf16 gradients (2), fp32 master weights (4), Adam m (4) and v (4). For \Psi = 13 \times 10^9 that is 16 \times 13 = 208 GB, against 26 GB for just the bf16 weights (verified arithmetic; a real fp32 AdamW step measures 4 + 4 + 8 = 16 B/param). Optimizer state dominates: 12 of the 16 bytes (75 percent) are master weights plus moments.
2. Strategies on N = 4 (with K = 12 bytes of optimizer state per parameter):
| Strategy | Bytes/param/GPU | Per GPU for 13B |
|---|---|---|
| DDP | 16 | 208 GB |
| ZeRO-1 (shard optimizer state) | 4 + 12/4 = 7 | 91 GB |
| ZeRO-2 (+ gradients) | 2 + 14/4 = 5.5 | 71.5 GB |
| ZeRO-3 / FSDP (+ parameters) | 16/4 = 4 | 52 GB |
DDP fails immediately (208 GB against 80 GB: the "OOM before the first step" is the optimizer state being allocated). ZeRO-1 does not fit either (91 GB). ZeRO-2 fits statically (71.5 GB) but leaves under 9 GB for activations and temporary buffers, which is not enough in practice. Only full sharding (FSDP/ZeRO-3, 52 GB) leaves about 28 GB for activations, plus the transient all-gathered layer (a few hundred MB). So the answer is FSDP with per-block wrapping.
3. 8-bit optimizer and LoRA. Block-wise 8-bit moments shrink m and v from 8 bytes to about 2, so the total becomes 2 + 2 + 4 + 2 = 10 B/param, 130 GB for 13B: it helps (7B-class models become comfortable) but still does not fit a single 80 GB GPU, so it does not replace sharding; it combines with it (4 GPUs: 32.5 GB each). LoRA attacks a different term: the base weights are frozen, so there are no gradients, master weights or moments for them. The base model stays in bf16 (26 GB) plus a small adapter state, which would fit on one 80 GB GPU, but it is a different training method with a different quality profile (Fine-Tuning: SFT, LoRA/QLoRA, RLHF and DPO), not a memory trick for the same full fine-tune.
4. Activation levers, in order. (a) FlashAttention: removes the stored s \times s score matrices (the 5as/h term), nearly free, should always be on. (b) Lower the micro-batch and accumulate (global batch unchanged, divide each loss by the accumulation count): free in FLOPs but small kernels and, with FSDP, repeated weight gathers per micro-batch. (c) Activation checkpointing per block: stores only layer inputs, cutting activation memory by an order of magnitude, at the cost of one extra forward pass, about 33 percent more FLOPs (HFU 4/3 of MFU). (d) Shorter sequences or packing, (e) CPU offload as a last resort (PCIe makes steps several times slower).
5. Communication. DDP sends 2\Psi per step (ring all-reduce: 2(N-1)/N of the gradient buffer per GPU); FSDP/ZeRO-3 all-gathers weights in the forward and again in the backward and reduce-scatters gradients: three collectives, about 1.5x DDP (for bf16 13B on 4 GPUs, 3 \times 3/4 \times 26 = 58.5 GB sent per GPU per step against 39 GB). Inside an NVLink node and with prefetch this hides behind compute; over slower links it can be exposed, which is when hybrid sharding or tensor parallelism enters.
Share this question