Practice — Distributed Training & GPU Systems (6 questions)

A team wants to fully fine-tune a 13B-parameter decoder model with AdamW in bf16 mixed precision (fp32 master weights and fp32 Adam moments). The only hardware available is a node with 4 x 80 GB GPUs (NVLink). An engineer launches plain DistributedDataParallel and reports "CUDA out of memory before the first step finishes". Another suggests "just buy 8-bit Adam".

1. Compute the static training state (weights, gradients, master weights, optimizer moments) per parameter and in total for 13B. Which part dominates?
2. Compute the per-GPU static memory for DDP, ZeRO-1, ZeRO-2 and ZeRO-3/FSDP on 4 GPUs. Which strategies leave room for activations on an 80 GB card?
3. What does 8-bit optimizer state buy, and does it change the conclusion? What does LoRA change instead?
4. After sharding, activations still do not fit at the desired micro-batch. List the levers in the order you would apply them and the cost of each.
5. State the communication consequence of your chosen strategy relative to DDP.

You must fine-tune (continued-pretrain) a 70B dense transformer (L = 80, h = 8192, sequence length 4096) on 64 H100 80 GB GPUs: 8 nodes of 8 GPUs, NVLink inside each node, about 50 GB/s of InfiniBand per GPU between nodes. Global batch: 4,194,304 tokens per step. AdamW, bf16 mixed precision, FlashAttention.

1. Show that DDP is impossible and compute the static state per GPU if everything is sharded across all 64 GPUs.
2. Estimate the step time at 40 percent MFU and the FSDP communication per GPU per step. Is communication likely to be hidden?
3. Evaluate a 2D alternative: tensor parallelism 8 inside each node with FSDP/DP 8 across nodes. What are its communication costs and where do they run?
4. Recommend a plan, say what you would measure first, and state the conditions under which you would change it (smaller global batch, slower network, 512 GPUs).