Distributed Training & GPU Systems
Memory math, parallelism strategies, FlashAttention and how to tell why your cluster is slow
Plan, size and debug multi-GPU training, and defend each choice in an interview: the GPU mental model (SMs, HBM versus on-chip SRAM, arithmetic intensity and the roofline with measured intensities for a GEMM, an elementwise add and a batch-1 matvec, tensor cores, and MFU as the honest utilization metric computed from the 6ND rule), memory math (16 bytes per parameter for mixed-precision Adam, verified by measuring a real AdamW step, optimizer-state dominance, activation memory and activation checkpointing with measured saved-tensor and recompute counts, a 7B estimate), mixed precision (fp16 versus bf16 versus fp8 ranges, master weights, loss scaling, all demonstrated numerically), data parallelism (DDP, ring all-reduce byte count verified by simulation as 2(N-1)/N of the buffer, gradient bucketing and overlap, gradient accumulation equivalence verified, global batch and learning-rate scaling, SyncBatchNorm statistics verified, replica consistency verified on a multi-process gloo run), sharded data parallelism (ZeRO stages 1 to 3 and FSDP with per-GPU memory for 7B on 8 GPUs, the 1.5x communication cost, CPU offload, a ZeRO-1 emulation and PyTorch FSDP2 checked for equivalence with single-process training), model parallelism (Megatron column and row tensor parallelism verified numerically, pipeline parallelism with the bubble fraction (p-1)/(m+p-1) verified by schedule simulation and 1F1B memory, sequence and context parallelism, expert parallelism, 3D parallelism and decision rules by model size and interconnect), FlashAttention (IO-aware exact attention, tiling and online softmax verified against naive attention, O(N) memory, why it is faster despite more FLOPs, versions 2 and 3), interconnect and topology (NVLink, PCIe, InfiniBand, NCCL, why placement matters), throughput debugging (profile first, dataloader starvation, stragglers simulated, communication overlap, checkpoint cost and the Young-Daly interval, elastic training, spot instances), scaling laws (Chinchilla and what it means for budgeting fine-tunes versus pretraining), and a full worked plan for fine-tuning a 7B model on 8x80 GB with memory per strategy, micro-batch and global batch, step time and MFU.
Practice questions (6)
-
View →
Will a 13B Full Fine-Tune Fit? Memory Estimate Across Strategies
Intermediate · Free -
View →
Choose a Parallelism Plan for Training a 70B Model on 64 GPUs
Advanced · Free -
View →
ZeRO-3/FSDP versus Tensor Parallelism: What Is Sharded and What Moves
Advanced -
View →
Explain Why FlashAttention Is Faster Despite Doing More FLOPs
Advanced -
View →
Diagnose a 32-GPU Job at 28 Percent MFU
Advanced