Distributed Training & GPU Systems

This subject assumes The PyTorch Training Loop from Scratch (single-GPU loop, mixed precision, checkpoints) and Training Deep Networks: Optimizers, Schedules & Regularization (Adam, batch size and learning rate), and uses the architecture from Transformers for AI Engineers. It extends the short "Distributed Training Basics" section of Model Training & Experimentation at Scale, which only answers "do I need it?"; here we answer how much memory, which parallelism, what communication, and why is it slow. Serving-side parallelism and the KV cache belong to LLM Inference Engineering, preference-tuning pipelines to Case Study: Designing an RLHF / Preference-Tuning Platform, and the training recipes themselves to Fine-Tuning: SFT, LoRA/QLoRA, RLHF and DPO and How LLMs Are Built: Pretraining to Chatbot.

Systems interviews for ML roles follow one script: "this model does not fit, or is too slow; walk me through it." The answer is always arithmetic first: bytes per parameter, FLOPs per token, bytes moved per step, and only then a strategy. Everything below that could run on a CPU box was executed (gloo backend, four processes, tiny models) and is marked "(verified)". Hardware figures (A100/H100 peak FLOPs and bandwidth, NVLink and InfiniBand speeds) are published spec values quoted from memory, not measured here; treat them as approximate and check the vendor datasheet before you put them in a design doc.