The PyTorch Training Loop from Scratch

This subject assumes Neural Networks & Backpropagation and Training Deep Networks: Optimizers, Schedules & Regularization (you know what a gradient step, AdamW, a warmup schedule and gradient clipping are); here you see how they appear in code and what silently goes wrong around them. It is strictly single-GPU: multi-GPU training lives in Distributed Training & GPU Systems, experiment-tracking tools in Experiment Tracking & Model Registries, and containerized environments in Containers & Reproducible ML Environments.

"Write a training loop on the whiteboard" is a standard ML-engineer screen, and the follow-ups are where candidates separate: "why model.eval() and also no_grad?", "what do you put in a checkpoint so training resumes exactly?", "your GPU is 30% utilized, what do you check?", "what does GradScaler do?". Most real-world training bugs do not crash; they produce a plausible but wrong number. This lesson is organized around the places those bugs live.