Training Deep Networks: Optimizers, Schedules & Regularization
How to make a network actually converge, and how to read a loss curve
Make deep networks converge and generalize: the non-convex landscape, saddle points and minibatch noise, batch-size effects and the linear-scaling rule, SGD with momentum and Nesterov, RMSProp, Adam with bias correction and AdamW's decoupled weight decay, learning-rate warmup and cosine/one-cycle schedules with the LR range test, BatchNorm/LayerNorm/RMSNorm/GroupNorm and pre- vs post-norm, residual connections, weight decay, dropout, label smoothing, augmentation and early stopping, gradient clipping and accumulation, a NaN checklist, a loss-curve diagnosis table, random-search hyperparameter tuning, and a worked example with a hand-computed AdamW step and four diagnosed training runs.
Practice questions (6)
-
View →
Explain Adam's Bias Correction With Numbers
Intermediate · Free -
View →
Why AdamW, Not Adam With L2?
Intermediate · Free -
View →
Choose an Optimizer and Schedule for a Fine-Tuning Job
Intermediate -
View →
Diagnose Three Loss Curves
Intermediate -
View →
A BatchNorm Bug Between Training and Serving
Intermediate