Training Deep Networks: Optimizers, Schedules & Regularization

This subject assumes Neural Networks & Backpropagation (you can compute gradients for an MLP and know why initialization and activations matter); it covers what happens after the gradient exists: which update rule to apply, how big a step to take, how to stabilize and regularize training, and how to read a loss curve. Mixed precision and the PyTorch loop mechanics are handled in PyTorch Training Loop from Scratch, distributed and memory-level details in Distributed Training & GPU Systems, and generic hyperparameter-search theory in Model Training & Experimentation. Here the focus is the DL-specific recipe.

Interviewers rarely ask you to recite Adam. They ask "your loss is flat at 2.30 — what do you check?", "why AdamW and not Adam with L2?", "why does BatchNorm behave differently at inference?", and "how did you choose the learning rate?". Each has a short, mechanistic answer, and this lesson builds those answers from the update rules up.

The Optimization Landscape

A neural-network loss is non-convex in the weights: there is no unique global minimum to roll into, and the result depends on init, data order and hyperparameters. Two facts calm the worry:

• Saddle points, not bad local minima, dominate high dimensions. At a critical point with d parameters, being a true local minimum needs all d Hessian eigenvalues positive; in a large net almost every critical point has some negative directions, so it is a saddle. The real danger is the plateau around a saddle where the gradient is tiny and progress crawls. Momentum and adaptive methods help escape. In wide, overparameterized networks, the local minima SGD reaches tend to have similar training loss.
• Minibatch noise is a feature. A minibatch gradient is an unbiased but noisy estimate of the full gradient, with standard deviation shrinking as 1/\sqrt{B}. That noise lets SGD bounce out of sharp basins and acts as implicit regularization; it also means the loss curve is jagged by construction.

Batch size. Going from B = 4 to B = 32 cuts gradient noise by \sqrt{8} \approx 2.8\times, but costs 8x the compute per step. Past a problem-dependent critical batch size, bigger batches stop reducing the number of steps needed, so you burn hardware for no wall-clock gain. Distinguish the two units of progress: an epoch is one pass over the data, a step is one parameter update. With 51,200 examples, B = 256 gives 200 steps per epoch and B = 1024 gives only 50; the larger batch makes 4x fewer updates per epoch, so at the same learning rate it learns less per epoch. This is the root of the linear scaling heuristic: when multiplying batch size by k, multiply the learning rate by k (SGD with momentum; typically with warmup), e.g. batch 256 at \eta = 0.1 becomes batch 1024 at \eta = 0.4. For Adam-family optimizers the square-root rule (\eta \to \sqrt{k}\,\eta) is the common heuristic; both break down beyond the critical batch size, so treat them as starting points to verify, not laws.