Neural Networks & Backpropagation
This subject assumes you already know Logistic Regression and Linear Regression (a linear score pushed through a loss, fitted by gradient descent); it stops before optimizers and regularization (how to train deep nets well is the next subject) and before any specific architecture such as CNNs or transformers. What it teaches is the one mechanism every one of those architectures reuses: a stack of simple differentiable layers, trained by computing the gradient of a scalar loss with respect to every parameter using the chain rule, applied in reverse. If you can derive that by hand and explain why each piece is there, the rest of deep learning is variations on a theme.
The interview framing is blunt. "Walk me through backprop," "why do we need activation functions," "why not initialize to zero," "what's the gradient of softmax cross-entropy," and "why does CrossEntropyLoss want logits" are asked nearly verbatim at ML engineer and applied scientist loops. The goal here is that none of them requires memorized phrases: each follows from the same few facts.
Why Stacking Linear Models Fails
Logistic regression is already a one-neuron network: \hat y = \sigma(w^\top x + b). Its decision boundary is a hyperplane, so it cannot represent XOR or any pattern where classes are not linearly separable in the input space. The tempting fix is "stack more of them":
Substitute the first line into the second:
The composition of affine maps is affine. Ten linear layers are exactly as expressive as one, just with a needlessly factored (and possibly rank-limited) weight matrix. The nonlinearity between layers is the whole point: it is what stops the stack from collapsing and lets depth add expressive power.
Insert an elementwise nonlinearity \phi between layers and the picture changes. The universal approximation theorem says that a network with a single hidden layer and a suitable non-polynomial activation can approximate any continuous function on a compact domain to arbitrary accuracy, given enough hidden units. Two honest caveats belong in any interview answer: the theorem is an existence result (it says nothing about how many units are needed, which can be astronomically large, or whether gradient descent will find the weights), and it is depth, not width, that makes representations compositional and parameter-efficient in practice. State it; do not claim it explains why deep nets work.