Intermediate
Open
Pro
Diagnose a Network Whose Hidden Activations Are All Zero
You train a 10-class MLP (balanced classes, ReLU hidden layers). After raising the learning rate from 10^{-3} to 1.0 for a "fast experiment," the loss jumps up on the first few steps, then flatlines at 2.303 and never moves. You print the activations of hidden layer 1 for a validation batch and every value is exactly 0.
- What is 2.303 and why does the loss flatline there?
- Explain the mechanism: why does the network not recover after the large step?
- How would you confirm the diagnosis in code, and what would you change? Give at least three fixes.
Share this question