Why Zero Initialization Fails, and What to Do Instead

A junior engineer initializes every weight and bias in a 2-layer MLP to 0 "so training is deterministic and unbiased," and reports that the model learns nothing beyond predicting the class prior.

1. Explain why this happens for a ReLU hidden layer. Trace what the gradients are on the first step.
2. Suppose instead they set all weights to the same small constant such as 0.1. Do things get better? Why or why not?
3. Propose a fix, state the initialization scheme for a ReLU network and its variance, and give the standard deviation for a layer with 256 inputs.