Explain Residual Blocks and the Bottleneck's Parameter Savings
A junior engineer proposes replacing the ResNet-50 stage-2 blocks (256-channel input and output) with a plain stack of two 3x3 convs at 256 channels "because the bottleneck's 1x1 convs look like wasted depth". Another says "skip connections are just there for the gradients". You need to settle both.
- Write out the bottleneck block (256 to 64 to 64 to 256) and compute its weight count (ignore BatchNorm and biases). Compare with two 3x3 convs at 256 channels. How many times smaller is it?
- In a stage transition the input has 256 channels at 56x56 and the output must have 512 channels at 28x28. The identity shortcut cannot be used. What replaces it, how many weights does it have, and why is it still compatible with the residual idea?
- Explain what the skip connection does beyond helping gradients, referring to the degradation problem, and say what initializing the last BatchNorm scale of each residual branch to zero achieves.
- When would you still prefer the basic (two 3x3) block?
1. Weight counts
Bottleneck: 1 \times 1 reduce, 3 \times 3 at low width, 1 \times 1 restore, summed with the shortcut (y = x + F(x), then ReLU):
- 1x1, 256 to 64: 256 \cdot 64 = 16{,}384
- 3x3, 64 to 64: 9 \cdot 64 \cdot 64 = 36{,}864
- 1x1, 64 to 256: 64 \cdot 256 = 16{,}384
- Total: 69,632
Two 3x3 convs at 256 channels: 2 \cdot 9 \cdot 256 \cdot 256 = 1{,}179{,}648. The bottleneck is 1{,}179{,}648 / 69{,}632 \approx 16.9 times smaller, and it has three nonlinearities rather than two. The 1x1 convs are not wasted depth: they compress channels so that the expensive spatial 3x3 operates on 64 instead of 256 channels (cost scales with C_{in} C_{out}, i.e. quadratically in width), then expand back so the block can be added to the 256-channel shortcut. That saved budget is what lets ResNet-50/101/152 go much deeper than ResNet-34 for moderate extra cost.
2. Projection shortcut
The shortcut becomes a 1 \times 1 conv with stride 2 from 256 to 512 channels (usually followed by BatchNorm): 256 \cdot 512 = 131{,}072 weights (1,024 BN parameters). It halves the spatial size and changes the width so the addition's shapes match. The block is still residual: the branch learns a correction on top of a (now linear, learned) path rather than a full transformation, and the gradient has a short path through the 1x1 conv instead of only through the branch. Only the first block of each stage needs it; the rest use identity shortcuts with no parameters.
3. More than gradients
The degradation problem (He et al., 2015) was that a plain 56-layer CNN had higher training error than a 20-layer one, which cannot be overfitting. A deeper network can represent the shallow one by learning identity layers, but plain stacks find that hard to optimize. With a skip connection the identity is the default: pushing the branch weights toward zero yields y = x. Hence "just for the gradients" is only half the story: the identity term I in \partial y/\partial x = I + \partial F/\partial x gives gradients a path that bypasses the Jacobian product (see Training Deep Networks), and it makes extra layers harmless instead of harmful, so depth improves accuracy. Zeroing the last BN scale \gamma in each branch makes every block the exact identity at initialization: the network starts as a shallow one and each block learns to contribute, which stabilizes early training (especially with large batches).
4. Basic block
Use it at shallow depths (ResNet-18/34), on small inputs or low-channel stages, where the 3x3 convs are cheap anyway and the bottleneck's 1x1 reduce could starve capacity; also where the simpler layout and better hardware utilization (fewer small kernels) matter more than parameter count. At 64 channels the basic block has 2 \cdot 9 \cdot 64 \cdot 64 = 73{,}728 weights, so there is little to save.
Share this question