Shapes, Parameters and Receptive Field of a ResNet-Style Stem
You are reviewing a teammate's PyTorch model for 3x224x224 images and 100 classes. They claim "the stem has a receptive field of about 20 pixels, so we need dilation to see more context". The layers are:
| # | Layer |
|---|---|
| A | Conv 7x7, 3 to 64, stride 2, padding 3, bias=False |
| B | MaxPool 3x3, stride 2, padding 1 |
| C | Conv 3x3, 64 to 64, stride 1, padding 1, bias=False, then BatchNorm |
| D | Conv 3x3, 64 to 128, stride 1, padding 2, dilation 2, bias=False |
| E | Conv 3x3, 128 to 256, stride 2, padding 1, bias=False |
| F | Global average pool, then Linear 256 to 100 |
- Compute the output shape after every layer, using the output-size formula.
- Count the parameters per layer and in total (count BatchNorm's trainable scale and shift).
- Compute the theoretical receptive field after each of A-E with the recurrence, and say whether the teammate's claim is right. What would the RF after E be if D used dilation 1 and padding 1?
- How many multiply-accumulates does layer D cost? Compare the parameter count of layer A with the size of its output (activations); what does that imply for training memory?
1. Shapes with n_{\text{out}} = \lfloor (n + 2p - d(k-1) - 1)/s \rfloor + 1:
| Layer | Computation | Output |
|---|---|---|
| A | \lfloor (224 + 6 - 6 - 1)/2 \rfloor + 1 = 112 | 64 \times 112 \times 112 |
| B | \lfloor (112 + 2 - 2 - 1)/2 \rfloor + 1 = 56 | 64 \times 56 \times 56 |
| C | same padding, s=1 | 64 \times 56 \times 56 |
| D | k^{\text{eff}} = 2 \cdot 2 + 1 = 5, \lfloor (56 + 4 - 4 - 1)/1 \rfloor + 1 = 56 | 128 \times 56 \times 56 |
| E | \lfloor (56 + 2 - 2 - 1)/2 \rfloor + 1 = 28 | 256 \times 28 \times 28 |
| F | GAP, then FC | 256, then 100 |
Padding 2 with dilation 2 is the "same" padding for a dilated 3x3 conv (p = d(k-1)/2). These shapes match running the model in PyTorch.
2. Parameters (k^2 C_{in} C_{out}, no biases):
- A: 7 \cdot 7 \cdot 3 \cdot 64 = 9{,}408
- B: 0
- C: 9 \cdot 64 \cdot 64 = 36{,}864, plus BatchNorm 2 \cdot 64 = 128 (the running mean and variance are buffers, not parameters)
- D: 9 \cdot 64 \cdot 128 = 73{,}728 (dilation adds no parameters)
- E: 9 \cdot 128 \cdot 256 = 294{,}912
- F: 256 \cdot 100 + 100 = 25{,}700
Total: 9{,}408 + 36{,}864 + 128 + 73{,}728 + 294{,}912 + 25{,}700 = 440{,}740.
3. Receptive field. Start with r_0 = 1, j_0 = 1 and use r_l = r_{l-1} + (k^{\text{eff}}_l - 1) j_{l-1}, j_l = j_{l-1} s_l:
| Layer | k^{\text{eff}} | s | r | j |
|---|---|---|---|---|
| A | 7 | 2 | 1 + 6 \cdot 1 = 7 | 2 |
| B | 3 | 2 | 7 + 2 \cdot 2 = 11 | 4 |
| C | 3 | 1 | 11 + 2 \cdot 4 = 19 | 4 |
| D | 5 | 1 | 19 + 4 \cdot 4 = 35 | 4 |
| E | 3 | 2 | 35 + 2 \cdot 4 = 43 | 8 |
The teammate is looking at the RF after C (19, "about 20"), but the network continues: after E it is 43 pixels, on a 224-pixel input, and the GAP head then pools 28x28 such windows. Without the dilation (kernel 3, d=1, p=1) layer D would add 2 \cdot 4 = 8 instead of 16, so r_D = 27 and r_E = 35: dilation buys 8 more pixels of RF for zero extra parameters. Whether it is needed depends on the task: 43 pixels is tiny relative to a 224-pixel image, but the full network continues with more stages. The theoretical RF also overstates the effective RF, which concentrates near the centre, so for tasks needing global context (segmentation) people add dilation, pyramid pooling or attention rather than relying on depth alone.
4. Compute and memory. Layer D costs 56 \cdot 56 \cdot 64 \cdot 128 \cdot 9 = 231{,}211{,}008 MACs (about 0.23 GMACs, 0.46 GFLOPs). Layer A has 9,408 parameters but its output has 64 \cdot 112 \cdot 112 = 802{,}816 values (3.06 MiB per image in fp32): about 85 activation values per weight. Backprop stores activations, so early high-resolution layers dominate training memory (and compute) even though they hold almost no parameters. The levers are lower resolution, smaller batch, mixed precision, gradient accumulation and activation checkpointing.
Share this question