Count the Parameters of an MLP
A tabular model takes 20 input features, has two hidden layers of 64 ReLU units each, and a single sigmoid output unit for binary classification. All layers are dense with biases.
- How many learnable parameters does it have? Show the per-layer counts.
- How much memory do the weights occupy in float32? A colleague says the training memory will be "about the same, since the parameters are all that matter" — what is wrong with that?
- If you widen the first hidden layer to 128 units (keeping everything else), what is the new parameter count? Does doubling a single layer's width double the model's parameters?
1. Parameter count
Use (n_{\ell-1} + 1) \cdot n_\ell for each dense layer (the +1 is the bias per output unit):
| Layer | Shape | Count |
|---|---|---|
| Input to hidden 1 | (20 + 1) \times 64 | 1,344 |
| Hidden 1 to hidden 2 | (64 + 1) \times 64 | 4,160 |
| Hidden 2 to output | (64 + 1) \times 1 | 65 |
| Total | 5,569 |
The activation functions (ReLU, sigmoid) and the batch size add no parameters.
2. Memory
5,569 parameters \times 4 bytes = 22{,}276 bytes, about 22 KB.
The colleague is wrong about training memory: besides the weights,
training stores one gradient per parameter (same size again), whatever
state the optimizer keeps per parameter, and, usually dominating, the
forward activations of every layer for the whole batch, because
the backward pass needs them to compute weight gradients and
\phi'. Activation memory scales with batch size and depth, not with
parameter count. That is also why inference under no_grad needs far
less memory than training.
3. Widening the first hidden layer
Layer 1: (20 + 1) \times 128 = 2{,}688. Layer 2 now has 128 inputs: (128 + 1) \times 64 = 8{,}256. Output: 65. Total 2{,}688 + 8{,}256 + 65 = 11{,}009. Doubling one layer's width roughly doubled the model (5,569 to 11,009, a factor of 1.98), because widening a layer grows both the matrix feeding it and the matrix leaving it. In general a layer's cost is the product of its neighbours' widths, so parameters scale quadratically if you widen everything.
Share this question