Convolutional Networks & Vision Basics

This subject assumes Neural Networks & Backpropagation and Training Deep Networks: Optimizers, Schedules & Regularization (you know backprop, BatchNorm, residual connections as an optimization fix, and how to read a loss curve) and The PyTorch Training Loop from Scratch; it covers what is specific to images: convolution arithmetic, the classic CNN architectures, transfer learning, and the exact metrics for detection and segmentation. Vision Transformers and multimodal models live in Multimodal LLMs and Vision, generative vision has its own track (start with Text-to-Image Generation with Diffusion Models), and efficiency and compression get a pointer here and a full treatment in Model Compression, Quantization & On-Device ML.

Vision interviews reward two skills: doing the arithmetic on a whiteboard ("input 224, kernel 7, stride 2, padding 3 - what is the output size and how many parameters?") and knowing what a metric means ("what is mAP@[.5:.95] and why is accuracy meaningless for detection?"). Both are mechanical once you have the formulas, so this lesson states each formula once, verifies it by execution, and then drills the failure modes.

Why Convolution

Consider a small 32x32 RGB image (3,072 values) mapped to a 16-channel 32x32 feature map (16,384 values). A fully connected layer needs 3072 \times 16384 + 16384 = 50{,}348{,}032 parameters. A 3x3 convolution producing the same output shape needs 3 \cdot 9 \cdot 16 + 16 = 448. Three ideas make that possible:

• Locality. Each output depends only on a small neighbourhood (the kernel footprint). Pixels far apart are rarely related at the lowest level; edges, corners and textures are local patterns.
• Weight sharing. The same kernel is applied at every position, so one detector for "vertical edge" serves the whole image. Parameters do not grow with image size, only with kernel size and channel counts.
• Translation equivariance. Shifting the input shifts the feature map by the same amount: f(T_\Delta x) = T_\Delta f(x) (exactly, up to border effects; strides make it hold only for shifts that are multiples of the stride). Equivariance is not invariance. Invariance ("it is a cat wherever it is") comes later, from pooling or global average pooling followed by a classifier, and is only approximate.

These are inductive biases: assumptions baked into the architecture that make learning from limited data far more efficient when they are true of images, and a handicap when they are not (for example, for data where absolute position matters, such as a centred medical-scan atlas, or a tabular matrix reshaped into a grid). Deep-learning libraries compute cross-correlation (no kernel flip) and call it convolution; for learned kernels it makes no difference.