Deep Learning Foundations

Neural Networks & Backpropagation

Build a neural network from the ground up: why stacked linear layers collapse and nonlinearity is the point, the neuron and layer with shapes and parameter counting, activation functions (sigmoid, tanh, ReLU, Leaky ReLU, GELU, SiLU) and their failure modes, loss and output-layer pairings with the log-sum-exp stability trick, computation graphs and reverse-mode autodiff, a full hand derivation of backprop for a 2-layer softmax-cross-entropy MLP, autograd mechanics, Xavier/He initialization, vanishing and exploding gradients, finite-difference gradient checking, and a fully numeric forward/backward/SGD worked example.

Training Deep Networks: Optimizers, Schedules & Regularization

Make deep networks converge and generalize: the non-convex landscape, saddle points and minibatch noise, batch-size effects and the linear-scaling rule, SGD with momentum and Nesterov, RMSProp, Adam with bias correction and AdamW's decoupled weight decay, learning-rate warmup and cosine/one-cycle schedules with the LR range test, BatchNorm/LayerNorm/RMSNorm/GroupNorm and pre- vs post-norm, residual connections, weight decay, dropout, label smoothing, augmentation and early stopping, gradient clipping and accumulation, a NaN checklist, a loss-curve diagnosis table, random-search hyperparameter tuning, and a worked example with a hand-computed AdamW step and four diagnosed training runs.

The PyTorch Training Loop from Scratch

Write and debug a single-GPU PyTorch training loop: tensor dtype/device rules, broadcasting traps, views vs copies and in-place autograd errors, the .item() graph leak, Dataset/DataLoader configuration (num_workers, pin_memory, drop_last, collate_fn, samplers), nn.Module parameters vs buffers and train/eval, the annotated canonical loop with correct weighted metric accumulation, autocast and GradScaler with fp16 vs bf16, checkpoints that resume exactly (optimizer, scheduler, scaler, RNG and sampler state), seeding and deterministic flags, torch.profiler and dataloader-bound diagnosis, a bug catalogue, and a worked example with real output and a find-the-five-bugs exercise with answers.

Convolutional Networks & Vision Basics

Understand and debug convolutional vision models: why convolution (locality, weight sharing, translation equivariance) and its parameter savings over fully connected layers, the exact output-size formula with padding, stride and dilation, receptive-field growth, 1x1 and depthwise-separable convolutions with their FLOP and parameter ratios, pooling vs strided convs and global average pooling, the LeNet to AlexNet to VGG to ResNet lineage with bottleneck blocks, EfficientNet and ConvNeXt, and the ViT bridge, CNN training details (BatchNorm placement, augmentation, normalization statistics, resolution), transfer learning (feature extraction, fine-tuning, discriminative learning rates, linear probes, domain gap), the task landscape with exact metrics (top-k, IoU, anchors vs anchor-free, NMS, COCO mAP, U-Net, mIoU and Dice, keypoints), Grad-CAM, efficiency, and a worked example with a verified 4-layer CNN shape and parameter count, a receptive-field calculation and mAP computed by hand from a tiny detection table.

Sequence Models: From RNNs to Attention

Understand sequence modelling end to end and be able to defend each step in an interview: tensor shapes (batch, time, features) with padding, masking and packed sequences, the many-to-one / one-to-many / seq2seq taxonomy, the vanilla RNN with parameter sharing across time, BPTT and truncated BPTT, why gradients vanish or explode through time (a product of Jacobians, with a verified experiment), LSTM and GRU gate equations, the cell state as a gradient highway and the forget-bias initialization trap, bidirectional and stacked variants with exact parameter counts, training details (teacher forcing and exposure bias, masked losses, gradient clipping, bucketing), the seq2seq fixed-vector bottleneck, Bahdanau and Luong attention and the query/key/value framing, self-attention and why Transformers won (path length, parallelism, the O(n^2) cost, and where RNNs and state-space models like Mamba remain), decoding (greedy, beam search with length penalty, temperature, top-k and top-p sampling), and a worked example with a hand-computed 3-token dot-product attention step, a one-unit LSTM step and verified LSTM parameter counts.

Transformers for AI Engineers

Covers what an AI Engineer interview actually probes about transformer internals: scaled dot-product and multi-head attention, why decoder-only architectures won, why the KV cache exists and how its memory footprint is computed, what parameter count does and doesn't predict, and positional encoding — with worked numbers for KV cache memory and prefill-vs-decode cost.

Embeddings & Representation Learning

Understand how embeddings are learned, compared and judged, and defend each choice in an interview: what an embedding is and the geometry of similarity (cosine vs dot product vs Euclidean distance, when they give the same ranking, normalization, anisotropy and hubness, both measured), classical word embeddings (skip-gram with negative sampling, the unigram^0.75 noise distribution, why SGNS is implicit factorization of a shifted PMI matrix, verified on a toy corpus; GloVe; item2vec and prod2vec for sessions), supervised embeddings (the penultimate layer, sampled and hierarchical softmax for million-class outputs, embedding tables and the hashing trick with collision arithmetic), contrastive learning (InfoNCE with temperature, in-batch negatives, hard and false negatives, batch-size dependence, SimCLR, CLIP and zero-shot classification, SBERT, triplet and margin losses, and why collapse needs negatives, stop-gradient or variance regularizers, with a measured collapse experiment), two-tower retrieval models (dot-product scoring, ANN serving, the log-Q sampling-bias correction verified by simulation, why cross features cannot enter the towers, content features for cold start), self-supervised pretext tasks (MLM, next-token, masked autoencoders) and the pretrain-finetune paradigm, multilingual and multimodal spaces with the modality gap, evaluation (neighbour sanity checks, recall@k, MRR, NDCG, linear probes, MTEB-style benchmarks, the offline-online gap), operating embeddings (dimension and storage arithmetic, Matryoshka embeddings with a measured experiment, versioning and why vectors from two model versions are not comparable, drift), and a worked example with hand-computed InfoNCE at two temperatures, a log-Q corrected two-tower score and recall/NDCG on a ranked list.

Distributed Training & GPU Systems

Plan, size and debug multi-GPU training, and defend each choice in an interview: the GPU mental model (SMs, HBM versus on-chip SRAM, arithmetic intensity and the roofline with measured intensities for a GEMM, an elementwise add and a batch-1 matvec, tensor cores, and MFU as the honest utilization metric computed from the 6ND rule), memory math (16 bytes per parameter for mixed-precision Adam, verified by measuring a real AdamW step, optimizer-state dominance, activation memory and activation checkpointing with measured saved-tensor and recompute counts, a 7B estimate), mixed precision (fp16 versus bf16 versus fp8 ranges, master weights, loss scaling, all demonstrated numerically), data parallelism (DDP, ring all-reduce byte count verified by simulation as 2(N-1)/N of the buffer, gradient bucketing and overlap, gradient accumulation equivalence verified, global batch and learning-rate scaling, SyncBatchNorm statistics verified, replica consistency verified on a multi-process gloo run), sharded data parallelism (ZeRO stages 1 to 3 and FSDP with per-GPU memory for 7B on 8 GPUs, the 1.5x communication cost, CPU offload, a ZeRO-1 emulation and PyTorch FSDP2 checked for equivalence with single-process training), model parallelism (Megatron column and row tensor parallelism verified numerically, pipeline parallelism with the bubble fraction (p-1)/(m+p-1) verified by schedule simulation and 1F1B memory, sequence and context parallelism, expert parallelism, 3D parallelism and decision rules by model size and interconnect), FlashAttention (IO-aware exact attention, tiling and online softmax verified against naive attention, O(N) memory, why it is faster despite more FLOPs, versions 2 and 3), interconnect and topology (NVLink, PCIe, InfiniBand, NCCL, why placement matters), throughput debugging (profile first, dataloader starvation, stragglers simulated, communication overlap, checkpoint cost and the Young-Daly interval, elastic training, spot instances), scaling laws (Chinchilla and what it means for budgeting fine-tunes versus pretraining), and a full worked plan for fine-tuning a 7B model on 8x80 GB with memory per strategy, micro-batch and global batch, step time and MFU.

Model Compression, Quantization & On-Device ML

Shrink a model for latency, cost or a phone and prove you did not break it: where the time and memory go (weights versus activations, memory-bandwidth-bound decode, energy and thermal limits on devices), quantization fundamentals (int8, int4, fp8, NF4 formats, symmetric versus asymmetric affine quantization with scale and zero-point worked by hand, per-tensor versus per-channel versus per-group granularity with measured error, calibration and clipping, int32 accumulation at the matmul verified), PTQ versus QAT (fake quantization and the straight-through estimator, with a 5-seed experiment showing QAT recovers 2-bit accuracy that PTQ loses), LLM quantization (the outlier-channel problem, SmoothQuant, GPTQ and AWQ ideas reproduced as toys, weight-only versus weight-and-activation, NF4 and GGUF families, KV-cache quantization, evaluation beyond perplexity), pruning and sparsity (unstructured versus structured, why zeros in a dense tensor save nothing, N:M 2:4 sparsity, iterative magnitude pruning measured, lottery tickets), knowledge distillation (soft targets, temperature and the T-squared factor verified by gradient, feature and sequence-level distillation, what actually drives the gain), low-rank factorization and LoRA merging (and why merging into int4 weights silently loses the adapter), compilers and runtimes (ONNX export pitfalls reproduced, TensorRT, torch.compile, operator fusion, BatchNorm folding verified), on-device ML (Core ML, LiteRT, ExecuTorch, NNAPI, constraints, the on-device versus server decision matrix, delivery, versioning and on-device A/B), honest benchmarking (p50 versus p99, warm versus cold, batch 1, device variance, accuracy-latency Pareto), and a worked example sizing a 7B model at fp16, int8 and int4 with decode-speed ceilings and an evaluation plan, plus a distill-then-int8-QAT plan for a phone vision model.

Model Training & Experimentation at Scale

Learn how to answer the training half of an ML system design interview: baselines, model family choice, temporal offline evaluation, tuning budgets, experiment tracking, distributed training, GPU cost estimation, embedding tables, retraining cadence, and off-policy evaluation.