Distributed Training & GPU Systems Quiz

Test your understanding of bytes per parameter, the roofline model, ring all-reduce, ZeRO stages, tensor versus pipeline parallelism and placement, the pipeline bubble, FlashAttention, mixed-precision formats, MFU, gradient accumulation and the Chinchilla compute-optimal ratio.

Start quiz