Diagnose a 32-GPU Job at 28 Percent MFU

A team trains a 13B model on 32 A100-80GB GPUs (4 nodes of 8, NVLink in node, 200 Gb/s InfiniBand between nodes) with FSDP full sharding, bf16 and sequence length 4096. Throughput is 36,000 tokens/s. nvidia-smi shows 95 to 100 percent GPU utilization on every GPU, so management says the cluster is fully used. A profiler trace of one steady-state step (about 4.0 s) shows: 0.35 s of idle GPU at the start of the step; NCCL all-gather/reduce-scatter kernels totalling 1.3 s of which only 0.4 s overlaps with compute; step-time spread across ranks of 3.8 to 4.0 s; compute kernels mostly large matmuls.

1. Compute the MFU and explain why nvidia-smi disagrees with it. What throughput does 45 percent MFU correspond to?
2. Account for the step time using the trace and rank the problems by the wall-clock they cost.
3. Propose a specific fix for each and the order you would try them.
4. Which measurements would you add to prevent regressions?