ZeRO-3/FSDP versus Tensor Parallelism: What Is Sharded and What Moves

A colleague says "FSDP and tensor parallelism both split the model across GPUs, so they are interchangeable." You are fine-tuning a 7B model on one 8-GPU NVLink node, and later a 70B model across several nodes.

1. State precisely what ZeRO-3/FSDP shards, what each GPU computes, and what is communicated per layer. Do the same for tensor parallelism (Megatron-style MLP).
2. Show with the MLP Y = \text{GELU}(XA)\,B why the first matrix is split by columns and the second by rows, and what goes wrong if the first is split by rows.
3. Compare the communication volume of the two for 7B on 8 GPUs and say how each scales with the number of tokens in a step.
4. When do you pick which, and why can they be combined?