Why Reverse-Mode Autodiff Wins for Training

A model has n = 10^8 parameters and a scalar loss. One forward pass takes F = 10 ms on your hardware.

1. Roughly how long would it take to compute the full gradient with forward-mode differentiation (one Jacobian-vector product per parameter), assuming each such pass costs about one forward pass? And with reverse mode?
2. Explain in terms of the shape of the Jacobian why the answer flips for a function with 3 inputs and 10^6 outputs.
3. What does reverse mode pay in exchange for its speed?