Estimating Training Compute and Cost

You must retrain a ranking model daily. The model has 20 M dense parameters (embedding lookups excluded), the daily training window is 400 M examples, one epoch, and you assume ~2 FLOPs per parameter per forward pass with backward costing 2× forward. Your GPUs sustain 1.5 \times 10^{14} FLOP/s in mixed precision, but you expect only 25 % utilisation because the pipeline is I/O and lookup-bound. On-demand price: $3.50 per GPU-hour.

1. Estimate the total FLOPs and the single-GPU wall-clock time.
2. The retrain must finish within 30 minutes. How many GPUs do you need, and what is the daily cost?
3. Name two changes to the training plan (not to the hardware) that would cut this cost, and one risk of each.