Estimate Embedding Memory, Optimizer State and Hashing Collisions

You are sizing the training cluster for a new ranker. Tables: 300M user IDs at 48 dimensions, 50M item IDs at 64 dimensions and 2M author IDs at 32 dimensions, all fp32. The optimizer is Adagrad with one fp32 accumulator per weight. The dense network has about 6M parameters. A teammate suggests hashing user IDs into 100M buckets to save memory.

1. Compute the weight memory per table and in total, and with the optimizer state. Compare to the dense network.
2. How many 80 GB devices are needed just to hold the training state if nothing else is stored, and what changes with Adam (two accumulators)?
3. Compute the fraction of user IDs that share a bucket after hashing 300M IDs into 100M buckets, and the new memory. Why is this not automatically acceptable, and what would you do?
4. What do you store at serving and in what precision? Estimate it.
5. What other things consume memory or bandwidth in practice?