Pipeline Parallelism: Compute the Bubble and Choose Micro-Batches

A 70B model is split across 8 pipeline stages (one per node) with 8-way tensor parallelism inside each node. Consider a single pipeline replica, which receives 64 sequences per optimizer step. A colleague configured 16 micro-batches of 4 sequences; the alternative is 64 micro-batches of 1 sequence.

1. Derive the bubble fraction for GPipe-style scheduling with p stages and m micro-batches, assuming equal stage times. Compute it for p = 8 with m = 16 and m = 64.
2. How many micro-batches would you need for a bubble of at most 10 percent at p = 8?
3. What does the 1F1B schedule change compared with GPipe, and why does it matter for activation memory? What does interleaving change and what does it cost?
4. Name the trade-offs of making micro-batches smaller (to increase m), and where the pipeline sends data.