Design Checkpoints for a Preemptible Training Job

You fine-tune a 1.3B-parameter model on a single 80 GB GPU on a preemptible (spot) instance that can be terminated at any time. The job runs AdamW in fp32 master weights with a cosine schedule, fp16 autocast with a GradScaler, a shuffled DataLoader with 8 workers and random augmentation, for about 40 hours. The persistent disk writes at roughly 1 GB/s.

1. Which exact objects go into the checkpoint and why? How large is it?
2. How do you write it so a kill during the save cannot destroy your only checkpoint, and what do you keep on disk (last/best/rolling)?
3. What extra state is needed so a resumed run follows the same trajectory as an uninterrupted one, and where does bit-exact resumption still break?
4. How often do you checkpoint? Justify with numbers.
5. How do you test that resume is correct before relying on it for 40 hours?