The AdamW Memory Trap

Senior PyTorch Engineer interview at Meta, and the interviewer asks:

“Your 70B parameter model training job on 1,024 H100s just crashed on day 5. You successfully load the saved model.state_dict() and resume, but your loss immediately spikes to the moon, destroying weeks of progress. What happened?”

Don’t say: “You probably forgot to set the random seeds for your data loader, so it fed the model a completely different data distribution, or you had a bad batch spike.”