The Optimizer State Memory Trap

Senior Machine Learning Engineer interview at Google DeepMind, and the interviewer asks:

“You just switched a 7B parameter training run from SGD to Adam to speed up convergence. The model size is identical, but the cluster immediately crashes with a 𝘊𝘜𝘋𝘈 𝘖𝘶𝘵-𝘖𝘧-𝘔𝘦𝘮𝘰𝘳𝘺 (𝘖𝘖𝘔) error. Why?”

Don’t say: “Adam is computationally more expensive so it uses more memory,”