The FP32 Hidden Tax

Senior AI Engineer interview at Meta, and the interviewer asks:

“You load a 7-billion parameter model onto an 80GB A100 in BF16. You calculate the weights take up a mere 14 gigabytes. But the moment you initialize your Adam optimizer and take a single training step, the script violently crashes with an Out-Of-Memory (OOM) error. Down to the exact byte multipliers, what hidden variables just silently consumed the vast majority of your memory footprint?”

Don’t say: “It’s the activations. Forward passes generate massive activation maps that scale with your sequence length, so you need to implement gradient checkpointing to save VRAM.”