The Precision Allocation Trap

Senior AI Engineer interview at Meta, and the interviewer asks:

“You’re trying to fit a 40B parameter model on 8 H100s. To save memory, you cast the entire model and optimizer state to BF16. Your training instantly goes haywire and diverges. What critical mixed-precision rule did you just violate, and why?”

Don’t say: “BF16 doesn’t have enough dynamic range. You should have used FP16 instead, or maybe your learning rate is just too high for half-precision training. I would clip the gradients and try again.”