LLM System Design Interview, issue 23, Nov 21, 2025
The Mantissa Trap
ML Engineer interview at OpenAI, and the interviewer asks:
“Your team is hitting OOM errors. An intern engineer proposes casting the entire model and optimizer state to bfloat16 to cut memory usage by 50%. Why is this a ticking time bomb that will cause training to go out of control, and what components must stay in FP32?”
Don’t say: “Actually, bfloat16 is safe because it shares the same dynamic range (exponent) as float32. Unlike float16, it doesn’t suffer from overflow, so the intern engineer is right, you can cast everything to save memory without issues.”
Why casting everything to bfloat16 silently freezes your model, and which FP32 states are non-negotiable for stable training.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.