LLM System Design Interview, issue 44, May 7, 2026
The Bandwidth-Precision Trap
Senior AI Engineer interview at Google DeepMind, and the interviewer asks:
“You aggressively cast your entire model to Float16 to double your memory bandwidth and halve your payload. It runs blazingly fast, but your loss diverges and produces NaNs immediately. What critical separation of concerns did you fail to implement in your arithmetic intensity strategy?”
Don’t say: “Float16 has a smaller dynamic range, so the gradients must have overflowed. We should just lower the learning rate, use gradient clipping, or add a larger epsilon to our layer norms to force stability.”
Why aggressive 16-bit casting silently triggers "swamping" inside your GPUs, and the critical separation of concerns required to scale hardware utilization without sacrificing convergence.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.