LLM System Design Interview, issue 61, Sep 5, 2026
The 4% Utilization Paradox
Senior ML Engineer interview at OpenAI, and the interviewer asks:
“Your model hits 40% MFU in training. In production, prefill looks healthy but the GPUs sit at 4% utilization during generation. Nothing is broken. What’s structurally different, and why won’t bigger batches fix it?”
Don’t say: “Inference is memory-bound, so we increase batch size to improve utilization.”
Why your H100s silently starve during autoregressive generation no matter how large you batch, and why shrinking cache footprint is your only real lever.
The full answer, with the mechanism and the arithmetic, is free on Substack.