LLM System Design Interview, issue 59, Sep 3, 2026
The Constant-Volume Trap
Senior ML Infrastructure Engineer interview at NVIDIA, and the interviewer asks:
“Your DDP run is healthy on 8 GPUs. You scale to 64 across 8 nodes and per-GPU throughput drops 40%. Why and what should you have calculated before you bought the nodes?”
Don’t say: “Communication overhead.”
Why all-reduce payloads stay virtually flat while scaling ruins your MFU, and how to calculate your true crossover point before hitting the critical batch size.
The full answer, with the mechanism and the arithmetic, is free on Substack.