LLM System Design Interview, issue 46, May 9, 2026
The ZeRO-1 Bandwidth Illusion
Senior ML Systems Engineer interview at OpenAI, and the interviewer asks:
“Your cluster is running standard Data Parallelism, but Adam optimizer states are causing a massive VRAM bottleneck. You suggest sharding the optimizer state across GPUs using ZeRO Stage 1, but the interviewer pushes back: Doesn’t that cause a massive network bottleneck from constantly transmitting state updates?”
Don’t say: “Yes, it adds communication overhead, but we can hide the latency by utilizing larger batch sizes or overlapping computation with communication using custom CUDA streams.”
Why assuming optimizer sharding adds network overhead is a fatal interview trap, and how decomposing an All-Reduce guarantees mathematically identical communication cost while slashing VRAM.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.