LLM System Design Interview, issue 39, May 2, 2026
The MoE Router Trap
Senior AI Engineer interview at Google DeepMind, and the interviewer asks:
“You are pre-training a massive Mixture of Experts (MoE) model on a cluster of H100s. Suddenly, you hit catastrophic loss spikes that completely derail convergence. Your team panics and suggests drastically lowering the learning rate from 1e-4 or aggressively tightening gradient clipping. You look at the MoE router layer and veto both ideas. Why?”
Don’t say: “The loss spikes are standard gradient explosions. Dropping the learning rate stabilizes the update steps, and clipping directly halts the exploding gradients before they corrupt the weights.”
Why dropping your learning rate to fix MoE loss spikes permanently tanks your compute economics, and the auxiliary loss trick that actually stabilizes convergence.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.