LLM System Design Interview, issue 38, May 1, 2026
The MoE Jitter Trap
Senior AI Engineer interview at Google DeepMind, and the interviewer asks:
“You deployed a top-K routed MoE. Within 500 steps, validation loss flatlines. You check the telemetry and see 6 of your 8 experts have near-zero activation. A junior MLE proposes injecting stochastic jitter to the router logits to force exploration. Do you merge the PR?”
Don’t say: “Yes, that makes sense. It’s essentially a multi-armed bandit problem. Adding Gaussian noise to the logits acts like epsilon-greedy exploration, forcing the router to sample the dead experts, learn their value, and escape the local minimum.”
Why adding random noise to fix dead experts silently destroys specialization and causes train/test mismatch and the deterministic auxiliary loss that actually saves your H100 cycles.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.