LLM System Design Interview, issue 12, Nov 10, 2025
The MoE Collapse Trap
Senior ML Engineer interview at Google DeepMind, and the interviewer asks:
โYou have just launched a new Mixture of Experts (MoE) training run. After a few thousand steps, you check the logs and see the validation loss has flatlined. What is the ๐ฆ๐จ๐ฌ๐ญ ๐ฅ๐ข๐ค๐๐ฅ๐ฒ ๐๐๐ฎ๐ฌ๐ specific to an MoE, and how do you fix it?โ
Donโt say: โMy learning rate is too high,โ
When your Mixture of Experts stops learning, it's not the optimizer - it's the router. How expert starvation silently turns a 500B model into a 50B one (and how to fix it).
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.