LLM System Design Interview, issue 17, Nov 14, 2025

The "Divine Benevolence" Fallacy

Senior ML Engineer interview at OpenAI, and the interviewer asks:

β€œOne of your junior researchers is burning compute time trying to build a theoretical proof for why 𝐒𝐰𝐒𝐆𝐋𝐔 outperforms standard π‘πžπ‹π” in your new model. Your pre-training deadline is in 48 hours. How do you handle this?”

Don’t say: β€œI’d encourage their curiosity. I’d ask them to time-box the research to one more day and present their findings. Understanding the why is key to long-term innovation.”

SwiGLU works. The ablations say so. In frontier-scale training, that's the only explanation that matters.

The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.

Read it on Substack

Get the next one

Free on Substack. Unsubscribe in one click.