LLM System Design Interview, issue 56, Aug 31, 2026
The SwiGLU Capacity Trap
Senior LLM Research Engineer interview at Meta, and the interviewer asks:
“You swapped your ReLU FFN for SwiGLU, loss dropped 2%, and you shipped it. Why should I reject that result?”
Don’t say: “SwiGLU is a better activation, gating adds expressiveness.”
Why swapping to a gated activation silently bloats your parameter count by 50%, and the non-negotiable 2/3 scaling trick required to prove your architecture actually works.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.