The Isomorphic MLP Trick

Senior AI Engineer interview at Meta, and the interviewer asks:

“You are upgrading our legacy ReLU-based transformer to use modern SwiGLU activations. To keep the architecture consistent, you leave the feed-forward dimension up-projection ratio at the standard 𝟒 × 𝐝_𝐦𝐨𝐝𝐞𝐥. What subtle but massive architectural mistake have you just made regarding your parameter budget?”

Don’t say: “There is no mistake. A 4x multiplier is the industry standard established by the original Transformer paper. SwiGLU just replaces the non-linearity function, so keeping the matrix dimensions exactly the same is the correct way to preserve the network’s width.”