LLM System Design Interview, issue 54, Aug 29, 2026
The LayerNorm Placement Paradox
Senior LLM Training Engineer interview at Anthropic, and the interviewer asks:
“You moved LayerNorm out of the residual stream and your gradient spikes disappeared. Your colleague says you should have used post-norm-outside-residual like Grok, Gemma 2, and OLMo 2. Who’s right, and what does ‘keep the residual stream clean’ actually buy you at the gradient level?”
Don’t say: “Pre-norm is more stable than post-norm.”
Why in-stream normalization quietly composed 80 layers of gradient scaling, and the elite-level distinction between bounding sublayer inputs versus sublayer updates.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.