The Gradient Highway

ML Engineer interview at Google, and the interviewer asks:

“Your team is struggling with training instability and exploding gradients in a new 100B+ model. The original ‘Attention Is All You Need’ paper used post-norm with learning rate warm-up. Why is that a bad idea for deep models, and what’s the one simple architectural change that solves this?”

Don’t say: “Post-norm is just less stable, so you need a learning rate warm-up to prevent the gradients from exploding at the start. The fix is to use pre-norm.”