The Learning Rate Warm-Up Trap

Senior ML Engineer interview at Google DeepMind, and the interviewer asks:

“We are training a 𝘛𝘳𝘢𝘯𝘴𝘧𝘰𝘳𝘮𝘦𝘳 from scratch using 𝘈𝘥𝘢𝘮. We set a constant Learning Rate of 1e-3. Predict the first 1000 steps.”

Don’t say: “It converges. 𝘈𝘥𝘢𝘮 is an adaptive optimizer, it adjusts per-parameter learning rates automatically. 1e-3 is a standard default. It might be noisy at first, but the loss will go down.”