The Vocab Embedding Paradox

Senior Pre-training Engineer interview at Google DeepMind, and the interviewer asks:

“You’ve trained a series of smaller proxy models to project scaling laws for your next 100B+ flagship LLM. However, your parameter-to-loss plot isn’t a straight line in log-log space, it’s bending noticeably at the low-parameter end. Assuming training was perfectly stable, what basic structural miscalculation is ruining your extrapolation curve?”

Don’t say: “The smallest models are underfitting due to a suboptimal learning rate schedule. We need to tune the cool-down phase or adjust the batch size to ensure the proxy models hit their true minimums.”