Checkpoints, tokenizers and evaluation. A second pass over NLP, one question an issue.
14 traps, Oct 2026. Still running.
Set in interviews at Google DeepMind (4), Google (3), Meta (3) and Cohere (1).
Each trap: the interviewer’s question, the answer most candidates give, and the mechanism that breaks it. The full answers are on Substack.
Senior AI Engineer interview at Google DeepMind
“Your model crushes math word problems but falls apart the moment a user pastes LaTeX. A competitor’s model, with the same size and the same token budget, doesn’t have this problem. Where in your data pipeline do you look first?”
Advanced NLP #14
Oct 29, 2026
Senior AI Engineer interview at Meta
“We scaled from 33B to 65B. Validation loss is clearly lower, but few-shot benchmarks barely moved, and one task got worse. Is the 65B model better? Which signal do you trust?”
Advanced NLP #13
Oct 28, 2026
Senior ML Engineer interview at Google DeepMind
“You doubled attention heads from 16 to 32 with d_model fixed, expecting the model to learn more patterns. Quality is flat and throughput dropped. What did you actually change, and what should drive head count instead?”
Advanced NLP #12
Oct 27, 2026
Senior ML Engineer interview at Google
“Your autoregressive model hits a suspiciously low training loss, but every generation is garbage. What’s the first thing you check in attention? And how can a −inf mask vs. a ‘large negative’ mask still give you NaNs or leaks under mixed precision?”
Advanced NLP #11
Oct 26, 2026
Senior ML Engineer interview at Google DeepMind
“Your model was trained with learned absolute positional embeddings at a 4K context. Product needs 16K for long agent traces. Someone wants to resize the position table and ship. What breaks at position 4,097, and why isn’t ‘just fine-tune on longer data’ the whole answer?”
Advanced NLP #10
Oct 24, 2026
Senior ML Engineer interview at Google
“To fix vanishing gradients, a junior engineer opens a PR that swaps tanh for ReLU in your RNN. Their argument: ReLU’s derivative isn’t squashed below 1. Do you approve it?”
Advanced NLP #9
Oct 23, 2026
Senior ML Engineer interview at Google DeepMind
“RNNs cost O(T) per sequence. Attention costs O(T²). So why did the field abandon the cheaper architecture for training large models, and why is that O(T) number misleading you?”
Advanced NLP #8
Oct 22, 2026
Senior AI Engineer interview at Google
“Your agent makes different tool calls on identical inputs. A teammate says “set temperature to 0 and we’re done”
Advanced NLP #7
Oct 21, 2026
Senior AI Engineer interview at Meta
“Your model gets each token of a tool call right 99% of the time. So why do your 300-token JSON tool calls still fail most of the time? And what would you change in the system, not the model, to fix it?”
Advanced NLP #6
Oct 20, 2026
NLP Engineer interview at Cohere
“Your cheap bag-of-words filter handles most traffic because the LLM costs far more per call. Support shows you it flags ‘not bad at all’ as negative. A teammate proposes adding bigrams. Why is that only a partial fix?”
Advanced NLP #5
Oct 19, 2026
Advanced NLP #4
Oct 18, 2026
AI Engineer interview at Microsoft
“You auto-approve anything your classifier scores above 0.9 after the softmax. Product adds a fourth label, accuracy holds, and the auto-approve queue fills with junk. What was that 0.9 actually measuring?”
Advanced NLP #3
Oct 17, 2026
Advanced NLP #2
Oct 16, 2026
Senior AI Engineer interview at Meta
“Overnight your classifier’s accuracy falls off a cliff. The weights file is byte-identical to yesterday’s and no training job ran. Where do you look first, and what would make you reopen the model itself?”
Advanced NLP #1
Oct 15, 2026