Advanced NLP Interview Questions

Tokenizers, attention, fine-tuning and evaluation.

25 traps, Dec 2025. Complete.

Set in interviews at Google DeepMind (8), OpenAI (7), Meta (4) and Google (2).

Each trap: the interviewer’s question, the answer most candidates give, and the mechanism that breaks it. The full answers are on Substack.

Senior NLP Engineer interview at Google DeepMind
The Back-Translation Direction Trap
“We need to improve our 𝘑𝘢𝘱𝘢𝘯𝘦𝘴𝘦-𝘵𝘰-𝘌𝘯𝘨𝘭𝘪𝘴𝘩 translation model. We have 10k parallel pairs and 1 billion lines of monolingual English text. To use 𝐁𝐚𝐜𝐤-𝐓𝐫𝐚𝐧𝐬𝐥𝐚𝐭𝐢𝐨𝐧 effectively, which direction do we generate data, and exactly how do we pair it for training?”
NLP #25
Dec 30, 2025
Senior AI Engineer interview at Anthropic
The Confidence Calibration Trap
“We’re bleeding money on inference. We want to build a 𝐌𝐨𝐝𝐞𝐥 𝐂𝐚𝐬𝐜𝐚𝐝𝐞 (𝐅𝐫𝐮𝐠𝐚𝐥𝐆𝐏𝐓) system, route easy queries to Llama-7B, and only send the hard stuff to GPT-4. What is the actual engineering bottleneck that makes this unreliable in production?”
NLP #24
Dec 29, 2025
Staff Research Scientist interview at DeepSeek
The Curriculum Learning Trap
“We have three massive datasets: 𝘎𝘦𝘯𝘦𝘳𝘢𝘭 𝘛𝘦𝘹𝘵, 𝘚𝘰𝘶𝘳𝘤𝘦 𝘊𝘰𝘥𝘦, and 𝘴𝘱𝘦𝘤𝘪𝘢𝘭𝘪𝘻𝘦𝘥 𝘔𝘢𝘵𝘩 𝘱𝘳𝘰𝘣𝘭𝘦𝘮𝘴. To build a State-of-the-Art Math reasoner, in what order do you feed this data during pre-training, and why?”
NLP #23
Dec 28, 2025
Senior ML Engineer interview at Google
The Inter-Annotator Agreement Trap
“We’re building a toxicity detection dataset where only 1% of comments are actually toxic. We hired two annotators. Their inter-annotator agreement is 99%. Are we good to go?”
NLP #22
Dec 27, 2025
Senior AI Engineer interview at OpenAI
The PPO vs DPO Implementation Trap
“Our engineers want to rip out 𝘗𝘗𝘖 (𝘗𝘳𝘰𝘹𝘪𝘮𝘢𝘭 𝘗𝘰𝘭𝘪𝘤𝘺 𝘖𝘱𝘵𝘪𝘮𝘪𝘻𝘢𝘵𝘪𝘰𝘯) and replace it with 𝘋𝘗𝘖 (𝘋𝘪𝘳𝘦𝘤𝘵 𝘗𝘳𝘦𝘧𝘦𝘳𝘦𝘯𝘤𝘦 𝘖𝘱𝘵𝘪𝘮𝘪𝘻𝘢𝘵𝘪𝘰𝘯). They argue it’s strictly better because it simplifies the stack. Do we approve the PR?”
NLP #21
Dec 27, 2025
Senior AI Engineer interview at Meta
The Quantization Gradient Trap
“We need to switch to 𝐐𝐮𝐚𝐧𝐭𝐢𝐳𝐚𝐭𝐢𝐨𝐧 𝐀𝐰𝐚𝐫𝐞 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 (𝐐𝐀𝐓) because post-training quantization is tanking our accuracy. But the rounding operation (Float -> Int8) is a step function with a derivative of zero. How do you actually backpropagate gradients through it to update the weights?”
NLP #20
Dec 26, 2025
Senior AI Engineer interview at NVIDIA
The QLoRA Compute Tax Trap
“We switched from standard FP16 fine-tuning to QLoRA (4-bit quantization) to save memory. The model fits now, but training speed hasn’t improved, it’s actually slightly slower. Why didn’t reducing precision by 4x result in a 4x speedup?”
NLP #19
Dec 25, 2025
Senior AI Engineer interview at Google
The Perplexity Tokenizer Trap
“We ran an eval on a fixed dataset. Llama3 achieved a perplexity of 2.1, while Gemma3 scored 2.4. Which model is the better probability estimator, and which one do we deploy?”
NLP #18
Dec 24, 2025
Senior ML Engineer interview at OpenAI
The Sparse Gradient Trap
“We’re training a model on a massive vocabulary. Some critical domain terms appear only once every 10,000 documents. Why will standard SGD fail to learn weights for these rare features, and how does Adam specifically fix this?”
NLP #17
Dec 23, 2025
Senior AI Engineer interview at Google DeepMind
The Hinge Loss Confidence Trap
“We’re training a massive binary text classifier. A junior engineer suggests using Hinge Loss because it creates a 𝘮𝘢𝘹 𝘮𝘢𝘳𝘨𝘪𝘯 and stops updating once a sample is correct, theoretically improving training stability. Why do we still prefer 𝐒𝐢𝐠𝐦𝐨𝐢𝐝 + 𝐋𝐨𝐠 𝐋𝐢𝐤𝐞𝐥𝐢𝐡𝐨𝐨𝐝 in production, specifically regarding the gradient signal on 𝘤𝘰𝘳𝘳𝘦𝘤𝘵 examples?”
NLP #16
Dec 22, 2025
Senior AI Engineer interview at Meta
The Positional Encoding Wall
“We used 𝘙𝘰𝘗𝘌 (𝘙𝘰𝘵𝘢𝘳𝘺 𝘗𝘰𝘴𝘪𝘵𝘪𝘰𝘯𝘢𝘭 𝘌𝘮𝘣𝘦𝘥𝘥𝘪𝘯𝘨𝘴) for Llama instead of standard absolute learned embeddings. Apart from the math, what is the critical advantage RoPE offers when we need to run inference on sequences longer than what we trained on?”
NLP #15
Dec 21, 2025
AI Engineer interview at Meta
The Tokenization Brittleness Trap
“We deployed a Llama-3 based app. We removed a single whitespace in the prompt template, and our benchmark accuracy tanked by 12%. Why is the model so brittle to a simple format change, and why didn’t instruction tuning prevent this?”
NLP #14
Dec 20, 2025
Senior AI Engineer interview at OpenAI
The Knowledge Distillation Trap
“We need to distill a massive 10-model ensemble into a single small model for low-latency serving. Why is training the student on the ensemble’s final output tokens a complete waste of compute?”
NLP #13
Dec 19, 2025
Senior Machine Learning Engineer interview at Google DeepMind
The Optimizer State Memory Trap
“You just switched a 7B parameter training run from SGD to Adam to speed up convergence. The model size is identical, but the cluster immediately crashes with a 𝘊𝘜𝘋𝘈 𝘖𝘶𝘵-𝘖𝘧-𝘔𝘦𝘮𝘰𝘳𝘺 (𝘖𝘖𝘔) error. Why?”
NLP #12
Dec 18, 2025
Senoir Machine Learning Engineer interview at Google DeepMind
The Argmax Deadlock Trap
“We are building a massive Mixture of Experts (MoE) model. To maximize training throughput on our H100 clusters, we want to route each token to only the single best expert (k=1). Is this a valid strategy?”
NLP #11
Dec 17, 2025
Machine Learning Engineer interview at OpenAI
The Contrastive Batch Size Trap
“We need to train a specialized CLIP model for medical imaging from scratch. You have a node of 8 A100s. What batch size do you configure?”
NLP #10
Dec 16, 2025
Senior NLP Engineer interview at Google DeepMind
The Tokenization Trap in Semitic Languages
“We need to adapt our English-centric LLM to support Arabic and Hebrew. How do you adjust the tokenizer?”
NLP #9
Dec 15, 2025
Senior Machine Learning Engineer interview at OpenAI
The Dropout Inference Trap
“We implemented a custom Dropout layer from scratch. How do you handle it during inference?”
NLP #8
Dec 14, 2025
Senior ML Engineer interview at Meta
The Exploding Gradient Trap
“You’re training a 7B parameter Llama-style model. In the first 1000 steps, your gradients start oscillating wildly and the loss spikes. How do you fix it?”
NLP #7
Dec 13, 2025
Senior ML Engineer interview at Google DeepMind
The LoRA Initialization Trap
“You are implementing 𝐋𝐨𝐑𝐀 (𝐋𝐨𝐰-𝐑𝐚𝐧𝐤 𝐀𝐝𝐚𝐩𝐭𝐚𝐭𝐢𝐨𝐧) from scratch. How do you initialize the down-projection matrix A and the up-projection matrix B?”
NLP #6
Dec 13, 2025
Machine Learning Engineer interview at OpenAI
The Speculative Decoding Illusion
“We need to optimize inference for batch size 128. Should we use Speculative Decoding?”
NLP #5
Dec 12, 2025
final round ML Engineer interview at Google DeepMind
The WEAT Bias Detection Trap
“How do you prove your word embeddings aren’t biased before we ship?”
NLP #4
Dec 11, 2025
Senior ML Engineer interview at OpenAI
The Attention Entropy Illusion
“How do we use the Attention mechanism’s weights to measure the model’s uncertainty?”
NLP #3
Dec 11, 2025
Senior ML Engineer interview at NVIDIA
The Gradient Shockwave Trap
“You attach a new, random linear head to a pre-trained Transformer. Do you unfreeze all layers and start backprop immediately?”
NLP #2
Dec 9, 2025
Senior ML Engineer interview at Google DeepMind
The Learning Rate Warm-Up Trap
“We are training a 𝘛𝘳𝘢𝘯𝘴𝘧𝘰𝘳𝘮𝘦𝘳 from scratch using 𝘈𝘥𝘢𝘮. We set a constant Learning Rate of 1e-3. Predict the first 1000 steps.”
NLP #1
Dec 9, 2025

Get the next one