Tools, planning and memory, and why agents lose the plot forty steps in.
25 traps, Feb 2026 to Mar 2026. Complete.
Set in interviews at Google DeepMind (10), OpenAI (7), Anthropic (6) and NVIDIA (1).
Each trap: the interviewer’s question, the answer most candidates give, and the mechanism that breaks it. The full answers are on Substack.
Senior AI Engineer interview at Google DeepMind
“You’ve implemented self-consistency (majority voting) to improve an agent’s math reasoning. But scaling the sample size from 10 to 40 yields zero performance gain. What is silently killing your scaling laws?”
LLM Agents #25
Mar 21, 2026
Senior AI Engineer interview at OpenAI
“You just started fine-tuning a highly capable base model to output internal reasoning thoughts before answering. In the first few PPO iterations, overall benchmark performance completely tanks compared to the base model. Why is this regression mathematically expected, and what forces the turnaround?”
LLM Agents #24
Mar 20, 2026
Senior AI Engineer interview at OpenAI
“Your LLM is generating long, list-based responses. It’s nailing the broad concepts, but constantly hallucinating specific entities, like slipping Michael Bloomberg into a list of politicians born in New York. Standard think step-by-step prompting is failing. How do you stop this?”
LLM Agents #23
Mar 19, 2026
Senior AI Engineer interview at OpenAI
“You’re fine-tuning an LLM for instruction following (IFEval) using PPO. By step 400, your reward curve is steadily climbing, but your actual evaluation scores are tanking. How do you fix the reward pipeline without just training a massive 70B reward model?”
LLM Agents #22
Mar 18, 2026
Senior AI Engineer interview at Google DeepMind
“Your theorem-proving LLM is blind to a researcher’s custom project. You stuff the entire active file into the 128k prompt context. Local accuracy spikes, but it still fails on cross-file theorems. Why?”
LLM Agents #21
Mar 17, 2026
Senior AI Engineer interview at Google DeepMind
“Your RLHF pipeline relies on top-tier medical and legal experts to score outputs. But as the model scales, your PPO updates start degrading its reasoning accuracy rather than refining it. What is breaking down, and how do you fix it?”
LLM Agents #20
Mar 16, 2026
Senior AI Engineer interview at Google DeepMind
“Your monolithic coding agent is handling both repo-wide search and patch generation, but as the context window fills up, the patch quality tanks. How do you architect the agent loop to fix this degradation?”
LLM Agents #19
Mar 15, 2026
Senior AI Engineer interview at Google DeepMind
“Your reasoning model hits 80% on miniF2F math benchmarks using just the current proof state. You deploy it to help researchers formalize a real paper in Lean, and its accuracy flatlines to 0%. Why?”
LLM Agents #18
Mar 14, 2026
Senior ML Engineer interview at Anthropic
“We need to train a rigorous LLM-as-a-Judge, but we have absolutely zero human-labeled preference pairs. How do you procedurally generate ‘rejected’ responses that are nuanced enough to actually train a robust evaluator?”
LLM Agents #17
Mar 12, 2026
Senior AI Engineer interview at Google DeepMind
“You upgraded your geometry autoformalization pipeline from a 70B text-only LLM to a state-of-the-art VLM. You feed it textbook diagrams alongside the text. Success rates barely nudge past 20%. Why?”
LLM Agents #16
Mar 10, 2026
AI Engineer interview at Google DeepMind
“Your new autonomous coding agent is hitting 40% on SWE-Bench. The PRs pass all historical unit tests perfectly. But when you manually review the code, the patches are technically incorrect and introduce massive regressions. Why is your agent passing the test but failing the engineering task?”
LLM Agents #15
Mar 9, 2026
Senior Engineer interview at Anthropic
“You found an open-source synthetic reasoning dataset that claims a 15% bump on MMLU and GSM8K. You have the compute budget to run an SFT pass. Before adding it to your mix, what is the exact programmatic validation step you run against your eval suites to ensure that 15% isn’t a lie?”
LLM Agents #14
Mar 8, 2026
Senior AI Engineer interview at Anthropic
“Our RLHF pipeline on an 8B policy model is flatlining on reasoning tasks. A junior engineer wants to scale the Reward Model (RM) from 8B to 70B parameters to get better preference signals. Do you approve the compute budget?”
LLM Agents #13
Mar 7, 2026
Senior AI Engineer interview at Google DeepMind
“You’re building an autonomous coding agent to navigate a massive monorepo and generate bug patches. But by the time the agent actually finds the right files, its patch generation quality completely tanks. How do you architect the agent loop to fix this degradation?”
LLM Agents #12
Mar 6, 2026
Senior AI Engineer interview at Stripe
“You’re building a complex text-to-SQL agent. You stuff all 50 database grammar rules into an 8k system prompt for Chain-of-Thought reasoning, but the model constantly loses track of constraints and hallucinates table joins. Why?”
LLM Agents #11
Mar 5, 2026
Senior AI Engineer interview at Anthropic
“Your production RAG system is suffering from severe semantic leakage. A user injects a biased, false premise in their prompt ( for example: ‘Since the sun is yellow from space...’), and the LLM blindly agrees, altering its output to match the bias. System prompts and few-shot examples aren’t stopping it. What is fundamentally happening at the attention layer to cause this sycophancy, and how do you architect a fix?”
LLM Agents #10
Mar 4, 2026
Senior AI Engineer interview at OpenAI
“Your transformer’s training loss just hit zero on a relational dataset. It’s perfectly overfit. Infra is screaming at you to kill the run and save the A100s. Why might pulling the plug right now completely destroy the model’s ability to reason implicitly in production?”
LLM Agents #9
Mar 3, 2026
Senior AI Engineer interview at OpenAI
“Your multimodal agent hits a 95% success rate on static benchmarks like Mind2Web, but completely falls apart when we deploy it in a live OS environment. Why is it failing, and how do we actually measure true reliability?”
LLM Agents #8
Mar 2, 2026
Senior AI Engineer interview at OpenAI
“You’ve built a web navigation agent using the full accessibility tree (DOM) to maximize contextual awareness. But latency is spiking, and it’s completely failing on complex web apps. What is the hidden architectural bottleneck here, and why is pure-vision grounding the superior production choice?”
LLM Agents #7
Mar 1, 2026
Senior AI Engineer interview at Google DeepMind
“We need to fine-tune the VLM on paired image-to-equation datasets.”
LLM Agents #6
Feb 27, 2026
Senior AI Engineer interview at Google DeepMind
“Your zero-shot LLM keeps failing on Olympiad-level formal proofs because the reasoning depth is too vast. You have 10,000 A100 hours of inference budget to burn. Instead of naively sampling full proofs 100 times, what multi-stage decomposition strategy do you implement?”
LLM Agents #5
Feb 26, 2026
Senior AI Engineer interview at OpenAI
“Your math tutor LLM consistently nails the final answer, but silently hallucinates logic flaws in step 4 or 5. You are operating at scale and cannot afford human-in-the-loop verification. What fundamental architectural shift guarantees 100% intermediate correctness?”
LLM Agents #4
Feb 25, 2026
Senior AI Engineer interview at Anthropic
“You’re deploying an LLM to solve highly niche competitive programming problems. Static few-shot examples don’t scale. Zero-shot ‘think step-by-step’ fails without domain context. You cannot use an external vector DB for RAG. How do you force the model to dynamically generate its own relevant context?”
LLM Agents #3
Feb 24, 2026
Senior AI Engineer interview at Anthropic
“Your autonomous coding agent is stuck in an infinite loop, writing bad Python, reading the stack trace, and confidently rewriting the exact same broken code. How do you fix it?”
LLM Agents #2
Feb 23, 2026
Senior AI Engineer interview at NVIDIA
“Your team just upgraded an internal LLM from a 7B to a 70B parameter model using the exact same training dataset and 100k step schedule. You expect a reasoning bump, but SecOps flags a 400% spike in PII ( Personally Identifiable Information ) extraction via simple prompting. Why does scaling up independently degrade privacy, and how do you fix it without rolling back?”
LLM Agents #1
Feb 23, 2026