58 traps set in interviews at Anthropic. Each one: the interviewer's question, the answer most candidates give, and the mechanism that breaks it.
Roles: Senior AI Engineer (21), Senior ML Engineer (11), Staff ML Engineer (4), AI Engineer (3), Staff AI Engineer (3).
From LLM System Design Interview, LLM Inference Interview Questions, RAG Interview Questions, LLM Agents Interview Questions, Advanced Reinforcement Learning Interview Questions, AI Agent Engineering Interview, Advanced NLP Interview Questions and Generative Vision Interview Questions.
Each trap is set in an interview at Anthropic. AI Interview Prep isn't affiliated with Anthropic.
Senior AI Engineer interview at Anthropic
“You swapped raw tool outputs for LLM-generated summaries in your coding agent. Prompt size dropped sharply. But cost per resolved issue didn’t move, and on some runs it went up. Where did the savings go, and how do you prove it from the traces?”
Agent Engineering #27
Nov 1, 2026
Senior AI Engineer interview at Anthropic
“Your long-running agent crashes mid-task. You resume from the saved history, and the API rejects the entire conversation. Nothing looks malformed. What invariant did you break, and how do you guarantee a crash can never put you here again?”
Agent Engineering #5
Sep 24, 2026
Senior AI Engineer interview at Anthropic
“You gave your coding agent 40 carefully written skills to make it smarter, and task success went DOWN. Each skill description is only a few lines, so it’s not a context-budget problem. What’s actually breaking, and how would you prove it?”
Agent Engineering #2
Sep 21, 2026
Staff AI Engineer interview at Anthropic
“Product wants one model that flips between instant answers and long reasoning with a prompt tag, so we only run one deployment. Reasoning benchmarks dropped 2 points. Do you ship it?”
LLM System Design #75
Sep 19, 2026
Senior Data Engineer interview at Anthropic
“You swapped Common Crawl’s WET files for your own extraction off the raw WARC files. Same URL list, same filters, same model, but your benchmark scores moved. Your teammate says extraction is just plumbing. What do you tell him?”
LLM System Design #69
Sep 13, 2026
Senior Data Infrastructure Engineer interview at Anthropic
“Your crawler is hitting record pages-per-second. A week later a site owner is publicly complaining you hammered their servers a million times in 24 hours, and your corpus is 40% near-duplicates. Which crawl-policy decisions did you get wrong?”
LLM System Design #68
Sep 12, 2026
Staff Research Engineer interview at Anthropic
“Your team wants to swap AdamW for Muon on the next frontier run because it wins on small-scale benchmarks. What two axes must their ablation cover before you sign off?”
LLM System Design #66
Sep 10, 2026
Principal ML Engineer interview at Anthropic
“StepFun says optimal batch size depends only on total tokens D, and optimal LR goes up with more data. DeepSeek says LR just decreases with compute. Both trained real frontier models. Which one do you use for our next run?”
LLM System Design #65
Sep 9, 2026
Staff ML Engineer interview at Anthropic
“You ran one big grid search over learning rate and batch size. The batch-size scaling law is a gorgeous straight line in log-log. The learning-rate fit is a shotgun blast with a trendline drawn through it out of politeness. Same runs, same data. Why?”
LLM System Design #64
Sep 8, 2026
Senior ML Engineer interview at Anthropic
“You switched your scaling law sweeps from cosine to WSD (warmup-stable-decay). Two weeks in, the intermediate loss curves look strictly worse than the cosine baselines and leadership wants to roll back. What do you tell them?”
LLM System Design #63
Sep 7, 2026
Senior AI Research Engineer interview at Anthropic
“Your team burned three weeks sweeping learning rates at every rung of the scaling ladder before the 7B run. A colleague says muP would have let you tune once at 100M and transfer for free. Were they right?”
LLM System Design #62
Sep 6, 2026
Staff ML Engineer interview at Anthropic
“Your model is 80 layers deep and your infra team is furious. Why does almost every production LLM land near a 100:1 width-to-depth ratio?”
LLM System Design #57
Sep 1, 2026
Senior LLM Training Engineer interview at Anthropic
“Your profiler says LayerNorm is 0.17% of your FLOPs but 25% of your step time. Do you optimize it?”
LLM System Design #55
Aug 30, 2026
Senior LLM Training Engineer interview at Anthropic
“You moved LayerNorm out of the residual stream and your gradient spikes disappeared. Your colleague says you should have used post-norm-outside-residual like Grok, Gemma 2, and OLMo 2. Who’s right, and what does ‘keep the residual stream clean’ actually buy you at the gradient level?”
LLM System Design #54
Aug 29, 2026
Senior ML Engineer interview at Anthropic
“Your BPE encode() is correct and passes every round-trip test. But tokenizing your 2TB pretraining corpus would take three weeks. Your tech lead says rewrite it in Rust. Why is that the second thing you should do?”
LLM System Design #52
Aug 27, 2026
Senior AI Engineer interview at Anthropic
“Your agent completes 59-minute tasks at 50% success. Leadership wants to ship it autonomously. What’s your deployment boundary?”
LLM Inference #24
Aug 24, 2026
Senior AI Engineer interview at Anthropic
“Your agent triggers a web search whenever its reasoning trace shows uncertainty, ‘perhaps,’ ‘wait,’ ‘alternatively.’ It works great on your eval set and misses the worst failures in production. Why?”
LLM Inference #22
Aug 22, 2026
ML Engineer interview at Anthropic
“Best-of-16 rollouts with a critic reranker moves your agent from 20% to 32% on SWE-bench. Your PM wants it shipped. What do you tell him?”
LLM Inference #18
Aug 18, 2026
Staff AI Engineer interview at Anthropic
“Your agent scores 12 points higher with reasoning effort set to high, and 40% of users abandon the session before it finishes. How do you decide where to spend that reasoning budget?”
LLM Inference #17
Aug 17, 2026
Senior ML Engineer interview at Anthropic
“You’re training a tool-use agent. Do you SFT on expert tool-call traces before RL, or go straight to RL from the base model?”
LLM Inference #16
Aug 15, 2026
Senior AI Engineer interview at Anthropic
“Your agent calls a broken file-editing tool 11 times in a row with the same malformed params. You add ‘if a tool fails, note it and try another approach.’ Failure detection improves. Task success doesn’t. Why?”
LLM Inference #14
Aug 13, 2026
Senior AI Engineer interview at Anthropic
“Your agent has 3,000 registered APIs, so you retrieve top-k tool specs into context per query. It starts confidently calling the wrong tool. Where’s the failure, and why is bumping k the wrong first move?”
LLM Inference #12
Aug 10, 2026
Senior AI Engineer interview at Anthropic
“Your coding agent keeps failing on multi-line Python, escaped quotes, mangled indentation. A teammate says just use a stronger model. What’s the real fix?”
LLM Inference #10
Aug 8, 2026
Senior AI Engineer interview at Anthropic
“Your agent has 40+ tools registered across five MCP servers. Accuracy is worse than when it had 8. Your team says ‘the model got confused.’ What’s actually degrading, and what do you strip versus route?”
LLM Inference #9
Aug 7, 2026
Staff AI Engineer interview at Anthropic
“Your team wants to split your agent into a planner, a coder, and a QA agent, ‘because that’s how real software teams work. Would you ship it?”
LLM Inference #8
Aug 6, 2026
Senior AI Infrastructure Engineer interview at Anthropic
“You enabled prompt caching on a 100-step agent trajectory expecting a 5–10x cost drop. In production you’re seeing barely 1.3x, and most per-step content is cache-missing. What’s actually happening, and why does the order of your context decide whether caching works at all?”
LLM Inference #7
Aug 5, 2026
AI Agents Engineer interview at Anthropic
“You want to evaluate whether your coding agent actually solved a task. A teammate says: skip the learned critic, just train a model to generate unit tests and let pass/fail decide. It’s system-agnostic and elegant. Why isn’t this the default?”
LLM Inference #5
Aug 3, 2026
Senior AI Engineer interview at Anthropic
“You’re reranking 16 agent trajectories with a critic model to boost your SWE-bench score. Accuracy went up. Now walk me through the exact shape of that accuracy-vs-rollouts curve, and tell me when this 16x spend is actually worth it in production.”
LLM Inference #4
Aug 2, 2026
Senior AI Engineer interview at Anthropic
“Your coding agent uses a task-tracker tool, and the current plan is already in its output. So why do you also write that plan to a tasks.md file on disk, what does that actually buy you?”
LLM Inference #3
Aug 1, 2026
AI Agent Engineer interview at Anthropic
“Your coding agent keeps overflowing the context window. You add LLM summarization to condense old steps, cost drops 2x on SWE-bench. But now the agent opens the same pull request three times. What did summarization actually break, and how do you fix it without losing the savings?”
LLM Inference #2
Jul 31, 2026
Senior ML Engineer interview at Anthropic
“Your self-correction loop improves output quality on every pass, but p99 latency just tripled in prod. Your teammate says ‘just cap the max iterations.’ Why is that a band-aid, and what’s actually broken?”
LLM Inference #1
Jul 30, 2026
Senior ML Engineer interview at Anthropic
“Your retriever pulls top-10 chunks, reranks them, and you feed all 10 to the LLM to be safe. Walk me through exactly what breaks, and when you’d retrieve fewer documents on purpose.”
RAG #25
Jul 29, 2026
Senior ML Engineer interview at Anthropic
“Your GraphRAG system beat vanilla RAG on summarization benchmarks. So why should I trust it on multi-hop reasoning?”
RAG #23
Jul 27, 2026
Staff ML Engineer interview at Anthropic
“Your Corrective-RAG system has an AI evaluator scoring its own retrievals. Six months in, answer quality is degrading, but your dashboards are green. What’s happening, and how would you have prevented it on day one?”
RAG #16
Jul 20, 2026
Senior ML Engineer interview at Anthropic
“You added an LLM query rewriter to fix ambiguous queries. Recall went up in your offline eval. So why did production accuracy quietly drop three weeks later?”
RAG #15
Jul 19, 2026
Staff ML Engineer interview at Anthropic
“Your iterative RAG boosted multi-hop accuracy, but p99 latency tripled and inference costs are bleeding. A PM suggests capping retrieval at 3 loops. Why is that the wrong fix, and what should actually decide when to stop?”
RAG #14
Jul 18, 2026
Senior ML Engineer interview at Anthropic
“Your RAG system passes the raw user query straight to the retriever. Recall looks fine in your evals, but it collapses on real traffic, ambiguous and multi-hop questions especially. Before you touch the retriever, what does your query layer actually need to decide?”
RAG #6
Jul 10, 2026
Senior ML Engineer interview at Anthropic
“You shipped ColBERT-style multi-vector retrieval because it won on nDCG@10. Two weeks later, p99 latency tripled and your index storage 30x’d. When is multi-vector actually worth that, and what exactly did you lose when you collapsed passages into single vectors before?”
RAG #3
Jul 7, 2026
Senior AI Engineer interview at Anthropic
“How does the formulation of time and noise in Flow Matching fundamentally differ from standard DDPMs?”
Generative Vision #6
Jun 14, 2026
Senior AI Engineer interview at Anthropic
“Your autoregressive decoding is completely memory-bound, making token generation painfully slow. Instead of trying to optimize the memory bandwidth directly, how can you leverage a significantly smaller, ‘weaker’ model to artificially parallelize the generation process of your massive frontier model?”
LLM System Design #28
Apr 18, 2026
Senior AI Engineer interview at Anthropic
“Byte-level tokenization elegantly gives us a perfectly lossless, predictable vocabulary size of exactly 256 with zero out-of-vocabulary errors. So why would pushing a pure byte-level tokenizer into a standard Transformer instantly bankrupt our compute budget?”
LLM System Design #27
Apr 17, 2026
Senior ML Engineer interview at Anthropic
“We need to train a rigorous LLM-as-a-Judge, but we have absolutely zero human-labeled preference pairs. How do you procedurally generate ‘rejected’ responses that are nuanced enough to actually train a robust evaluator?”
LLM Agents #17
Mar 12, 2026
Senior Engineer interview at Anthropic
“You found an open-source synthetic reasoning dataset that claims a 15% bump on MMLU and GSM8K. You have the compute budget to run an SFT pass. Before adding it to your mix, what is the exact programmatic validation step you run against your eval suites to ensure that 15% isn’t a lie?”
LLM Agents #14
Mar 8, 2026
Senior AI Engineer interview at Anthropic
“Our RLHF pipeline on an 8B policy model is flatlining on reasoning tasks. A junior engineer wants to scale the Reward Model (RM) from 8B to 70B parameters to get better preference signals. Do you approve the compute budget?”
LLM Agents #13
Mar 7, 2026
Senior AI Engineer interview at Anthropic
“Your production RAG system is suffering from severe semantic leakage. A user injects a biased, false premise in their prompt ( for example: ‘Since the sun is yellow from space...’), and the LLM blindly agrees, altering its output to match the bias. System prompts and few-shot examples aren’t stopping it. What is fundamentally happening at the attention layer to cause this sycophancy, and how do you architect a fix?”
LLM Agents #10
Mar 4, 2026
Senior AI Engineer interview at Anthropic
“You’re deploying an LLM to solve highly niche competitive programming problems. Static few-shot examples don’t scale. Zero-shot ‘think step-by-step’ fails without domain context. You cannot use an external vector DB for RAG. How do you force the model to dynamically generate its own relevant context?”
LLM Agents #3
Feb 24, 2026
Senior AI Engineer interview at Anthropic
“Your autonomous coding agent is stuck in an infinite loop, writing bad Python, reading the stack trace, and confidently rewriting the exact same broken code. How do you fix it?”
LLM Agents #2
Feb 23, 2026
Senior AI Engineer interview at Anthropic
“You implemented Rejection Fine-Tuning (RFT) by sampling N solutions per problem and training on the correct ones. To push pass@1 accuracy, you drastically scale N, generating 100x more samples per prompt. Why does your test set error suddenly spike?”
Reinforcement Learning #25
Feb 20, 2026
Principal ML Engineer interview at Anthropic
“We need to scale our RLHF dataset 10x. To maximize variety, we present human labelers with two model outputs generated from different user prompts (e.g., Response A to Prompt X vs. Response B to Prompt Y). We ask them to pick the better response.”
Reinforcement Learning #11
Feb 6, 2026
Senior AI Engineer interview at Anthropic
“Our Reward scores are climbing, but the 𝘒𝘓 𝘋𝘪𝘷𝘦𝘳𝘨𝘦𝘯𝘤𝘦 term is spiking. A junior engineer suggests setting the KL coefficient (Beta) to zero to unblock the model and maximize the reward faster. Do we approve the PR?”
Reinforcement Learning #8
Feb 3, 2026
Machine Learning Engineer interview at Anthropic
“In Supervised Learning, we assume data is IID (Independent and Identically Distributed). Why does applying this assumption to a Reinforcement Learning agent, like a coding assistant, cause catastrophic failure?”
Reinforcement Learning #1
Jan 27, 2026
Senior AI Engineer interview at Anthropic
“We’re bleeding money on inference. We want to build a 𝐌𝐨𝐝𝐞𝐥 𝐂𝐚𝐬𝐜𝐚𝐝𝐞 (𝐅𝐫𝐮𝐠𝐚𝐥𝐆𝐏𝐓) system, route easy queries to Llama-7B, and only send the hard stuff to GPT-4. What is the actual engineering bottleneck that makes this unreliable in production?”
NLP #24
Dec 29, 2025
AI Engineer interview at Anthropic
“Our model is lagging on the Chatbot Arena leaderboard. We need to boost our ELO score by 50 points next release to match GPT-5. How do you adjust the post-training pipeline to make this happen?”
LLM System Design #25
Nov 23, 2025
Senior ML Engineer interview at Anthropic
“Our ops team wants to 8x our batch size to cut costs and improve throughput. Why is this a dangerous move for user experience, and at what point does this strategy stop making sense... before you run out of memory?”
LLM System Design #18
Nov 15, 2025
ML Engineer interview at Anthropic
“You’re pre-training a 500B parameter model. You’re only doing one epoch over a 10T token dataset, so you’re clearly not overfitting. Why on earth are you still using weight decay?”
LLM System Design #13
Nov 10, 2025
Lead AI Engineer interview at Anthropic
“Our new model just hit 95% on MMLU, beating GPT-4. The marketing team is drafting a press release. As the engineering lead, what’s the 𝘧𝘪𝘳𝘴𝘵 𝘵𝘩𝘪𝘯𝘨 you check for that could invalidate this result?”
LLM System Design #8
Nov 6, 2025
AI Engineer interview at Anthropic
“We have a fixed compute budget - 32 H100s for two weeks. To get the best possible model, should we train a larger model on less data, or a smaller model on more data? How do you begin to answer this?”
LLM System Design #3
Nov 5, 2025
AI Engineer interview at Anthropic
“We’re training a new model for the legal and medical domain. What’s the production risk of just using a standard, pre-trained Llama 3 tokenizer, and what’s your fix?”
LLM System Design #1
Nov 5, 2025