Training runs, serving stacks and the bills they produce.
75 traps, Nov 2025 to Sep 2026. Still running.
Set in interviews at Anthropic (20), Google DeepMind (20), OpenAI (16) and Meta (10).
Each trap: the interviewer’s question, the answer most candidates give, and the mechanism that breaks it. The full answers are on Substack.
Staff AI Engineer interview at Anthropic
“Product wants one model that flips between instant answers and long reasoning with a prompt tag, so we only run one deployment. Reasoning benchmarks dropped 2 points. Do you ship it?”
LLM System Design #75
Sep 19, 2026
Senior AI Engineer interview at OpenAI
“Your RLVR math pipeline has a verifier, so reward hacking isn’t your problem. But accuracy has plateaued far below what manual review says the model can do. Where is ‘verifiable’ failing you?”
LLM System Design #74
Sep 18, 2026
Senior ML Engineer interview at Google DeepMind
“You switched to GRPO. Average chain-of-thought length grows every training step. Leadership calls it ‘the model learning to think harder.’ What’s the less flattering explanation, and how do you verify it before your inference bill doubles?”
LLM System Design #73
Sep 17, 2026
Senior ML Engineer interview at OpenAI
“Your PPO reward model scores climbed for six weeks while you kept adding compute. Human evals got worse. Why did more RL compute stop working, and what does your reward need before scaling RL actually pays off?”
LLM System Design #72
Sep 16, 2026
Senior LLM Post-Training Engineer interview at OpenAI
“Your new SFT mix lifts head-to-head win rates by 15 points, but MMLU, GSM8K, and your internal capability suite are completely flat. Leadership wants to ship it as a capability gain. What do you tell them?”
LLM System Design #71
Sep 15, 2026
Staff AI Engineer interview at OpenAI
“Your team paid expert annotators to write the highest-quality SFT responses possible, detailed, sourced, with citations. After fine-tuning, factual hallucination rate went up. What happened?”
LLM System Design #70
Sep 14, 2026
Senior Data Engineer interview at Anthropic
“You swapped Common Crawl’s WET files for your own extraction off the raw WARC files. Same URL list, same filters, same model, but your benchmark scores moved. Your teammate says extraction is just plumbing. What do you tell him?”
LLM System Design #69
Sep 13, 2026
Senior Data Infrastructure Engineer interview at Anthropic
“Your crawler is hitting record pages-per-second. A week later a site owner is publicly complaining you hammered their servers a million times in 24 hours, and your corpus is 40% near-duplicates. Which crawl-policy decisions did you get wrong?”
LLM System Design #68
Sep 12, 2026
Senior AI Engineer interview at Google DeepMind
“Your VP says: ‘We scraped the web in 2020 and got a great corpus. Just re-run the crawler and we’ll get the same data, only bigger.’ Why is that wrong, and how does it change your pre-training data strategy?”
LLM System Design #67
Sep 11, 2026
Staff Research Engineer interview at Anthropic
“Your team wants to swap AdamW for Muon on the next frontier run because it wins on small-scale benchmarks. What two axes must their ablation cover before you sign off?”
LLM System Design #66
Sep 10, 2026
Principal ML Engineer interview at Anthropic
“StepFun says optimal batch size depends only on total tokens D, and optimal LR goes up with more data. DeepSeek says LR just decreases with compute. Both trained real frontier models. Which one do you use for our next run?”
LLM System Design #65
Sep 9, 2026
Staff ML Engineer interview at Anthropic
“You ran one big grid search over learning rate and batch size. The batch-size scaling law is a gorgeous straight line in log-log. The learning-rate fit is a shotgun blast with a trendline drawn through it out of politeness. Same runs, same data. Why?”
LLM System Design #64
Sep 8, 2026
Senior ML Engineer interview at Anthropic
“You switched your scaling law sweeps from cosine to WSD (warmup-stable-decay). Two weeks in, the intermediate loss curves look strictly worse than the cosine baselines and leadership wants to roll back. What do you tell them?”
LLM System Design #63
Sep 7, 2026
Senior AI Research Engineer interview at Anthropic
“Your team burned three weeks sweeping learning rates at every rung of the scaling ladder before the 7B run. A colleague says muP would have let you tune once at 100M and transfer for free. Were they right?”
LLM System Design #62
Sep 6, 2026
Senior ML Engineer interview at OpenAI
“Your model hits 40% MFU in training. In production, prefill looks healthy but the GPUs sit at 4% utilization during generation. Nothing is broken. What’s structurally different, and why won’t bigger batches fix it?”
LLM System Design #61
Sep 5, 2026
Principal ML Infrastructure Engineer interview at NVIDIA
“Your pipeline-parallel run is showing 60% GPU idle time. Your junior says ‘just increase the micro-batches.’ Is he right?”
LLM System Design #60
Sep 4, 2026
Senior ML Infrastructure Engineer interview at NVIDIA
“Your DDP run is healthy on 8 GPUs. You scale to 64 across 8 nodes and per-GPU throughput drops 40%. Why and what should you have calculated before you bought the nodes?”
LLM System Design #59
Sep 3, 2026
Senior ML Systems Engineer interview at Meta
“You moved your matmuls to FP8 on H100s expecting a 2× speedup. You got 25%. Is FP8 overhyped, or did you do it wrong?”
LLM System Design #58
Sep 2, 2026
Staff ML Engineer interview at Anthropic
“Your model is 80 layers deep and your infra team is furious. Why does almost every production LLM land near a 100:1 width-to-depth ratio?”
LLM System Design #57
Sep 1, 2026
Senior LLM Research Engineer interview at Meta
“You swapped your ReLU FFN for SwiGLU, loss dropped 2%, and you shipped it. Why should I reject that result?”
LLM System Design #56
Aug 31, 2026
Senior LLM Training Engineer interview at Anthropic
“Your profiler says LayerNorm is 0.17% of your FLOPs but 25% of your step time. Do you optimize it?”
LLM System Design #55
Aug 30, 2026
Senior LLM Training Engineer interview at Anthropic
“You moved LayerNorm out of the residual stream and your gradient spikes disappeared. Your colleague says you should have used post-norm-outside-residual like Grok, Gemma 2, and OLMo 2. Who’s right, and what does ‘keep the residual stream clean’ actually buy you at the gradient level?”
LLM System Design #54
Aug 29, 2026
Senior ML Engineer interview at OpenAI
“Your tokenizer fails to round-trip on 0.3% of production traffic, emoji, mixed-script usernames, truncated UTF-8 from a bad client. Your teammate maps them all to UNK and ships. What did he just break?”
LLM System Design #53
Aug 28, 2026
Senior ML Engineer interview at Anthropic
“Your BPE encode() is correct and passes every round-trip test. But tokenizing your 2TB pretraining corpus would take three weeks. Your tech lead says rewrite it in Rust. Why is that the second thing you should do?”
LLM System Design #52
Aug 27, 2026
Staff ML Engineer interview at Google
“Your tokenizer gets 2.5 bytes/token on English but 1.1 on Vietnamese and Thai. Your teammate wants to grow the vocab from 100k to 250k. Do you ship it?”
LLM System Design #51
Aug 26, 2026
Senior AI Engineer interview at Google DeepMind
“You deployed a 70B target model with a 1B draft model for speculative decoding. Accuracy is identical, but your expected 2x speedup is sitting at exactly 0%. Why?”
LLM System Design #50
May 13, 2026
Senior Pre-training Engineer interview at Google DeepMind
“You’ve trained a series of smaller proxy models to project scaling laws for your next 100B+ flagship LLM. However, your parameter-to-loss plot isn’t a straight line in log-log space, it’s bending noticeably at the low-parameter end. Assuming training was perfectly stable, what basic structural miscalculation is ruining your extrapolation curve?”
LLM System Design #49
May 12, 2026
Senior AI Engineer interview at Google DeepMind
“You scaled your pre-training dataset by 10x, but the error rate barely budged. Your model is massively over-parameterized, so capacity isn’t the issue. What ‘intrinsic’ statistical property of your target task is fundamentally bottlenecking your power-law returns?”
LLM System Design #48
May 11, 2026
Senior Pre-training Engineer interview at Google DeepMind
“Compute is tight, but you need to find the exact optimal ratio of code, web, and book data for a 100B parameter model. How do you empirically determine the perfect mixture without wasting millions of dollars running ablation tests at massive scale?”
LLM System Design #47
May 10, 2026
Senior ML Systems Engineer interview at OpenAI
“Your cluster is running standard Data Parallelism, but Adam optimizer states are causing a massive VRAM bottleneck. You suggest sharding the optimizer state across GPUs using ZeRO Stage 1, but the interviewer pushes back: Doesn’t that cause a massive network bottleneck from constantly transmitting state updates?”
LLM System Design #46
May 9, 2026
Senior AI Engineer interview at Meta
“You load a 7-billion parameter model onto an 80GB A100 in BF16. You calculate the weights take up a mere 14 gigabytes. But the moment you initialize your Adam optimizer and take a single training step, the script violently crashes with an Out-Of-Memory (OOM) error. Down to the exact byte multipliers, what hidden variables just silently consumed the vast majority of your memory footprint?”
LLM System Design #45
May 8, 2026
Senior AI Engineer interview at Google DeepMind
“You aggressively cast your entire model to Float16 to double your memory bandwidth and halve your payload. It runs blazingly fast, but your loss diverges and produces NaNs immediately. What critical separation of concerns did you fail to implement in your arithmetic intensity strategy?”
LLM System Design #44
May 7, 2026
Senior AI Systems Engineer interview at OpenAI
“To handle a few edge cases in your custom loss function, you add a basic if/else statement inside your CUDA kernel. Suddenly, your execution time doubles. What just happened?”
LLM System Design #43
May 6, 2026
Senior AI Engineer interview at Google DeepMind
“Your training job is unacceptably slow, so you secure the budget to upgrade to a new cluster with 5x the raw teraFLOPs. However, your end-to-end throughput barely increases by 1.2x. What fundamental hardware scaling reality did you fail to profile before upgrading?”
LLM System Design #42
May 5, 2026
Senior LLM Engineer interview at DeepSeek
“You’ve implemented Multi-Head Latent Attention (MLA) to crush your KV-cache footprint. But uncompressing that latent vector requires an extra up-projection matrix, blowing up your inference FLOPs. How do you completely erase that computational cost entirely during the forward pass?”
LLM System Design #41
May 4, 2026
Senior ML Engineer interview at Google DeepMind
“Sending the exact same prompt yields slightly different outputs depending on the time of day.”
LLM System Design #40
May 3, 2026
Senior AI Engineer interview at Google DeepMind
“You are pre-training a massive Mixture of Experts (MoE) model on a cluster of H100s. Suddenly, you hit catastrophic loss spikes that completely derail convergence. Your team panics and suggests drastically lowering the learning rate from 1e-4 or aggressively tightening gradient clipping. You look at the MoE router layer and veto both ideas. Why?”
LLM System Design #39
May 2, 2026
Senior AI Engineer interview at Google DeepMind
“You deployed a top-K routed MoE. Within 500 steps, validation loss flatlines. You check the telemetry and see 6 of your 8 experts have near-zero activation. A junior MLE proposes injecting stochastic jitter to the router logits to force exploration. Do you merge the PR?”
LLM System Design #38
May 1, 2026
Senior LLM Pre-training Engineer interview at Google DeepMind
“We are setting up a single-epoch pre-training run on a 10-petabyte text corpus. A junior engineer removes the weight decay, arguing we can’t possibly overfit a dataset this large in one epoch. Do you merge their PR?”
LLM System Design #37
Apr 30, 2026
Senior AI Engineer interview at Meta
“You are upgrading our legacy ReLU-based transformer to use modern SwiGLU activations. To keep the architecture consistent, you leave the feed-forward dimension up-projection ratio at the standard 𝟒 × 𝐝_𝐦𝐨𝐝𝐞𝐥. What subtle but massive architectural mistake have you just made regarding your parameter budget?”
LLM System Design #36
Apr 29, 2026
Senior LLM Engineer interview at Google DeepMind
“You inherit a legacy Transformer codebase and notice the team is dropping all bias terms in the linear layers. Someone suggests adding them back to increase the representational power of the model. What production-level training catastrophe are you actively inviting if you approve that pull request?”
LLM System Design #35
Apr 28, 2026
Senior AI Engineer interview at Meta
“We are optimizing our next 70B parameter LLM and decide to swap standard LayerNorm for RMSNorm. Why are we doing this?”
LLM System Design #34
Apr 27, 2026
Senior Machine Learning Engineer interview at OpenAI
“You are tasked with training an LLM on a massive 2.8TB text dataset. How do you feed this to your PyTorch dataloader without instantly OOMing your system’s CPU RAM?”
LLM System Design #33
Apr 25, 2026
Senior PyTorch Engineer interview at Meta
“Your 70B parameter model training job on 1,024 H100s just crashed on day 5. You successfully load the saved model.state_dict() and resume, but your loss immediately spikes to the moon, destroying weeks of progress. What happened?”
LLM System Design #32
Apr 22, 2026
Senior ML Engineer interview at Google DeepMind
“You are passing a transposed matrix into a custom loss function to fix a shape mismatch. Your code throws a contiguity error, so you add a .reshape() or .contiguous().view() to make it compile. Suddenly, your PyTorch profiler shows a massive, unexpected spike in VRAM usage. What silent memory allocation trap did you just trigger?”
LLM System Design #31
Apr 21, 2026
Senior AI Engineer interview at Meta
“You’re trying to fit a 40B parameter model on 8 H100s. To save memory, you cast the entire model and optimizer state to BF16. Your training instantly goes haywire and diverges. What critical mixed-precision rule did you just violate, and why?”
LLM System Design #30
Apr 20, 2026
Principal AI Engineer interview at Meta
“We just secured a cluster of 100,000 H100s, but we’ve completely run out of high-quality internet text. How does entering this ‘data-constrained’ regime completely invert our standard assumptions about epochs and architecture design?”
LLM System Design #29
Apr 19, 2026
Senior AI Engineer interview at Anthropic
“Your autoregressive decoding is completely memory-bound, making token generation painfully slow. Instead of trying to optimize the memory bandwidth directly, how can you leverage a significantly smaller, ‘weaker’ model to artificially parallelize the generation process of your massive frontier model?”
LLM System Design #28
Apr 18, 2026
Senior AI Engineer interview at Anthropic
“Byte-level tokenization elegantly gives us a perfectly lossless, predictable vocabulary size of exactly 256 with zero out-of-vocabulary errors. So why would pushing a pure byte-level tokenizer into a standard Transformer instantly bankrupt our compute budget?”
LLM System Design #27
Apr 17, 2026
Senior AI Engineer interview at OpenAI
“You spent months optimizing the attention layers on our 1.4B model and got a massive speedup. Why will that exact same profiling strategy completely fail to move the needle when we scale that architecture to 175B parameters?”
LLM System Design #26
Apr 16, 2026
AI Engineer interview at Anthropic
“Our model is lagging on the Chatbot Arena leaderboard. We need to boost our ELO score by 50 points next release to match GPT-5. How do you adjust the post-training pipeline to make this happen?”
LLM System Design #25
Nov 23, 2025
Machine Learning Systems Engineer interview at Google DeepMind
“You’re asked to budget a training run. An intern engineer estimates the total FLOPs as 2 * num_params * num_tokens, arguing the backward pass is roughly symmetrical to the forward pass. Why is this cost estimate off by 300%, and what two distinct gradient calculations (totaling 4x, not 2x) are they failing to account for?”
LLM System Design #24
Nov 22, 2025
ML Engineer interview at OpenAI
“Your team is hitting OOM errors. An intern engineer proposes casting the entire model and optimizer state to bfloat16 to cut memory usage by 50%. Why is this a ticking time bomb that will cause training to go out of control, and what components must stay in FP32?”
LLM System Design #23
Nov 21, 2025
technical Engineer interview at NVIDIA
“An intern excitedly claims they achieved a 1000x speedup on a new matrix multiplication kernel. You look at their script and see they simply wrapped the function call with standard Python timers: 𝘴𝘵𝘢𝘳𝘵 = 𝘵𝘪𝘮𝘦.𝘵𝘪𝘮𝘦() ... 𝘦𝘯𝘥 = 𝘵𝘪𝘮𝘦.𝘵𝘪𝘮𝘦() Why are their results a complete lie?”
LLM System Design #22
Nov 18, 2025
Senior AI Engineer interview at Google DeepMind
“We’ve implemented the original DeepSeek GRPO paper to train our new math chatbot. On uncertain queries, the Chain-of-Thought (CoT) is suddenly exploding to 10000 tokens. An engineer on the team says this is great, the model is just thinking harder and learning to backtrack. What’s your diagnosis?”
LLM System Design #21
Nov 17, 2025
Senior ML Engineer interview at Perplexity
“Your PM wants to fine-tune our new model on a random 1M sample of live user prompts to improve real-world performance. You tell them it’s a terrible idea. Why?”
LLM System Design #20
Nov 17, 2025
AI Engineer interview at OpenAI
“A project plan budgets 1 day for data prep: 𝘋𝘰𝘸𝘯𝘭𝘰𝘢𝘥 𝘊𝘰𝘮𝘮𝘰𝘯 𝘊𝘳𝘢𝘸𝘭. Why is this 𝘵𝘳𝘢𝘪𝘯 𝘰𝘯 𝘵𝘩𝘦 𝘪𝘯𝘵𝘦𝘳𝘯𝘦𝘵 mindset a complete fantasy that guarantees a 𝐭𝐫𝐚𝐬𝐡 model ?”
LLM System Design #19
Nov 16, 2025
Senior ML Engineer interview at Anthropic
“Our ops team wants to 8x our batch size to cut costs and improve throughput. Why is this a dangerous move for user experience, and at what point does this strategy stop making sense... before you run out of memory?”
LLM System Design #18
Nov 15, 2025
Senior ML Engineer interview at OpenAI
“One of your junior researchers is burning compute time trying to build a theoretical proof for why 𝐒𝐰𝐢𝐆𝐋𝐔 outperforms standard 𝐑𝐞𝐋𝐔 in your new model. Your pre-training deadline is in 48 hours. How do you handle this?”
LLM System Design #17
Nov 14, 2025
AI Engineer interview at Google DeepMind
“A new engineer implements RoPE by adding a rotational embedding to the token embeddings at the bottom of the model. The training loss is flat. What fundamental misunderstanding do they have about how and where RoPE is actually applied?”
LLM System Design #16
Nov 14, 2025
Senior AI Engineer interview at Meta
“You’re A/B testing two 70B models - one Multi-Head Attention (MHA), one Grouped Query Attention (GQA). Your colleague argues they’ll have the same inference speed since FLOPs and parameter counts are identical. Is this assumption correct?”
LLM System Design #15
Nov 12, 2025
ML Engineer interview at Google
“We need to serve our model for two different use cases: a low-latency chatbot that needs a fast 𝐓𝐢𝐦𝐞-𝐭𝐨-𝐅𝐢𝐫𝐬𝐭-𝐓𝐨𝐤𝐞𝐧 (𝐓𝐓𝐅𝐓), and a high-throughput batch summarization job. How do these two workloads stress the GPU differently, and what fundamental tradeoff are you managing?”
LLM System Design #14
Nov 11, 2025
ML Engineer interview at Anthropic
“You’re pre-training a 500B parameter model. You’re only doing one epoch over a 10T token dataset, so you’re clearly not overfitting. Why on earth are you still using weight decay?”
LLM System Design #13
Nov 10, 2025
Senior ML Engineer interview at Google DeepMind
“You have just launched a new Mixture of Experts (MoE) training run. After a few thousand steps, you check the logs and see the validation loss has flatlined. What is the 𝐦𝐨𝐬𝐭 𝐥𝐢𝐤𝐞𝐥𝐲 𝐜𝐚𝐮𝐬𝐞 specific to an MoE, and how do you fix it?”
LLM System Design #12
Nov 10, 2025
AI Engineer interview at OpenAI
“You’ve successfully fine-tuned a model with RL. It’s now excellent at following instructions, but it’s become ‘dumber’ at general knowledge and creative writing. What is this phenomenon called, and what specific term would you add to your loss function to prevent this?”
LLM System Design #11
Nov 9, 2025
Senior ML Engineer interview at Google DeepMind
“Our new reasoning model is great, but it uses a 2000 token Chain of Thought even for simple questions like ‘What is 2+2?’. This is killing our inference budget. How do you fix this without sacrificing its ability to solve complex problems?”
LLM System Design #10
Nov 8, 2025
Staff ML Engineer interview
“Your team needs to support a 10M context window. An engineer says it’s impossible because standard attention is O(N²) compute. Why is that the wrong bottleneck to focus on, and how does FlashAttention actually solve the real problem?”
LLM System Design #9
Nov 7, 2025
Lead AI Engineer interview at Anthropic
“Our new model just hit 95% on MMLU, beating GPT-4. The marketing team is drafting a press release. As the engineering lead, what’s the 𝘧𝘪𝘳𝘴𝘵 𝘵𝘩𝘪𝘯𝘨 you check for that could invalidate this result?”
LLM System Design #8
Nov 6, 2025
AI Engineer interview at Google DeepMind
“Your 1B parameter proxy model trains perfectly with a 1.2e-4 learning rate. You scale the model to 70B, and the training immediately explodes. What’s the most 𝘭𝘪𝘬𝘦𝘭𝘺 reason and how do you fix it 𝐰𝐢𝐭𝐡𝐨𝐮𝐭 running a new, expensive hyperparameter sweep?”
LLM System Design #7
Nov 5, 2025
AI Engineer interview at Meta
“We all know KV Caching speeds up token generation. What’s the primary bottleneck this technique creates in a high-throughput production system, and how do you conceptually solve it?”
LLM System Design #6
Nov 5, 2025
AI Engineer interview at OpenAI
“Why don’t we just use a simple byte-based tokenizer? It has a fixed 256-token vocabulary, it’s simple, and it never has an ‘unknown’ token. Why are we still using a complex BPE tokenizer?”
LLM System Design #5
Nov 5, 2025
ML Engineer interview at Google
“Your team is struggling with training instability and exploding gradients in a new 100B+ model. The original ‘Attention Is All You Need’ paper used post-norm with learning rate warm-up. Why is that a bad idea for deep models, and what’s the one simple architectural change that solves this?”
LLM System Design #4
Nov 5, 2025
AI Engineer interview at Anthropic
“We have a fixed compute budget - 32 H100s for two weeks. To get the best possible model, should we train a larger model on less data, or a smaller model on more data? How do you begin to answer this?”
LLM System Design #3
Nov 5, 2025
AI Engineer interview at OpenAI
“The product team wants a 2x speedup on our Llama 3 70B endpoint, but they’ve forbidden any lossy techniques like quantization or pruning. How can you losslessly accelerate inference, and what core asymmetry in the Transformer are you exploiting?”
LLM System Design #2
Nov 5, 2025
AI Engineer interview at Anthropic
“We’re training a new model for the legal and medical domain. What’s the production risk of just using a standard, pre-trained Llama 3 tokenizer, and what’s your fix?”
LLM System Design #1
Nov 5, 2025