LLM System Design Interview, issue 47, May 10, 2026
The Grid Search Trap
Senior Pre-training Engineer interview at Google DeepMind, and the interviewer asks:
“Compute is tight, but you need to find the exact optimal ratio of code, web, and book data for a 100B parameter model. How do you empirically determine the perfect mixture without wasting millions of dollars running ablation tests at massive scale?”
Don’t say: “I’d train a grid of 1B parameter models on different data mixtures, evaluate them on downstream benchmarks like MMLU and HumanEval, and just pick the mixture that scores the highest. Then we scale that winning ratio up to 100B.”
The hidden reason testing data mixtures at the 1B scale is computationally reckless - and how exploiting the log-log scaling offset lets you perfectly transfer optimal weights to your 100B run.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.