The L2 Optimization Trap

Senior LLM Pre-training Engineer interview at Google DeepMind, and the interviewer asks:

“We are setting up a single-epoch pre-training run on a 10-petabyte text corpus. A junior engineer removes the weight decay, arguing we can’t possibly overfit a dataset this large in one epoch. Do you merge their PR?”

Don’t say: “Yes, merge it. Weight decay is just L2 regularization designed to penalize complex weights and prevent overfitting. Since we are doing a single epoch over trillions of tokens, the model will never see the same data twice. Overfitting is mathematically impossible here, so weight decay just needlessly restricts the model’s capacity.”