The Dimensionality Trap

Senior AI Engineer interview at Google DeepMind, and the interviewer asks:

“You scaled your pre-training dataset by 10x, but the error rate barely budged. Your model is massively over-parameterized, so capacity isn’t the issue. What ‘intrinsic’ statistical property of your target task is fundamentally bottlenecking your power-law returns?”

Don’t say: “It is a data quality issue. The new 10x scrape is full of low-entropy garbage, or the duplication rate is too high. We need to run strict MinHash deduplication, upsample high-quality sources, or drop our learning rate to escape a local minimum.”