The Data Lineage Illusion

Senior AI Engineer interview at OpenAI, and the interviewer asks:

“You just appended 1 million newly hand-labeled samples to your pristine 100K training dataset to scale performance, but your production accuracy immediately dropped. The schemas match perfectly and there are no formatting errors. What went wrong?”

Don’t say: “It’s a classic data leakage or hyperparameter issue. You probably overfit to the original 100K distribution, or your batch size was too small for the new 1.1M payload. Just drop your learning rate to 1e-5, re-balance the classes, or use a data validation framework to check for feature drift.”