The Curriculum Learning Trap

Staff Research Scientist interview at DeepSeek, and the interviewer asks:

“We have three massive datasets: 𝘎𝘦𝘯𝘦𝘳𝘢𝘭 𝘛𝘦𝘹𝘵, 𝘚𝘰𝘶𝘳𝘤𝘦 𝘊𝘰𝘥𝘦, and 𝘴𝘱𝘦𝘤𝘪𝘢𝘭𝘪𝘻𝘦𝘥 𝘔𝘢𝘵𝘩 𝘱𝘳𝘰𝘣𝘭𝘦𝘮𝘴. To build a State-of-the-Art Math reasoner, in what order do you feed this data during pre-training, and why?”

Don’t say: “Just shuffle them all together into one big dataset to avoid catastrophic forgetting.”