Why ‘Train on the Internet’ Guarantees a Trash Model

AI Engineer interview at OpenAI, and the interviewer asks:

“A project plan budgets 1 day for data prep: 𝘋𝘰𝘸𝘯𝘭𝘰𝘢𝘥 𝘊𝘰𝘮𝘮𝘰𝘯 𝘊𝘳𝘢𝘸𝘭. Why is this 𝘵𝘳𝘢𝘪𝘯 𝘰𝘯 𝘵𝘩𝘦 𝘪𝘯𝘵𝘦𝘳𝘯𝘦𝘵 mindset a complete fantasy that guarantees a 𝐭𝐫𝐚𝐬𝐡 model ?”

Don’t say: “Because the raw data is low-quality. You need to apply quality filters to remove spam, filter out harmful content, and deduplicate the data so the model doesn’t memorize web pages.”