The Python Streaming Trap

Senior Machine Learning Engineer interview at OpenAI, and the interviewer asks:

“You are tasked with training an LLM on a massive 2.8TB text dataset. How do you feed this to your PyTorch dataloader without instantly OOMing your system’s CPU RAM?”

Don’t say: “I’ll write a custom Python generator. I can open the file, yield the text chunks line-by-line to avoid loading it all at once, and stream it asynchronously into the training loop.”