Machine Learning System Design Interview, issue 38, May 26, 2026
The Retraining Window Fallacy
Staff Machine Learning Engineer interview at Netflix, and the interviewer asks:
“Your recommendation system works perfectly in staging. But once it goes live, users request products and media that fall completely outside your model’s 30-day training window. How do you handle these out-of-vocabulary (OOV) categories safely at scale without breaking downstream inference SLAs?”
Don’t say: “Just write a hardcoded default fallback rule to recommend top-trending items, or simply expand the model’s training window to 6 months to capture everything.”
The dangerous trap of solving data volatility with frequent model updates, and the low-latency external lookup trick that safely routes out-of-vocabulary features.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.