Advanced NLP Interview Questions, issue 13, Dec 19, 2025
The Knowledge Distillation Trap
Senior AI Engineer interview at OpenAI, and the interviewer asks:
“We need to distill a massive 10-model ensemble into a single small model for low-latency serving. Why is training the student on the ensemble’s final output tokens a complete waste of compute?”
Don’t say: “It’s not a waste. The ensemble is the ‘teacher.’ If the ensemble predicts Cat with 99% confidence, we should treat Cat as the ground truth and train the student to predict it using standard Cross-Entropy Loss.”
Why training on ensemble hard targets throws away the most valuable signal—and how dark knowledge actually transfers intelligence.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.