Machine Learning System Design Interview, issue 25, Dec 8, 2025
The Greedy Search Trap
Senior ML Engineer interview at OpenAI, and the interviewer asks:
“We need a low-latency geography trivia bot. Since the questions are factual, should we just use Greedy Search to save compute?”
The hidden failure mode behind greedy decoding and why minimal beam search fixes hallucinated factual answers.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.