LLM Inference Interview Questions, issue 22, Aug 22, 2026
The Self-Doubt Trap
Senior AI Engineer interview at Anthropic, and the interviewer asks:
“Your agent triggers a web search whenever its reasoning trace shows uncertainty, ‘perhaps,’ ‘wait,’ ‘alternatively.’ It works great on your eval set and misses the worst failures in production. Why?”
Don’t say: “The threshold needs tuning”
Why building agent routing around a reasoning trace is a trap that ignores epistemic uncertainty, and how semantic entropy catches the silent errors your benchmarks miss.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.