LLM Agents Interview Questions, issue 23, Mar 19, 2026
The CoT Self-Verification Trap
Senior AI Engineer interview at OpenAI, and the interviewer asks:
“Your LLM is generating long, list-based responses. It’s nailing the broad concepts, but constantly hallucinating specific entities, like slipping Michael Bloomberg into a list of politicians born in New York. Standard think step-by-step prompting is failing. How do you stop this?”
Don’t say: “I’d just lower the temperature and add a stricter ‘double-check your facts’ clause to the system prompt.”
Chain-of-thought fails on entity accuracy because the model reuses its own corrupted context instead of querying an uncontaminated signal.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.