LLM Inference Interview Questions, issue 5, Aug 3, 2026
The Verifier Trap
AI Agents Engineer interview at Anthropic, and the interviewer asks:
“You want to evaluate whether your coding agent actually solved a task. A teammate says: skip the learned critic, just train a model to generate unit tests and let pass/fail decide. It’s system-agnostic and elegant. Why isn’t this the default?”
Don’t say: “Great idea, tests are ground truth, so pass = solved.”
Why relying on pass/fail tests for agent evaluation doesn't remove the critic, it just hides it, and the "test the tests" regime decision that elite teams use instead.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.