The Stateless Benchmark Trap

Senior AI Engineer interview at Microsoft, and the interviewer asks:

“Your coding agent resolves most isolated issues on your internal benchmark. But after it lands 20 consecutive tasks in the same repo, the team wants it turned off. What did your eval miss, and how would you measure it?”

Don’t say: “The benchmark needs harder issues and better test coverage.”