The Execution Benchmark Trap

Senior AI Engineer interview at Meta, and the interviewer asks:

“We moved our coding eval from reference-similarity scoring to execution-based tests. This quarter, a model everyone agrees is stronger scores lower than last quarter’s model on the same Docker benchmark. What do you investigate before you call it a regression?”

Don’t say: “I’d look at the failed outputs and work out what the new model got wrong.”