The Harness Overfitting Trap

Senior AI Engineer interview at Google DeepMind, and the interviewer asks:

“You fine-tuned an agent on 50K trajectories, all collected through one agent harness. A customer runs it in a different harness on the same kind of task, and performance collapses. Same weights, same task. What did your model actually learn?”

Don’t say: “The new harness has a different system prompt. Tweak the prompt, add a few examples, maybe collect more data.”