The Step-Function Accuracy Trap

Senior AI Engineer interview at Meta, and the interviewer asks:

“We scaled from 33B to 65B. Validation loss is clearly lower, but few-shot benchmarks barely moved, and one task got worse. Is the 65B model better? Which signal do you trust?”

Don’t say: “Trust the benchmarks. Loss is just a training number.”