LLM Inference Interview Questions, issue 11, Aug 9, 2026
The Redundant Tool Paradox
Senior ML Engineer interview at Meta, and the interviewer asks:
“You’re building synthetic tool-call training data with Toolformer’s filter, keep the call if the tool output raises the likelihood of the correct continuation. Your eval improves. Production accuracy doesn’t move. What’s wrong with the filter?”
Don’t say: “The tool output could be wrong even if the probability improves.”
When higher eval scores just mean your model learned a copy shortcut. Why utility-under-the-prior is a flawed proxy, and how to mine the hard negatives your agents actually need to survive.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.