The Tokenization Brittleness Trap

AI Engineer interview at Meta, and the interviewer asks:

“We deployed a Llama-3 based app. We removed a single whitespace in the prompt template, and our benchmark accuracy tanked by 12%. Why is the model so brittle to a simple format change, and why didn’t instruction tuning prevent this?”

Don’t say: “The model is confused by the bad grammar.”