Advanced NLP Interview Questions, issue 14, Dec 20, 2025
The Tokenization Brittleness Trap
AI Engineer interview at Meta, and the interviewer asks:
“We deployed a Llama-3 based app. We removed a single whitespace in the prompt template, and our benchmark accuracy tanked by 12%. Why is the model so brittle to a simple format change, and why didn’t instruction tuning prevent this?”
Don’t say: “The model is confused by the bad grammar.”
Why deleting a single whitespace can cause a 12% accuracy drop and why instruction tuning doesn’t actually make LLMs robust.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.