Advanced NLP Interview Questions, issue 18, Dec 24, 2025
The Perplexity Tokenizer Trap
Senior AI Engineer interview at Google, and the interviewer asks:
“We ran an eval on a fixed dataset. Llama3 achieved a perplexity of 2.1, while Gemma3 scored 2.4. Which model is the better probability estimator, and which one do we deploy?”
Don’t say: “Llama3 is better. Perplexity measures how ‘confused’ the model is, so a lower score means it understands the text distribution better. We should deploy Llama3.”
Why lower perplexity across models can be pure metric illusion - and how tokenizer choice silently breaks probability comparisons.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.