LLM System Design Interview, issue 1, Nov 5, 2025
The Tokenizer Trap
AI Engineer interview at Anthropic, and the interviewer asks:
“We’re training a new model for the legal and medical domain. What’s the production risk of just using a standard, pre-trained Llama 3 tokenizer, and what’s your fix?”
Don’t say: “It’s not optimal, but the tokenizer will just use subwords for the OOV (out-of-vocabulary) terms like ‘aneurysm’. The model will eventually learn the combinations during fine-tuning.”
Why reusing a general tokenizer can quietly 16x your compute bill - and how to avoid it.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.