LLM System Design Interview, issue 27, Apr 17, 2026
The Sequence Length Explosion Trap
Senior AI Engineer interview at Anthropic, and the interviewer asks:
“Byte-level tokenization elegantly gives us a perfectly lossless, predictable vocabulary size of exactly 256 with zero out-of-vocabulary errors. So why would pushing a pure byte-level tokenizer into a standard Transformer instantly bankrupt our compute budget?”
Don’t say: “Because individual bytes lack semantic meaning, making it much harder for the model to learn context compared to whole words.”
Why dropping to raw bytes feels elegant, but quietly multiplies your tokens, your FLOPs, and your inference cost beyond control.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.