The Tokenizer Speed Paradox

Senior ML Engineer interview at Anthropic, and the interviewer asks:

“Your BPE encode() is correct and passes every round-trip test. But tokenizing your 2TB pretraining corpus would take three weeks. Your tech lead says rewrite it in Rust. Why is that the second thing you should do?”

Don’t say: “Python is slow, so I’d port the hot loop to Rust or C++ and parallelize across cores.”