LLM System Design Interview, issue 2, Nov 5, 2025
The Lossless Speedup Trick
AI Engineer interview at OpenAI, and the interviewer asks:
“The product team wants a 2x speedup on our Llama 3 70B endpoint, but they’ve forbidden any lossy techniques like quantization or pruning. How can you losslessly accelerate inference, and what core asymmetry in the Transformer are you exploiting?”
Don’t say: “Well, we could improve our batching strategy with vLLM’s PagedAttention...”
How top AI teams double inference speed without quantization, pruning, or touching model weights - by exploiting a core asymmetry in Transformers.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.