LLM System Design Interview, issue 9, Nov 7, 2025
The Memory Wall
Staff ML Engineer interview, and the interviewer asks:
“Your team needs to support a 10M context window. An engineer says it’s impossible because standard attention is O(N²) compute. Why is that the wrong bottleneck to focus on, and how does FlashAttention actually solve the real problem?”
Don’t say: “The O(N²) compute is the bottleneck. FlashAttention must be a sparse or approximate attention to reduce the FLOPs.”
Why attention isn’t too slow - it’s too hungry. How FlashAttention wins by trading O(N²) memory I/O for O(N²) fast compute.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.