LLM System Design Interview, issue 15, Nov 12, 2025
The FLOPs Fallacy
Senior AI Engineer interview at Meta, and the interviewer asks:
“You’re A/B testing two 70B models - one Multi-Head Attention (MHA), one Grouped Query Attention (GQA). Your colleague argues they’ll have the same inference speed since FLOPs and parameter counts are identical. Is this assumption correct?”
Don’t say: “Well, the FLOPs are similar, so maybe the GQA implementation has better kernel fusion or is just slightly more optimized...”
Why two 70B models with identical compute still run at different speeds - and how GQA wins by breaking the memory bottleneck.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.