The Attention Slicing Trap

Senior ML Engineer interview at Google DeepMind, and the interviewer asks:

“You doubled attention heads from 16 to 32 with d_model fixed, expecting the model to learn more patterns. Quality is flat and throughput dropped. What did you actually change, and what should drive head count instead?”

Don’t say: “More heads let the model attend to more relationships in parallel, so it should capture richer patterns.”