The Argmax Deadlock Trap

Senoir Machine Learning Engineer interview at Google DeepMind, and the interviewer asks:

“We are building a massive Mixture of Experts (MoE) model. To maximize training throughput on our H100 clusters, we want to route each token to only the single best expert (k=1). Is this a valid strategy?”