The Large Batch Generalization Trap

Senior AI Engineer interview at Google DeepMind, and the interviewer asks:

“We just scaled our infrastructure to 4x our batch size (256 to 1024) to speed up training. We followed the 𝘓𝘪𝘯𝘦𝘢𝘳 𝘚𝘤𝘢𝘭𝘪𝘯𝘨 𝘙𝘶𝘭𝘦 and multiplied our 𝘓𝘦𝘢𝘳𝘯𝘪𝘯𝘨 𝘙𝘢𝘵𝘦 by 4. But our test accuracy still degraded. What fundamental property of SGD did we accidentally kill?”