Advanced NLP Interview Questions, issue 17, Dec 23, 2025
The Sparse Gradient Trap
Senior ML Engineer interview at OpenAI, and the interviewer asks:
“We’re training a model on a massive vocabulary. Some critical domain terms appear only once every 10,000 documents. Why will standard SGD fail to learn weights for these rare features, and how does Adam specifically fix this?”
Don’t say: “Adam is better because it uses momentum to converge faster.”
Why SGD silently ignores rare tokens - and how Adam rescues them with variance normalization.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.