Advanced Deep Learning Interview Questions, issue 13, Apr 3, 2026
The Per-Feature Learning Rate Trap
Senior ML Engineer interview at Google DeepMind, and the interviewer asks:
“You are training a dense recommender system. One feature dimension has violently massive gradient swings, while another dimension is completely sparse and barely updates at all. How do you stabilize it?”
Don’t say: “I’ll write a custom, per-feature learning rate schedule. I can manually drop the LR for the noisy dimension to 1e-5 to prevent explosions, and boost the sparse dimension to 1e-2 to force it to learn.”
Hardcoding per-dimension LRs ignores that gradient variance is non-stationary, causing optimizers to either freeze or explode as distributions shift mid-training.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.