The Per-Feature Learning Rate Trap

Senior ML Engineer interview at Google DeepMind, and the interviewer asks:

“You are training a dense recommender system. One feature dimension has violently massive gradient swings, while another dimension is completely sparse and barely updates at all. How do you stabilize it?”

Don’t say: “I’ll write a custom, per-feature learning rate schedule. I can manually drop the LR for the noisy dimension to 1e-5 to prevent explosions, and boost the sparse dimension to 1e-2 to force it to learn.”