The ReLU RNN Trap

Senior ML Engineer interview at Google, and the interviewer asks:

“To fix vanishing gradients, a junior engineer opens a PR that swaps tanh for ReLU in your RNN. Their argument: ReLU’s derivative isn’t squashed below 1. Do you approve it?”

Don’t say: “Yes. ReLU fixed vanishing gradients in deep CNNs, so it’ll fix them here too.”