Issue 18, Feb 13, 2026
The Gumbel-Softmax Trap
Senior AI Engineer interview at Google, and the interviewer asks:
“If the discrete token bottleneck is the main reason we need RLHF, why not just use the Gumbel-Softmax trick to make sampling differentiable and backpropagate end-to-end?”
What works as a differentiable trick in low-dimensional models becomes a VRAM-destroying, gradient-unstable nightmare when your action space is a 100k-token vocabulary.
Share this trapShare on LinkedIn
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.