Advanced Reinforcement Learning Interview Questions, issue 19, Feb 14, 2026
The Small-Batch Policy Gradient Trap
Senior RL Engineer interview at OpenAI, and the interviewer asks:
“We collected 6 robot trajectories. 5 failed (low reward). 1 succeeded (high reward). We run a vanilla Policy Gradient update on this small batch. What happens to the gradient?”
When N is small and rewards are strictly positive, the gradient pushes up both failures and successes, turning learning into a variance-dominated coin flip.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.