Advanced Reinforcement Learning Interview Questions, issue 23, Feb 18, 2026
The State Visitation Trap
Senior Robotics Engineer interview at OpenAI, and the interviewer asks:
“We want to minimize human interventions. So, we increase the episode length (H) from 1k to 100k steps in our SAC agent. We are collecting 100x more data per reset. Why does the policy’s success rate collapse to near-zero?”
Long episodes don’t give you better data, they give you skewed data, where off-task trajectories dominate and overwrite the transitions that actually solve the problem.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.