Advanced Reinforcement Learning Interview Questions, issue 12, Feb 7, 2026
The OOD Extrapolation Trap
Senior RL Engineer interview at Google DeepMind, and the interviewer asks:
“We have 50TB of static historical logs. If we run a standard off-policy algorithm (like Soft Actor-Critic) on this buffer without collecting new data, what happens to the Q-values?”
In static datasets, Q-functions assign arbitrary value to unseen actions, and policy optimization amplifies that noise into catastrophic overestimation.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.