LLM Agents Interview Questions, issue 24, Mar 20, 2026
The Early RLHF Regression Trap
Senior AI Engineer interview at OpenAI, and the interviewer asks:
“You just started fine-tuning a highly capable base model to output internal reasoning thoughts before answering. In the first few PPO iterations, overall benchmark performance completely tanks compared to the base model. Why is this regression mathematically expected, and what forces the turnaround?”
Don’t say: “The learning rate is too high, or catastrophic forgetting is wiping out the base model’s weights. We need to lower the LR or use stricter LoRA constraints.”
Forcing chain-of-thought pushes the policy out-of-distribution, and until reward consistently beats the baseline, KL divergence mathematically suppresses performance.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.