Advanced Reinforcement Learning Interview Questions, issue 11, Feb 6, 2026
The Context Leakage Trap
Principal ML Engineer interview at Anthropic, and the interviewer asks:
“We need to scale our RLHF dataset 10x. To maximize variety, we present human labelers with two model outputs generated from different user prompts (e.g., Response A to Prompt X vs. Response B to Prompt Y). We ask them to pick the better response.”
Breaking prompt consistency in RLHF collapses the Bradley-Terry signal, causing the reward model to learn prompt difficulty instead of response quality.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.