The Context Leakage Trap

Principal ML Engineer interview at Anthropic, and the interviewer asks:

“We need to scale our RLHF dataset 10x. To maximize variety, we present human labelers with two model outputs generated from different user prompts (e.g., Response A to Prompt X vs. Response B to Prompt Y). We ask them to pick the better response.”