The Semantic Leakage Trap

Senior AI Engineer interview at Anthropic, and the interviewer asks:

“Your production RAG system is suffering from severe semantic leakage. A user injects a biased, false premise in their prompt ( for example: ‘Since the sun is yellow from space...’), and the LLM blindly agrees, altering its output to match the bias. System prompts and few-shot examples aren’t stopping it. What is fundamentally happening at the attention layer to cause this sycophancy, and how do you architect a fix?”

Don’t say: “I’d write a stricter system prompt like ‘DO NOT agree with false user premises,’ tweak the temperature, or retrieve more RAG chunks to try and drown out the user’s bias.”