LLM Inference Interview Questions, issue 7, Aug 5, 2026
The Summarization Paradox
Senior AI Infrastructure Engineer interview at Anthropic, and the interviewer asks:
“You enabled prompt caching on a 100-step agent trajectory expecting a 5–10x cost drop. In production you’re seeing barely 1.3x, and most per-step content is cache-missing. What’s actually happening, and why does the order of your context decide whether caching works at all?”
Don’t say: “Caching must be broken, or the provider isn’t caching enough of my prompt.”
How your cost-saving context condenser is secretly fighting your prompt cache, and why accepting an expensive hard reset is cheaper than constantly re-editing history.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.