The KV Cache Memory Trap

Senior ML Engineer interview at Google, and the interviewer asks:

“With a KV cache, each decode step only computes Q, K, V for the newest token. So it’s a free speedup, right? What did you actually trade away?”

Don’t say: “KV cache stores past keys and values so we don’t recompute them. Generation gets faster.”