The Absolute Position Cliff

Senior ML Engineer interview at Google DeepMind, and the interviewer asks:

“Your model was trained with learned absolute positional embeddings at a 4K context. Product needs 16K for long agent traces. Someone wants to resize the position table and ship. What breaks at position 4,097, and why isn’t ‘just fine-tune on longer data’ the whole answer?”

Don’t say: “The model hasn’t seen those positions, so fine-tune on 16K data and it’ll learn them.”