Issue 8, Oct 22, 2026
The Sequential Recurrence Trap
Senior ML Engineer interview at Google DeepMind, and the interviewer asks:
“RNNs cost O(T) per sequence. Attention costs O(T²). So why did the field abandon the cheaper architecture for training large models, and why is that O(T) number misleading you?”
Don’t say: “Transformers are just better at long-range dependencies.”
Why saving FLOPs with O(T) complexity silently destroys your wall-clock training time and how linear state-space models finally bridged the gap between parallel training and O(1) decoding.
Share this trapShare on LinkedIn
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.