LLM System Design Interview, issue 24, Nov 22, 2025
Why Backprop Is 3× Harder Than You Think
Machine Learning Systems Engineer interview at Google DeepMind, and the interviewer asks:
“You’re asked to budget a training run. An intern engineer estimates the total FLOPs as 2 \ num_params \ num_tokens, arguing the backward pass is roughly symmetrical to the forward pass. Why is this cost estimate off by 300%, and what two distinct gradient calculations (totaling 4x, not 2x) are they failing to account for?”
Don’t say: “The estimate is low because they aren’t accounting for the optimizer step or the data loading overhead. The backward pass is technically the reverse of the forward pass, so the compute cost is the same, but you need to add a 20-30% buffer for Python overhead and GPU communication.”
Why intern engineers underestimate training FLOPs by 300% - and how the dual gradient calculations in backprop make the backward pass twice as expensive as the forward pass.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.