Machine Learning System Design Interview, issue 49, Jun 6, 2026
The Cross-Entropy Trap
Senior ML Engineer interview at Google DeepMind, and the interviewer asks:
“We’re deploying a stateful neural network backbone that continuously updates on a non-stationary data stream. Our standard cross-entropy loss is massively accelerating catastrophic forgetting. What alternative representational objective do you implement to stop the bleeding, and why?”
Don’t say: “I’d use a regularization penalty like Elastic Weight Consolidation (EWC), lower the learning rate, or just freeze the early layers and retrain the classification head.”
Why optimizing for rigid classification boundaries overwrites historical data during continuous updates, and how enforcing stable latent topologies cures catastrophic forgetting at the source.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.