Machine Learning System Design Interview, issue 20, Dec 5, 2025
The Vanishing Update Paradox
Senior ML Engineer interview at OpenAI, and the interviewer asks:
“Our LoRA fine-tuning isn’t capturing the domain complexity. We increased the rank 𝐫 from 8 to 256 to give the model more capacity. But the loss curve flatlined. Why?”
Why increasing LoRA rank from 8 to 256 kills learning - and how rsLoRA fixes gradient collapse.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.