Advanced NLP Interview Questions, issue 19, Dec 25, 2025
The QLoRA Compute Tax Trap
Senior AI Engineer interview at NVIDIA, and the interviewer asks:
“We switched from standard FP16 fine-tuning to QLoRA (4-bit quantization) to save memory. The model fits now, but training speed hasn’t improved, it’s actually slightly slower. Why didn’t reducing precision by 4x result in a 4x speedup?”
Don’t say: “That sounds like a bug. Since 4-bit integers are 4x smaller than 16-bit floats, we are moving less data. Less memory bandwidth usage always equals faster training.”
Why 4-bit quantization saves VRAM but can slow down training on A100s and H100s.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.