Machine Learning System Design Interview, issue 26, May 14, 2026
The Inference Bottleneck Illusion
Senior ML Engineer interview at Meta, and the interviewer asks:
“You’ve built a two-tower recommendation system balancing high recall and high precision. The problem? It takes 400ms to run the pipeline, but product demands a strict 100ms SLA. Where do you cut latency without destroying the user experience?”
Why obsessing over model compression silently ignores your true RecSys latency killer, and how attacking network I/O and feature fetching actually hits your strict 100ms SLA.
The full answer, with the mechanism and the arithmetic, is for paid subscribers on Substack.