The Throughput–Latency Paradox

Senior ML Engineer interview at Anthropic, and the interviewer asks:

“Our ops team wants to 8x our batch size to cut costs and improve throughput. Why is this a dangerous move for user experience, and at what point does this strategy stop making sense... before you run out of memory?”