The Two Faces of Inference

ML Engineer interview at Google, and the interviewer asks:

“We need to serve our model for two different use cases: a low-latency chatbot that needs a fast 𝐓𝐢𝐦𝐞-𝐭𝐨-𝐅𝐢𝐫𝐬𝐭-𝐓𝐨𝐤𝐞𝐧 (𝐓𝐓𝐅𝐓), and a high-throughput batch summarization job. How do these two workloads stress the GPU differently, and what fundamental tradeoff are you managing?”

Don’t say: “For the chatbot, I’d use a small batch size (like 1) for low latency. For the batch job, I’d use a large batch size for high throughput.”