| Blog | https://blog.kevinhu.me |
The second part of Grant Sanderson's video interview with myself on the cosmic distance ladder is now out: https://www.youtube.com/watch?v=hFMaT9oRbs4
I wrote a blog post with additional commentary and corrections on both videos at https://terrytao.wordpress.com/2025/02/13/cosmic-distance-ladder-video-with-grant-sanderson-3blue1brown-commentary-and-corrections/
#WeekendReads https://hao-ai-lab.github.io/blogs/distserve/
Or rather, it should better be named The Introduction to LLM Dynamic Batching Inference
TL;DR: LLM apps today have diverse latency requirements. For example, a chatbot may require a fast initial response (e.g., under 0.2 seconds) but moderate speed in decoding which only needs to match human reading speed, whereas code completion requires a fast end-to-end generation time for real-time code suggestions. In this blog post, we show existing serving systems that optimize throughput are not optimal under latency criteria. We advocate using goodput, the number of completed requests per second adhering to the Service Level Objectives (SLOs), as an improved measure of LLM serving performance to account for both cost and user satisfaction.
Terence Tao
