Rethinking Bursty Workloads and KV Cache Hierarchies for Efficient LLM Serving

Abstract

Modern LLM inference workloads are bursty and exhibit heavy prefix reuse, and serving them efficiently requires understanding how we can turn these traits to our advantage. In this talk, I’ll present two recent projects: the first investigates how latency is impacted by bursty arrivals, and the second explores how we can make best use of a distributed KV cache. First, we’ll see how burstiness can improve the time-per-output-token of your LLM cluster and discuss the conditions which make this possible, as well as what it implies for request routing. Afterwards, we’ll look at how to control the private vs shared split of your KV cache to optimize time-to-first-token. Both projects show how intuitive systems choices can sometimes leave performance gains on the table, motivating the need to question seemingly obvious decisions.

Bio

Akira van de Groenendaal is a senior studying Computer Science at Carnegie Mellon University, currently doing research with the Harvard Systems Group under Prof. Juncheng Yang. He works on ML systems, with interests in workload analysis and the queueing dynamics of LLM inference.