Blog
When Bursty Traffic Makes LLM Inference Faster
Akira van de Groenendaal
Bursty arrivals usually cause a drop in performance and are seen as headaches in production systems. In this blog, we investigate a phenomenon where burstiness actually improves performance, uncovering what conditions produce this effect.
Why FIFO Is (Almost) All You Need for Cache Eviction
Juncheng Yang
Cache eviction is usually presented as a ranking problem. LRU ranks objects by recency. LFU ranks them by frequency. More advanced algorithms combine several signals, adapt their parameters, and maintain increasingly sophisticated data structures to decide which object is least valuable.
MorphServe: Making Model Precision Elastic for Bursty LLM Serving
Juncheng Yang
Most LLM servers treat a deployed model as fixed: its weights have one precision, its KV cache has one memory budget, and both remain unchanged until the process restarts.