#LLM
When Bursty Traffic Makes LLM Inference Faster
Akira van de Groenendaal
Bursty arrivals usually cause a drop in performance and are seen as headaches in production systems. In this blog, we investigate a phenomenon where burstiness actually improves performance, uncovering what conditions produce this effect.
MorphServe: Making Model Precision Elastic for Bursty LLM Serving
Juncheng Yang
Most LLM servers treat a deployed model as fixed: its weights have one precision, its KV cache has one memory budget, and both remain unchanged until the process restarts.