Blog
From Requests to Sessions: Workload characterization of Agentic Workloads
Stop Waiting for Weights: Efficient RL Updates with WeightBridge
WeightBridge automatically bridges trainer and rollout layouts and reduces GPU stall time during RL weight updates by up to 42×.
A Year in LLM Serving: Workloads, Caching, and Routing
An analysis of 6.12 billion production LLM requests, examining workload evolution, traffic patterns, prefix reuse, cache eviction, and request routing.
Learning-Augmented Heuristics: A Different Way to Build Smart Caches
Smart cache eviction algorithms can adapt to workloads, but they often pay for it with complexity, instability, or overhead. Learning-Augmented Heuristics takes a different approach: keep the fast heuristic on the data path, and use learning to configure it at a slower timescale.
When Bursty Traffic Makes LLM Inference Faster
Bursty arrivals usually cause a drop in performance and are seen as headaches in production systems. In this blog, we investigate a phenomenon where burstiness actually improves performance, uncovering what conditions produce this effect.
Why FIFO Is (Almost) All You Need for Cache Eviction
Cache eviction is usually presented as a ranking problem. LRU ranks objects by recency. LFU ranks them by frequency. More advanced algorithms combine several signals, adapt their parameters, and maintain increasingly sophisticated data structures to decide which object is least valuable.
MorphServe: Making Model Precision Elastic for Bursty LLM Serving
Most LLM servers treat a deployed model as fixed: its weights have one precision, its KV cache has one memory budget, and both remain unchanged until the process restarts.
Pelikan Segcache: a memory-efficient and scalable in-memory key-value cache for small objects
In collaboration with Carnegie Mellon University, Twitter is building Pelikan’s next generation of storage backend, which enables high memory efficiency, high throughput, and scalability for workloads that are primarily small objects and widely use TTL (time-to-live).