Seminars
When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse
LLM serving systems use prefix caching to reuse KV states across requests, yet almost universally default to LRU eviction. Yiyu will present a large-scale study of production traces showing that advanced and frequency-based policies fail to beat LRU, and outline design principles for closing the gap to the offline optimum.
Notes on Systems Research in the A(G?)I Era
AI agents are automating the design-and-evaluate loop of systems research while also emerging as a new workload with new requirements. Kostis will argue for an optimistic reading of this shift, presenting TuxBot for LLM-driven Linux tuning and speculative actions for cutting the latency of agent workflows.
Energy-Based Relational Reasoning
Humans naturally reason about relations such as cause and effect, and this reasoning is both bidirectional and compositional. Matthew will present energy-based relational programming, a purely neural framework that represents each relation with a smoothed energy function and composes them zero-shot at query time to solve reasoning problems.
Building Efficient Multi-GPU Systems for LLMs
Large language models increasingly rely on multiple GPUs, but communication overhead and synchronization limit their efficiency. Yang will present UCCL, a family of systems that improves LLM performance through efficient parallelism and extensible networking, reducing inference latency and improving communication throughput.
Networking for Distributed AI: design principles, technologies, standards, and a sample product
Distributed AI workloads use the network in unique ways, so traditional networking solutions are poorly suited to interconnecting the massive number of processors in AI training and inference clusters. Mario will analyze these networking requirements and describe the architectures and standard protocol stacks built for scale-out and scale-up AI clusters, concluding with how they come together in a commercial solution.
WeightBridge: An Efficient Weight Transfer Library for Reinforcement Learning
Weight transfer, the propagation of updated parameters from trainers to rollout generators, is becoming a key bottleneck in RL systems for LLMs, and existing solutions perform poorly or lack support across diverse layouts and synchronization modes. Xuanlin will present WeightBridge, a library that automatically maps trainer and rollout weight layouts to plan redundancy-free, load-balanced transfers, reducing average GPU stall time by up to 42× over a state-of-the-art open-source RL framework.
H-Spec: Parallel Speculative Decoding Without A Drafter-Side KV Cache
Speculative decoding accelerates LLM inference with a lightweight draft model, but block diffusion drafters pay per-request memory and KV-write overhead for a separate drafter-side KV cache that grows with concurrency. Weifan will present H-Spec, a hybrid Mamba-attention parallel drafter that reuses target KVs in place and injects target hidden states only at the last input position, removing the drafter-side cache while preserving draft quality.
Houston, We Have a Problem: Systems Challenges of Large Scale Spacecraft Deployments
Cheap launches have moved satellites from a few expensive fault-tolerant high-orbit craft to constellations of commodity SmallSats, but flight software still carries the ad-hoc infrastructure built for one-off missions. Haoda will present Orbital, which derives tight worst-case execution time bounds for hard real-time multi-tenant scheduling, and Radshield, a programming model that replicates execution to protect commodity hardware from radiation-induced silent data corruption.
KV Cache on Flash: What to Write When Writes Wear Out
Long agentic sessions make the KV cache the dominant memory cost in LLM serving, and flash offers capacity HBM cannot, but every block written spends a fraction of the drive’s lifetime, so a flash tier needs an admission policy and not only an eviction policy. Anshvardhan will present a study of that one-shot admission decision on a 222-day production trace of agentic coding traffic, including a learned prefill-time scorer and a two-tier implementation inside vLLM.
RouteWise: Latency–Cost Optimization for Multi-Provider LLM Routing
The same open-weight LLM is now served by many providers with very different pricing models and time-varying latencies, so routing requests well can cut cost and latency without touching model quality. Haoran will present RouteWise, a multi-provider router that unifies heterogeneous pricing and jointly optimizes cost and latency, cutting cost by 26.6% and mean time to first token by 58.4% against OpenRouter’s automatic routing.