Open Source Software (OSS) Inference Stack

Abstract

Open source inference systems have become critically important as models get bigger and bigger and agentic applications create demanding workloads. In this talk, we will discuss the key trends in model architecture and hardware accelerator server design and how open-source inference systems like vLLM and llm-d optimize performance against these trends.

Bio

Robert Shaw is a co-lead of the vLLM project and co-maintainer of the llm-d project. He works as a Member of the Technical Staff at Red Hat (formerly Sr Director of Engineering at Neural Magic (acquired by Red Hat)). He studied in undergrad in the CS department at Harvard.