vLLM
High-throughput serving engine with PagedAttention
Tools
In plain English
An operating system memory manager for AI that stops memory from being wasted when lots of chats happen at once.
Technically
A high-performance LLM serving framework developed at UC Berkeley. Features PagedAttention (virtual memory management for KV cache), continuous batching, and chunked prefill, boosting serving throughput by 2x-4x.