vLLM

What is vLLM?

vLLM is an open-source inference engine for large language models, built to answer one question: how do you let a single GPU serve more people, faster? Its core trick is called PagedAttention, which manages VRAM as precisely as an operating system manages memory — and throughput jumps as a result.

What hurts in traditional inference?

Fragmented VRAM
Every request reserves a chunk of VRAM for its KV cache, but requests vary in length. Reserve too much and you waste it; too little and you run short. Either way, memory ends up in fragments.
Requests queue and wait
Poor memory use means fewer simultaneous requests, so users just wait.

How does PagedAttention fix it?

Slice memory into pages
Like an OS using paging, it chops the KV cache into fixed-size blocks allocated on demand.
Store only what's actually used
Instead of reserving a big block per request, it allocates as much as needed — fragmentation nearly disappears.
Dynamic batching
With memory used well, it can serve more requests at once and smartly batch different ones together.

Why it matters

vLLM has pushed "how many users one card can serve" to a new level and sits behind many online model services. For anyone shipping an AI app, it means lower cost, higher concurrency, and shorter waits.

Bottom line: vLLM treats large-model inference like an OS treats memory — paging out the fragmentation so one GPU can serve more people.