KV Cache

What is a KV cache?

A language model writes text one token at a time. Each new token has to "settle accounts" with every token that came before through attention. The KV cache stores the results of that accounting — the already-computed keys and values — so nothing gets recomputed. Generation speeds up by several times.

Why is there repeated work in the first place?

Autoregressive generation is one token at a time
Every token the model spits out makes it recompute attention against everything before it.
What's already computed hasn't changed
The part that's been generated never gets rewritten, so its keys and values are fixed. No need to recompute them each step.

How does it speed things up?

Compute the new, reuse the old
For a new token, attention only needs the new query plus the historical keys and values — everything historical comes straight from cache.
Memory for time
The cost is storing all those K and V values (which eats a lot of VRAM), in exchange for much faster inference.

Why it matters

The KV cache is one of the foundations of modern LLM inference, and basically every production inference engine uses it. Understand it and you've found the common answer to two familiar complaints: "why does generation get slower" and "why is my VRAM always full."

Bottom line: a KV cache means "what's been computed stays computed" — keep the history's attention ledger, and only settle new entries as you generate.