Speculative Decoding

What is speculative decoding?

When a large model writes text, it spits out one token at a time, and every step is expensive. Speculative decoding has a clever workaround: a small model drafts first, then the big model reviews the whole draft at once — keeping what's right and fixing what's wrong. Same result, several times faster.

How does it work?

The small model guesses first
A small, fast model quickly generates a chunk of candidate text — a few upcoming words.
The big model verifies in one pass
The large model reads the whole chunk at once and checks which guesses were right. The correct part is accepted; where it's wrong, it stops and corrects.
Where the savings come from
The big model validates several tokens in one go instead of generating one at a time — batching the expensive step.

Why is it useful?

Faster, not dumber
Since the big model has the final say, output quality is nearly identical to direct generation — just faster.
No architecture changes
No need to retrain the big model; you just pair it with a small one, so it's easy to ship.
Broadly useful
Translation, writing, code generation — all generative tasks benefit.

What are its limits?

The speedup depends on how well the small model guesses. Guess well and you verify a long stretch at once; guess poorly and you fall back to normal speed. The two models have to be well matched.

Bottom line: speculative decoding has a small model draft and a big model review, trading batched verification for speed.