Top-p Sampling

What is Top-p sampling?

When a model picks its next word, there can be thousands of candidates. Top-p sampling — also called nucleus sampling — sorts words by probability, highest first, then keeps adding them until the cumulative probability reaches p, say 0.9. Then it picks at random only within that “core”.

How is it different from Top-k?

Top-k always takes the top k words
No matter what the distribution looks like, it grabs the k most likely words. The trouble is choosing k: too small and you lose creativity, too big and junk sneaks in.
Top-p looks at cumulative probability
It doesn't count words, it cares how much probability is covered. A concentrated distribution gives a small pool, a spread-out one gives a large pool — it adapts on its own.

What problem does it solve?

Natural-language probabilities are wildly uneven. Sometimes the top two words cover 90%, sometimes you need a hundred. A fixed Top-k can't serve both cases; Top-p uses probability coverage as its ruler and resizes the candidate pool dynamically, keeping output steady without feeling stiff.

How do you use it in practice?

It usually shows up alongside temperature: adjust temperature to reshape the distribution, then use Top-p to draw the boundary. Many APIs default to p = 0.9 or 1.0. Lower p for more stability, higher p for more variety — and always tune it together with temperature.

Bottom line: Top-p draws words from a “probability circle” — and the circle's size depends on how spread out the candidates are.