Quantization

What is quantization?

Every parameter in a model is normally stored as a 32- or 16-bit floating-point number — precise, but heavy. Quantization compresses those numbers into 8-bit, 4-bit or even smaller integers. Like squashing a high-res photo into a smaller file: your eye barely notices, but the file is several times smaller.

Why does it help?

Saves memory
A 7-billion-parameter model takes about 28GB at 32 bits. At 4 bits it's roughly 3.5GB — an ordinary computer can hold it.
Speeds things up
Integer math runs faster than floating point, so inference gets noticeably quicker.
Lowers the bar
Quantization is why large models can run on phones and laptops instead of living only in the cloud.

Does it make the model dumber?

A little precision is lost, but usually not much. Good methods pick representative values, or keep the sensitive layers unquantized, to minimize the damage. In practice, 8-bit is nearly lossless, and 4-bit is acceptable on most tasks.

Common approaches

Post-training quantization (PTQ)
Compress after training — simple and fast.
Quantization-aware training (QAT)
Simulate quantization error during training so the model adapts early, for higher accuracy.

What does it unlock?

Quantization makes it possible for anyone to run a big model locally, and it's the underlying logic of formats like GGUF, AWQ and GPTQ. It's the bridge that carries big models out of the lab and onto everyone's device.

Bottom line: quantization slims a model down, trading a bit of precision for big savings in memory and speed.