QLoRA

What is QLoRA?

The biggest barrier to fine-tuning a large model is memory: a 70B model's weights alone run to hundreds of gigabytes — out of reach for most people. QLoRA's trick is to combine quantization with LoRA. First it quantizes the weights down to 4-bit so the base model becomes tiny, then it hangs a trainable LoRA add-on on top. The result: a single consumer GPU, like a 24GB 4090, can fine-tune models you'd never have dreamed of touching.

How does it save so much?

Quantization: compress the weights to 4-bit
Weights normally live in 16- or 32-bit. QLoRA stores them in 4-bit, cutting memory needs by an order of magnitude.
Decompress only when needed
The 4-bit values are temporarily restored to higher precision only for the step that needs them; the rest of the time they stay compressed in memory.
LoRA trains only a small add-on
The base stays frozen, and training only updates the low-rank matrices plugged in on top, so gradient memory shrinks too.

What else does it solve?

Makes personal fine-tuning possible
Fine-tuning a big model used to be for labs and giants. After QLoRA, individual researchers and startups can get going with a gaming GPU.
Quality close to full fine-tuning
The paper shows QLoRA trades blows with full fine-tuning on many tasks, at a fraction of the memory and time.

What should you watch with QLoRA?

It's still a "bang for your buck" method. Quantization brings a little precision loss, and on extremely demanding tasks full fine-tuning can still edge ahead. But for the vast majority of cases, QLoRA's convenience outweighs that cost.

Bottom line: QLoRA = a quantized base plus a LoRA add-on, so you can fine-tune a big model on a single consumer GPU.