Parameter-Efficient Fine-Tuning

What is parameter-efficient fine-tuning?

Traditional fine-tuning touches every parameter in the model. Updating hundreds of millions — or hundreds of billions — of weights eats memory and compute. Parameter-efficient fine-tuning (PEFT) flips that: freeze the big model and train only a tiny set of new parameters, sometimes just a few hundred thousand, to pick up the new task.

Why can less be more?

The base model is already strong
The language skills and common sense from pre-training are general-purpose. A new task is usually just "add a direction" on top of what's there, so you don't retrain from scratch — you nudge a small part.
The new parameters take the job
Methods like LoRA insert low-rank matrices into the model, like bolting a lightweight add-on onto the base. Only that add-on gets trained.

What are the common approaches?

LoRA
Adds low-rank decomposition to the weights — tiny training load — and the extra matrices can be merged back in or swapped out.
Adapters
Small modules dropped between layers; only those get trained.
Prefix / prompt tuning
Only a learnable "soft prompt" is trained, without touching the inside of the model at all.

Why did it change fine-tuning?

It used to be "one full model per task" — expensive and hard to manage. With PEFT, one base model can host endless lightweight "plugins", each saved as a file of a few megabytes, swapped and merged on demand. That puts fine-tuning within reach of individual developers and small teams.

Bottom line: PEFT freezes the big model and trains only a small handful of new parameters, turning it into the specialist you want for a fraction of the cost.