Gradient Checkpointing

What is gradient checkpointing?

When you train a neural network, backpropagation needs a bunch of intermediate results (activations) that the forward pass leaves behind. The bigger the model, the more of these pile up — and sooner or later you run out of GPU memory. Gradient checkpointing takes a dead-simple approach: don't store all those intermediate results. Keep only a few "checkpoints", and recompute the rest on the fly when backprop actually needs them.

Why does it save memory?

Memory and compute are two currencies you can trade
You can either spend memory to store intermediate results, or spend compute to recalculate them later. Gradient checkpointing picks the latter — a bit more number-crunching in exchange for a big cut in memory.
Only key snapshots are kept
It doesn't record every single step. It saves snapshots at certain points, and recomputes whatever's missing from the nearest snapshot when needed.

What's the catch?

You pay extra compute
Because of the recomputation, total work goes up by roughly 20% to 30%, and training gets a little slower.
But the trade is usually worth it
The same GPU can now fit a larger model or a bigger batch size, which often leads to better results and faster convergence overall.

When should you use it?

When you're training a big model and memory is the thing that won't fit, gradient checkpointing is a high-value trick. It's commonly used alongside mixed precision training and gradient accumulation — a standard move for squeezing more model out of the same hardware.

Bottom line: gradient checkpointing trades a little extra compute for a lot less memory, so your GPU stops being the reason you can't train a big model.