Mixed Precision Training

What is mixed precision training?

Traditionally, a neural network's weights and math all run in 32-bit floating point (FP32): precise, but slow and memory-hungry. Mixed precision training takes a "be precise where it matters, cheap everywhere else" approach: most of the computation switches to 16-bit floating point (FP16) for speed and lower memory, while a few precision-sensitive spots fall back to FP32.

Why is FP16 faster?

It computes quicker
On modern GPUs, FP16 matrix math runs several times faster than FP32, and hardware built for half precision (like Tensor Cores) can multiply throughput.
It saves memory and bandwidth
Half the bits means weights and intermediate results take less space, so you can fit a bigger model or a bigger batch.

Why not use FP16 everywhere?

Precision can overflow or vanish
FP16 has a limited range and precision: big numbers overflow to infinity, tiny gradients get flushed to zero, and training becomes unstable or won't converge.
That's where "mixed" comes in
Run the tricky parts — gradient updates, certain sums — in FP32, and everything else in FP16, and you get speed and stability together.

A couple of key tricks

Loss scaling
Gradients too small to survive FP16? Multiply the loss by a factor before backprop, then scale back down, so the small gradients don't get wiped out.
Keep master weights in FP32
Store weights in FP32, compute in FP16, and return to FP32 for the update — best of both worlds.

Is it worth it?

These days mixed precision is basically the default for training large models: roughly double the speed, half the memory, and precision loss you can ignore. Combined with tricks like gradient checkpointing, it often makes models that wouldn't fit runnable at all.

Bottom line: mixed precision training runs most of the math in FP16 for speed and keeps the critical bits in FP32 for accuracy — fast, cheap, and it doesn't fall apart.