Distributed Training

What is distributed training?

A single GPU has finite compute and memory. The moment your model is too big to fit — or too slow to wait for — you have to spread the work around. Distributed training splits the job into many pieces, hands them to multiple machines and GPUs running in parallel, and merges the results back together.

Why isn't one machine enough anymore?

Models keep getting bigger
From hundreds of millions to hundreds of billions of parameters, models have long outgrown a single card's memory.
You can't wait forever
Even when a model fits, training it on one GPU could take months or years — a non-starter commercially.
Data piles up too
With oceans of training text, a single machine can barely read it all, let alone learn from it.

How does it get faster?

Split the data (data parallelism)
Each device holds a full copy of the model and works on a different batch, then gradients get synced.
Split the model (model parallelism)
When the model won't fit on one card, its layers get spread across several.
Mix both
Real large-model training usually stacks data, model and pipeline parallelism together.

What's hard about it?

Communication overhead
Devices constantly sync gradients and parameters — a slow network becomes the bottleneck.
Keeping things consistent
Results across machines have to stay in sync, and debugging and fault-tolerance get a lot trickier.

What it unlocks

Distributed training turns "train a massive model" from impossible into an engineering problem. Every big model you've heard of is, underneath, thousands of GPUs working together.

Bottom line: distributed training is splitting a job too big for one person across a team — and making sure everyone's answers still line up.