Data Parallelism

What is data parallelism?

Data parallelism is the most intuitive kind of distributed training. The idea is simple: copy the full model onto every GPU, then chop the training data into pieces so each card chews through its own batch. When they're done, everyone's gradients get summed and averaged, and all copies update with the same averaged gradients.

How does it work?

Duplicate the model, split the data
Every card holds an identical model. The only difference is the data each one sees.
Sync the gradients
Each card computes its own gradients, then they're added and averaged across devices so everyone gets the same "unified answer" before updating locally.
It acts like a bigger batch
Four cards processing 32 samples each effectively behave like one step over 128 samples — a larger batch size.

Why it's great

Simple to implement
You barely touch the model structure. It's the first lever most people pull to speed up training.
Scales nicely
Add more cards and it goes faster — however many GPUs and however big a batch you want.

Where it falls short

The model has to fit
The catch is that each card must hold the whole model. If it's too big, data parallelism can't help — you need model parallelism instead.
Communication is the bottleneck
More cards mean more gradient syncing, and slow sync drags everyone down.
Batch size can't grow forever
Huge batches can hurt convergence, so you end up tuning things like the learning rate to compensate.

When to use it

When the model is small enough to fit on one card but you have lots of data and want to train faster, data parallelism is the easiest win. It's also the base layer in many large-model systems, with model and pipeline parallelism stacked on top.

Bottom line: data parallelism copies one model many times, gives each copy different data, then has everyone compare gradients before updating.