Model Parallelism

What is model parallelism?

Data parallelism has a hidden assumption: every card can hold the whole model. But when a model reaches tens or hundreds of billions of parameters, no single GPU can fit it. Model parallelism takes the other route — it slices up the model itself, placing different layers (or different parts of a layer) on different GPUs so they cooperate to compute one giant model.

How is it different from data parallelism?

Data parallelism splits the data
Each card holds a full copy and processes different samples.
Model parallelism splits the model
The model is carved into pieces, each card owns one piece, and data flows through all the cards in order.
They solve different problems
Data parallelism fixes "it's too slow"; model parallelism fixes "it doesn't fit".

Two common ways to slice

By layer (inter-layer)
Layers 1–10 go on GPU A, 11–20 on GPU B. Data passes through A then B, like an assembly line.
By tensor (tensor parallelism)
Split the matrix math inside a single layer across cards, compute in parallel, then stitch the result back — often used for the big attention matrices in transformers.

What's tricky about it?

Heavy communication
Data shuttles constantly between layers and tensors, so you need high bandwidth between cards.
Uneven load
If the pieces don't have equal compute, fast cards sit idle waiting for slow ones.
Harder to implement
Unlike data parallelism's near plug-and-play, this needs more careful engineering.

Why it's irreplaceable

Almost every ultra-large model trained today leans on model parallelism. Without it, a model with hundreds of billions of parameters couldn't even be loaded, let alone trained.

Bottom line: model parallelism is what you reach for when the model won't fit — slice it up and have several cards lift it together.