Pipeline Parallelism

What is pipeline parallelism?

Model parallelism slices the model by layer, and data flows through each card in order. The catch: at any moment only one card is busy, and the rest are twiddling their thumbs. Pipeline parallelism fixes that idle time — it feeds many batches through like a factory line, one after another, so each card grabs the next batch the moment it finishes the last. Device utilization climbs sharply.

How to picture it

Think car assembly line
While station one bolts the chassis, station two is already fitting the engine of the previous car, and station three is painting an even earlier one. No station sits idle, and finished cars keep rolling off the line.
Model layers are the stations
Different layers sit on different GPUs, and batches of samples are the "cars" on the line.

What problem does it solve?

It shrinks the "bubbles"
If you feed one batch at a time, later cards sit idle while earlier ones compute, then everyone waits again on the way back — leaving idle gaps called "bubbles".
It fills the gaps with more batches
Split the data into many small micro-batches and let them flow nose-to-tail through the cards. Later batches fill the idle time, and throughput goes up.

The cost and the trade-offs

More memory used
With several batches on the line at once, each card holds intermediate state for multiple batches, pushing up memory.
Trickier scheduling
You have to choose how many micro-batches and how to lay them out so the pipeline stays full without blowing up memory.

Where it fits in big models

Pipeline parallelism is usually used together with model and data parallelism — part of the combo that trains ultra-large models. It keeps hundreds or thousands of GPUs busy and nudges training speed even higher.

Bottom line: pipeline parallelism keeps a model running like a factory line that never stops — batches flow nose-to-tail and fill every idle gap.