Mixture of Experts

What is Mixture of Experts?

A traditional large model runs every parameter on every question — like an office where any request wakes up every department. Mixture of Experts (MoE) splits the model into many small "expert" networks and only calls the few most relevant experts for each input, letting the rest rest. Faster and cheaper.

How does it pick the experts?

A router
A scheduler — think of it as a doorman — glances at the input and decides which experts should handle it.
Only a few fire
A model might have a hundred experts but activate only two or three per input, cutting the compute way down.

Why does that save money and boost speed?

Many parameters, less compute
MoE can hold enormous knowledge in its parameters, yet each pass only touches a small slice — so it runs fast.
Experts specialize
One expert is great at code, another at math. Each owns a lane, and together they're stronger.

What's the catch?

More experts mean more memory (all of them have to be stored) and trickier training and routing. Load imbalance is common too — one expert gets called over and over while others sit idle. That takes real engineering to smooth out.

Bottom line: Mixture of Experts keeps a big army but sends small squads to work — a few experts' worth of compute, a giant model's worth of ability.