Router Network

What is a router network?

In a Mixture-of-Experts (MoE) architecture, one model doesn't do all the work alone. There's a crowd of "experts," each covering a different area, and a dispatcher at the door. The router network is that dispatcher: for each input, it quickly decides which experts should handle it and hands the task over. Route well, and the model is both strong and cheap.

How does it work?

Scoring
The router gives every expert a score for how relevant it is to this input.
Picking Top-k
Usually it selects just the top few experts — say Top-2 — instead of waking everyone up.
Weighted merge
The chosen experts produce their outputs, and the router combines them by weight into the final result.

Why do we need it?

Saves compute
Only a few experts activate per request, so lots of parameters don't mean lots of compute.
Right expert, right job
Different experts specialize in different areas; routing matches the right expert to the right question.
Scales
Want more capability? Add experts, and let the router organize them.

What are the challenges?

Uneven load
If tasks keep landing on a few "popular" experts, the rest sit idle — you need load-balancing.
Training instability
Routing decisions are discrete, so training needs tricks to let gradients flow properly.

Bottom line: the router network is MoE's control center — it sends each question to the right experts, keeping things fast and cheap.