Mixture of Experts
A layer made of many expert sub-networks plus a router that sends each token to only a few of them, so the model can be huge while each token uses a small part.
Also MoE · sparse model · router · gating network · conditional computation
Shazeer et al., 2017 (idea dates to Jacobs et al., 1991)
A hospital with many specialists. A receptionist sends each patient to the two most suitable doctors instead of every doctor seeing everyone. The hospital knows a lot, but each visit stays quick.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
Uses / built on
Applies to
Used by
How it works
In a standard Transformer, every token passes through every weight of each feed-forward layer. A mixture-of-experts layer replaces that one feed-forward network (MLP) with many smaller ones, the experts, plus a small router:
- For each token, the router scores all experts.
- Only the top one or two experts process that token.
- Their outputs are combined, weighted by the router’s scores.
The model’s total knowledge grows with the number of experts, but each token only pays for the few it visits. Experts often end up specializing loosely, for example in certain kinds of tokens or topics.
The catch: all experts still have to sit in memory, and training needs care so the router spreads tokens evenly instead of overloading a few favorite experts.
Everyday use cases
- Case 01
Large open language models
Several open-weight language models use MoE layers to get more capacity per unit of compute.
- Case 02
Translation at scale
Early large MoE models were built for multilingual translation.
Famous examples
- 2025DeepSeek-R1
An open-weight reasoning model from DeepSeek, released in January 2025 under the MIT license, trained with large-scale reinforcement learning to think step by step before answering.
- 2023Mistral AI Models
Language models from the Paris start-up Mistral AI, known for efficient open-weight releases such as Mistral 7B and the mixture-of-experts Mixtral 8x7B.
Strengths, weaknesses and when to use it
Strengths
- Much more capacity without proportionally more compute per token.
Weaknesses
- All experts must still fit in memory.
- Training is trickier: experts can be overloaded or left unused.
When to use itFor very large models where compute per token, not memory, is the main constraint.
Compare with
Check yourself
- Q01
Why can a mixture-of-experts model be huge yet fast per token?
Show answer
A: Each token is processed by only a few experts