Mixture of experts (MoE)
Uzmanlar karışımı (MoE)
D2 · Generative AI, capabilities and limitations
A mixture-of-experts architecture routes inputs to expert components and combines or selects their contributions.
Review status: 2026-12-05
Technical explanation
In sparse language-model implementations, only some experts are activated for a token, separating total from active parameters.
Conceptual boundaries
Experts are model components, not necessarily separate agents or human-defined subject specialists.
Provider-neutral example
Illustrative: a router sends each token to selected neural-network components before the model continues processing.
Limitations
Sparse activation can reduce active computation, while total and active parameter counts can differ; routing and communication introduce tradeoffs.
Related concepts
Atomic claims and evidence
1.1Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Source
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Source role
- Authoritative source
- Exact locator
- Abstract: selecting parameters for incoming examples, routing and training instability
- Supported claim
- Switch Transformers describe sparse expert routing and discuss communication costs and training instability.
- Last verification
- Review due
- Scope limitation
- This describes sparse MoE, not a requirement for every mixture model.
2.1Mixtral of Experts
- Source
- Mixtral of Experts
- Source role
- Authoritative source
- Exact locator
- Abstract: router selects two experts for each token; total and active parameters
- Supported claim
- Mixtral selects two experts per token in each layer and distinguishes total from active parameters.
- Last verification
- Review due
- Scope limitation
- Two experts is a Mixtral design choice, not a universal MoE count.