Mixture of Experts (MoE)
Uzmanlar karışımı (MoE)
D2 · Generative AI, capabilities and limitations
A mixture-of-experts architecture routes inputs to expert components and combines or selects their contributions.
Review status: 2026-12-05
Technical explanation
In sparse language-model implementations, only some experts are activated for a token, separating total from active parameters.
Conceptual boundaries
Experts are model components, not necessarily separate agents or human-defined subject specialists.
Provider-neutral example
An MoE model activates selected expert components for each token instead of all of them.
Limitations
Sparse activation can reduce active computation, while total and active parameter counts can differ; routing and communication introduce tradeoffs.
Related concepts
Atomic claims and evidence
1.1Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Source
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Source role
- Authoritative source
- Exact locator
- Abstract: selecting parameters for incoming examples, routing and training instability
- Supported claim
- Switch Transformers describe sparse expert routing and discuss communication costs and training instability.
- Last verification
- Review due
- Scope limitation
- This describes sparse MoE, not a requirement for every mixture model.
2.1Mixtral of Experts
- Source
- Mixtral of Experts
- Source role
- Authoritative source
- Exact locator
- Abstract: router selects two experts for each token; total and active parameters
- Supported claim
- Mixtral selects two experts per token in each layer and distinguishes total from active parameters.
- Last verification
- Review due
- Scope limitation
- Two experts is a Mixtral design choice, not a universal MoE count.