Knowledge distillation
Bilgi damıtma
D2 · Generative AI, capabilities and limitations
Knowledge distillation trains a student model using information supplied by a teacher model or ensemble.
Review status: 2026-12-05
Technical explanation
The student learns from teacher outputs or representations; this can support a more compact model.
Conceptual boundaries
Distillation transfers learned behavior through training; quantization changes numerical representation.
Provider-neutral example
Illustrative: a team trains a compact text classifier using a larger classifier’s predictions, then evaluates it on held-out cases.
Limitations
Compression results in particular studies do not guarantee that every student retains all teacher capabilities.
Related concepts
Atomic claims and evidence
1.1Distilling the Knowledge in a Neural Network
- Source
- Distilling the Knowledge in a Neural Network
- Source role
- Authoritative source
- Exact locator
- Abstract: compressing ensemble knowledge into one model
- Supported claim
- Hinton and colleagues describe transferring ensemble knowledge into a single model that is easier to deploy.
- Last verification
- Review due
- Scope limitation
- This is a foundational distillation formulation, not a requirement to use an ensemble.
2.1TinyBERT: Distilling BERT for Natural Language Understanding
- Source
- TinyBERT: Distilling BERT for Natural Language Understanding
- Source role
- Authoritative source
- Exact locator
- Abstract: teacher-student knowledge transfer and two-stage distillation
- Supported claim
- TinyBERT studies teacher-student transfer for compressing Transformer language models.
- Last verification
- Review due
- Scope limitation
- Reported efficiency and accuracy concern the evaluated models and tasks.