Knowledge Distillation
Bilgi damıtma
D2 · Generative AI, capabilities and limitations
Knowledge distillation trains a student model using information supplied by a teacher model or ensemble.
Review status: 2026-12-05
Technical explanation
The student learns from teacher outputs or representations; this can support a more compact model.
Conceptual boundaries
Distillation transfers learned behavior through training; quantization changes numerical representation.
Provider-neutral example
A compact model learns from a larger model's answers to classify support tickets.
Limitations
Compression results in particular studies do not guarantee that every student retains all teacher capabilities.
Related concepts
Atomic claims and evidence
1.1Distilling the Knowledge in a Neural Network
- Source
- Distilling the Knowledge in a Neural Network
- Source role
- Authoritative source
- Exact locator
- Abstract: compressing ensemble knowledge into one model
- Supported claim
- Hinton and colleagues describe transferring ensemble knowledge into a single model that is easier to deploy.
- Last verification
- Review due
- Scope limitation
- This is a foundational distillation formulation, not a requirement to use an ensemble.
2.1TinyBERT: Distilling BERT for Natural Language Understanding
- Source
- TinyBERT: Distilling BERT for Natural Language Understanding
- Source role
- Authoritative source
- Exact locator
- Abstract: teacher-student knowledge transfer and two-stage distillation
- Supported claim
- TinyBERT studies teacher-student transfer for compressing Transformer language models.
- Last verification
- Review due
- Scope limitation
- Reported efficiency and accuracy concern the evaluated models and tasks.