Knowledge Distillation Loss
Medium · Solved in PyTorch · Machine learning coding practice
Problem
Knowledge distillation (Hinton et al., 2015) allows us to compress large models into smaller, faster ones while maintaining high accuracy. The key insight is that the student learns not just from hard labels (one-hot encoded), but from the soft probability distributions produced by the teacher, which encode valuable "dark knowledge" about the relationships between classes.
Topics: model-compression, knowledge-distillation, loss-functions, transfer-learning
The full statement, worked examples, hints and the test suite are available once you sign in. You can then solve Knowledge Distillation Loss in the browser and run it against the tests.