This repository includes official implementation for the following papers:
-
NKD and tf-NKD: Rethinking Knowledge Distillation via Cross-Entropy
-
ViTKD: ViTKD: Practical Guidelines for ViT feature knowledge distillation
It also provides unofficial implementation for the following papers:
If this repository is helpful, please give us a star โญ and cite relevant papers.
- Prepare the dataset in data/imagenet
-
# Set environment pip install -r requirements.txt pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
- This repo uses mmcls = 0.23.2. If you want to use lower mmcls version for distillation, you can refer MGD to change the codes.
- Please refer nkd.md and vitkd.md to train the student and get the weight.
- You can modify the configs to choose different distillation methods and pairs.
- The implementation details of different methods can be seen in the folder distillation.
Our code is based on the project MMClassification.