想问一下关于CE_WEIGHT的问题 about logit-standardization-kd HOT 2 CLOSED

您好，因为我们继承的是DKD（ https://github.com/megvii-research/mdistiller/blob/a08d46f10d6102bd6e3f258ca5ac880b020ea259/mdistiller/engine/cfg.py ）的实验设置，而他们设置KD的CE_WEIGHT也是来自于Hinton的设置，即CE_WEIGHT=0.1、KD_WEIGHT=0.9；而其他算法都是其作者设计的CE_WEIGHT=1.0。在实验中我们其实也注意到了，CE_WEIGHT=1.0会比0.1略好一些，但是我们的实验是为了验证我们的logit标准化可以优化KD，所以保留了KD算法的CE_WEIGHT=0.1的设置

Hi, it is because we inherited the experimental setup of DKD (https://github.com/megvii-research/mdistiller/blob/a08d46f10d6102bd6e3f258ca5ac880b020ea259/mdistiller/engine/cfg.py) and the CE_WEIGHT of KD they set is also derived from Hinton's setting, i.e., CE_WEIGHT=0.1, KD_WEIGHT=0.9, while the other algorithms are designed by their authors, i.e., CE_WEIGHT=1.0.

In the experiment, we actually noticed that CE_WEIGHT=1.0 would be slightly better than 0.1. But our experiment was to verify that our logit standardization can improve those logit-based KD. So we kept the KD algorithm setting of CE_WEIGHT=0.1. Hope this help you.

from logit-standardization-kd.

感谢解惑！这对我非常有帮助！

from logit-standardization-kd.