adlab-autodrive / ickd Goto Github PK

作者你好，请问这部分代码在哪里体现？

Hello. Thank you for your great work and sharing your code.

I run your code several times, but i cannot reproduce accuracy in paper.

I get 74.3% final valdiation accuracy in the setting(teacher : resnet32x4 / student : resnet8x4), but paper says 75.25% (ICKD without Hinton KD).

I also run ICKD-C(w/o KL) on the other settings, is it hyper parameter issue on diverse settings?(such as beta2 which multipied on CC Loss)

And I found some different procedure between paper and code, in calculating ICC matrices and L_CC.
In your code, row-wise L2 normalization has been applied on ICC matrices and Loss is divided by (channel * batch size) not (channel * channel * batch size)

Please let me know, if there are some implementation detail or error.

Thank you

Hello Author,

Thanks for sharing this great work.

There's been an issue with size mismatch when I run your code (ickd) using different arch type (e.g. teacher: resnet50, student: MobileNetV2). With same arch type (e.g. teacher: resnet110, student: resnet20), it works fine.

I'd appreciate it if you could check this out.

Thanks,

The paper described that
In terms of ImageNet, we use the AdamW optimizer [18] to train the network for 100 epochs with a total batch size of 256. The initial learning rate is 2e-4 reduced by 0.1 at epochs 30, 60, and 90.

However, on ImageNet, most papers (e.g., CRD) adopt SGD optimizer with an initial learning rate 0.1 on ResNet34-ResNet18 models.

Why the authors choose an uncommon AdamW optimizer on ImageNet?
Can you provide the results of ICKD with the same strategy as previous works for fair comparisons?

Thanks :)

I have run your code and found a question:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

I want to konw how to solve this questions, thank you!

Hello @ADLab-AutoDrive

Thank you for open-sourcing your work!
I wanted to reproduce the 72.19% in Table 2, which I assume corresponds to this configuration file

I found that the student model did not train from scratch as pretrained: True

Because it was not described in the paper and doesn't look like a fair comparison (other baselines in Table 2 trained student model from scratch), I thought it was a mistake and reran the experiment with pretrained: False, but the resulting accuracy was 68.34%, which is far from the reported result.

Could you clarify configs used in the paper?

when i edit the n_cls=100 to 2, it gave me the error below, how could i solve it? Thanks.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [0,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [1,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [2,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [3,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [4,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [5,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [6,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [7,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [8,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [9,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [10,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [11,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [12,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [13,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [14,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [15,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [16,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [17,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [18,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [19,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [20,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [21,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [22,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [23,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [24,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [25,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [26,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [27,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [28,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [29,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [30,0,0] Assertion t >= 0 && t < n_classes failed.
C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\Loss.cu:247: block: [0,0,0], thread: [31,0,0] Assertion t >= 0 && t < n_classes failed.
Traceback (most recent call last):
File "C:/Users/liaol/Desktop/ICKD/Cifar100/train_teacher.py", line 179, in
main()
File "C:/Users/liaol/Desktop/ICKD/Cifar100/train_teacher.py", line 126, in main
train_acc, train_loss = train(epoch, train_loader, model, criterion, optimizer, opt)
File "C:\Users\liaol\Desktop\ICKD\Cifar100\helper\loops.py", line 31, in train_vanilla
loss = criterion(output, target) #target
File "C:\Users\liaol\anaconda3\envs\tumor\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\liaol\anaconda3\envs\tumor\lib\site-packages\torch\nn\modules\loss.py", line 1150, in forward
return F.cross_entropy(input, target, weight=self.weight,
File "C:\Users\liaol\anaconda3\envs\tumor\lib\site-packages\torch\nn\functional.py", line 2846, in cross_entropy
return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: CUDA error: device-side assert triggered

请问作者能提供异构网络在cifar100上蒸馏上对结果吗？论文里是图标，没有具体数值啊，不好与你对比。