about train details about puzzlecam HOT 3 OPEN

Hello,
thanks for question.

I already put up prerequisite on my github page. You can refer to that.
When I make ResNet-50 train PASCAL VOC 2012 dataset, I don't fix the parameters of batch normalization. Specifically, the size of VRAM on 4 GTX 1080 Ti is approximately 42GB. However, the size of VRAM on 4 Titan RTX is approximately 96GB. As you know, batch normalization is very sensitive to batch size. So, your mIoU is reasonable. In conclusion, I recommend two ways which are to increase batch size or to replace normal batch normalization to fixed batch normalization.

Best regard,
Sanghyun

from puzzlecam.

Thanks for your reply, but i dont have the 96GB of VRAM. and i try the fix BN,the loss turned to NAN soon.
and I have noted that the syncBN will compute BN parametres of all devices and will it be
effective?

from puzzlecam.

@mouzhengdong I'm also getting NaN when setting fixed batch norm. Did you manage to fix this somehow?

from puzzlecam.