训练技巧提问 about pns-net HOT 2 CLOSED

Hi, @Moqixis

Sorry for the late reply. Since we got a very, very limited video dataset for this task before submission, we adopted two-stage training to achieve promising performance. But, in general, the two-stage models indeed have some tricky training treatments, incurring them hard to train. Thus, please fine-tune your model after ensuring the best pre-training performance you can ever get, and then try to fine-tune your model on video datasets by using a small learning rate, fixing the static branch, increasing the fine-tuning epochs, or something else. You can feel free to discuss me with your experience for more inspiration.

I also note that there has a similar issue #13. Today, we retrain our model on the pre-trained weight and get a similar performance reported in our conference paper. Next is the comparison; please check it out.

Original Performance:

• (Dataset:CVC-ClinicDB-612-Test) Sm:0.903;meanEm:0.903;MAE:0.038;maxEm:0.923;maxDice:0.860;maxIoU:0.795;maxSpe:0.992.
• (Dataset:CVC-ClinicDB-612-Valid) Sm:0.923;meanEm:0.944;MAE:0.012;maxEm:0.962;maxDice:0.873;maxIoU:0.800;maxSpe:0.991.
• (Dataset:CVC-ColonDB-300) Sm:0.909;meanEm:0.921;MAE:0.013;maxEm:0.942;maxDice:0.840;maxIoU:0.745;maxSpe:0.996.

New Performance (that we retrained again):

• (Dataset:CVC-ClinicDB-612-Test) Sm:0.891;meanEm:0.876;MAE:0.043;maxEm:0.921;maxDice:0.854;maxIoU:0.785;maxSpe:0.995.
• (Dataset:CVC-ClinicDB-612-Valid) Sm:0.917;meanEm:0.923;MAE:0.013;maxEm:0.952;maxDice:0.861;maxIoU:0.787;maxSpe:0.991.
• (Dataset:CVC-ColonDB-300) Sm:0.897;meanEm:0.892;MAE:0.012;maxEm:0.936;maxDice:0.833;maxIoU:0.734;maxSpe:0.997.

These numerical results show some performance fluctuations due to different random seed or PyTorch versions or something. But I think it is acceptable for us. We don't take further action to improve the performance as we just fine-tune our model with one epoch. Overall It does not show a 'huge' drop after the fine-tuning process.

Thank you all guys to pay attention to our work. Further problems and discussions are welcome here.

ONE MORE THING We also release a large-scale VPS dataset, named SUN-SEG to promote this field. By using this dataset, you can feel free to train your large model without worries about over-fitting problems. You can also discard the two-stage training strategy using our enhanced method, PNS+, which only needs training once. We strongly suggest you use them.

Best,
Ge-Peng.

from pns-net.

Hi, @Moqixis

Sorry for the late reply. Since we got a very, very limited video dataset for this task before submission, we adopted two-stage training to achieve promising performance. But, in general, the two-stage models indeed have some tricky training treatments, incurring them hard to train. Thus, please fine-tune your model after ensuring the best pre-training performance you can ever get, and then try to fine-tune your model on video datasets by using a small learning rate, fixing the static branch, increasing the fine-tuning epochs, or something else. You can feel free to discuss me with your experience for more inspiration.

I also note that there has a similar issue #13. Today, we retrain our model on the pre-trained weight and get a similar performance reported in our conference paper. Next is the comparison; please check it out.

Original Performance:

• (Dataset:CVC-ClinicDB-612-Test) Sm:0.903;meanEm:0.903;MAE:0.038;maxEm:0.923;maxDice:0.860;maxIoU:0.795;maxSpe:0.992.
• (Dataset:CVC-ClinicDB-612-Valid) Sm:0.923;meanEm:0.944;MAE:0.012;maxEm:0.962;maxDice:0.873;maxIoU:0.800;maxSpe:0.991.
• (Dataset:CVC-ColonDB-300) Sm:0.909;meanEm:0.921;MAE:0.013;maxEm:0.942;maxDice:0.840;maxIoU:0.745;maxSpe:0.996.

New Performance (that we retrained again):

• (Dataset:CVC-ClinicDB-612-Test) Sm:0.891;meanEm:0.876;MAE:0.043;maxEm:0.921;maxDice:0.854;maxIoU:0.785;maxSpe:0.995.
• (Dataset:CVC-ClinicDB-612-Valid) Sm:0.917;meanEm:0.923;MAE:0.013;maxEm:0.952;maxDice:0.861;maxIoU:0.787;maxSpe:0.991.
• (Dataset:CVC-ColonDB-300) Sm:0.897;meanEm:0.892;MAE:0.012;maxEm:0.936;maxDice:0.833;maxIoU:0.734;maxSpe:0.997.

These numerical results show some performance fluctuations due to different random seed or PyTorch versions or something. But I think it is acceptable for us. We don't take further action to improve the performance as we just fine-tune our model with one epoch. Overall It does not show a 'huge' drop after the fine-tuning process.

Thank you all guys to pay attention to our work. Further problems and discussions are welcome here.

ONE MORE THING We also release a large-scale VPS dataset, named SUN-SEG to promote this field. By using this dataset, you can feel free to train your large model without worries about over-fitting problems. You can also discard the two-stage training strategy using our enhanced method, PNS+, which only needs training once. We strongly suggest you use them.

Best, Ge-Peng.

谢谢您的回复，我再尝试尝试。

from pns-net.