Does uncertainty help? about ua-mt HOT 6 CLOSED

Hi Jun,

Have you tried other seeds or do not set the random seed? Since there exists performance perturbation during different training, it is meaningless to compare one pair of experiment results. You can run the experiments many times with/without uncertainty scheme and compare the average performance.

from ua-mt.

Hi @yulequan ，

Thanks for your quick reply.
Not yet, I will try another 2 seeds and compare the average performance.

BTW, have you done the uncertainty scheme ablation study? If yes, how much performance gain?

from ua-mt.

I forget the exact numbers. In our experiments, the uncertainty scheme improves about 0.5% average dice on unlabeled data.

from ua-mt.

Generally, there will be no statistically significant difference (p<0.05) with less then 1% improvements (e.g., in MICCAI KiTS challenge).

In other words, compared to (bayesian) Vanilla V-Net, the key to the performance improvements is the unlabelled data and Mean Teacher Framework, right?

from ua-mt.

According to the concept of p-value, you can think so. In my experiment, I mainly focus on average dice and have not calculated p-value.

from ua-mt.

Got it. Thank you very much.

In addition, I learn a lot from your code which is well-written. I really appreciate that you make the great work publicly available.

Kindest regards,
Jun

from ua-mt.