Example notebook doesn't work and has poor results when fixed about torchxrayvision HOT 5 CLOSED

Hey, sorry for the delay, I will have a chance to look into this soon!

from torchxrayvision.

Thanks! Interested to know what you find. Curious if one of the issues is a calibration one.

from torchxrayvision.

Hey I had a chance to look at this. I updated the notebook so it runs and computes more metrics. One issue there was that it was applying the sigmoid after the model was already doing it as you said. I'm not sure why that was there but it wouldn't have impacted the AUC anyway and the AUC values were similar. The samples were randomly selected before so it is not possible to process the same samples again. I set the random seed this time.

The accuracy and F1 scores seem bad. Setting a threshold at 0.5 appears not optimal for this data. The "all" model was calibrated using the test splits over all the datasets so it is likely not perfect for these samples.

from torchxrayvision.

Hi thanks for looking into this and fixing the double sigmoid bug!

I agree that the threshold of 0.5 is not optimal. In your code there are some custom thresholds set, and we tried using those, but it didn't seem to make a huge difference. However it could have been from the double sigmoid pushing the numbers up.

from torchxrayvision.

Let me know if this issue is not resolved. I'll close it for now.

from torchxrayvision.