kl loss is nan about probabilistic-unet-pytorch HOT 5 CLOSED

Hi! I have implemented the architecture as in the original paper (link). In all experiments I ran, I never had this issue so I am not sure what is causing it.

from probabilistic-unet-pytorch.

Hi! I have implemented the architecture as in the original paper (link). In all experiments I ran, I never had this issue so I am not sure what is causing it.

Thank you for your reply, I ran it directly with your code, except for the data, everything else is the same, maybe I should try using LIDC data

from probabilistic-unet-pytorch.

That would be great, if it happens there then something odd is happening and I can look into it. Good luck!

from probabilistic-unet-pytorch.

That would be great, if it happens there then something odd is happening and I can look into it. Good luck!

That would be great, if it happens there then something odd is happening and I can look into it. Good luck!

Hi, i'm back. Thank you for your reply. I have used LIDC data you processed to train，It was normal when it started training, and then it was abnormal：

`in elbo: reconstruction_loss is 2312.861083984375, kl is 17.464019775390625

in elbo: reconstruction_loss is 2914.47802734375, kl is 31.355321884155273

in elbo: reconstruction_loss is 1369.12255859375, kl is 37.53509521484375

in elbo: reconstruction_loss is 1478.605712890625, kl is 55.792762756347656

in elbo: reconstruction_loss is nan, kl is nan

in elbo: reconstruction_loss is nan, kl is nan

in elbo: reconstruction_loss is nan, kl is nan`

This happened in the epoch 1 and step 404.But sometimes it’s completely normal during training.I don’t know why sometimes it’s normal and sometimes it’s abnormal

from probabilistic-unet-pytorch.

Hmm, I do remember that in very rare occassions I also had these issues. But I couldn't figure out what was causing it to be honest.

from probabilistic-unet-pytorch.