The segmentation fault(core dump) error after hundreds of training iterations about oanet HOT 11 CLOSED

Is there any other information given when it stops? Sometimes it will go wrong in torch.symeig but pytorch will print something.

from oanet.

There is no other information. It seems like a memory error.

from oanet.

It's quite weird. I should never encounter this problem recently.
Do you use the same code and data? What is your pytorch version?

from oanet.

Yes, I used the same code and data and encountered this problem. So I've integrated your dataloader into my training code, but I still encounter the same problem. My pytorch version is 1.4.

from oanet.

One possible reason I can think of is that there are some strange data that cause unexpected errors. To debug this, you can set shuffle=False and see whether the code dumps at the same data.

from oanet.

I have successfully trained one epoch so I don't think it is caused by strange data. I have googled it and it might be because one is working with a lot of data and the RAM is full, or the Python version is improper. Could you please tell me your Python version and the RAM size of your computer?

from oanet.

124G and python 3.5. You can monitor the memory usage to see whether it is because of the memory.

from oanet.

I have monitored the memory usage. Memory leak will occur if I set num_workers > 0. Someone encounters the similar error: pytorch/pytorch#13246 and I need to further debug to figure it out.

from oanet.

I retrained several models yesterday, but no memory leaks were found...

from oanet.

I've retrained you method and successfully reproduced the results reported in your paper. Thanks for your help.

from oanet.

So, did you find the reason for the segmentation fault?

from oanet.