error in training about open_clip HOT 4 CLOSED

Hi @nikky4D, thanks for opening up this issue. Is this the full stack trace? I haven't encountered anything like this before, so I'm wondering whether this could potentially be a mismatch in python / pytorch / cuda versions. Do your versions match those in environment.yml?

Also, could you provide more details on what hardware you are using?

from open_clip.

For the stack trace: That is all I was given. I was running 10 epochs so this is at the final epoch.
For python/pytorch/cuda: I'm using python 3.9.10, with pytorch 1.10.2, and cuda 11.3.
For environment.yml: The versions don't match with those in the environment.yml as I'm using a windows machine and I couldn't install many of the packages in environment.yml in my conda environment.

from open_clip.

I see, that could be the issue here. Our codebase was only tested with the versions in environment.yml and on unix machines

from open_clip.

You may be correct. I'm using 4 workers, for batchsize of 32, on a 2080TI. And everything runs well until the last epoch where the after calculating the evaluation metrics, I get that error. It appears to only be affecting the saving of the checkpoint of the epoch directly precedeing the error as I don't seem able to load that checkpoint. I can load other checkpoints.

Anyway, I'll close this until I find a solution that would work.

from open_clip.