The program hangs when runs into parallel_apply() function in util.py about univl HOT 7 CLOSED

Hi @butterluo, I found nothing wrong with your given messages. As I know, the retrieval eval is indeed slow. You can debug with a small part of the test set (e.g., 50 pairs), or print something in _run_on_single_gpu() here to see whether the program is still running.

from univl.

Hi @butterluo, I found nothing wrong with your given messages. As I know, the retrieval eval is indeed slow. You can debug with a small part of the test set (e.g., 50 pairs), or print something in _run_on_single_gpu() here to see whether the program is still running.

I tried last night, but nothing print out, and the log are still hanging on the last line 'NCCL INFO comm 0x7f7eb8003010 rank 1 nranks 2 cudaDev 1 busId b9000 - Init COMPLETE' for the whole night....

How long running the eval_epoch() function cost in your experience?

from univl.

Different test dataset has different time-cost. On average, less than half an hour.

I have no idea about your problem now. A choice is to comment line 406-441 and only call _run_on_single_gpu() temporally.

I also want to make sure that your modules = nn.parallel.replicate(model, device_ids) has valid device_ids and correspondingly valid GPUs. Thanks.

from univl.

Different test dataset has different time-cost. On average, less than half an hour.

I have no idea about your problem now. A choice is to comment line 406-441 and only call _run_on_single_gpu() temporally.

I also want to make sure that your modules = nn.parallel.replicate(model, device_ids) has valid device_ids and correspondingly valid GPUs. Thanks.

When i was using more than 1 gpu, i set 'export CUDA_VISIBLE_DEVICES=3,4' and the device_ids which is passed into 'nn.parallel.replicate(model, device_ids)' is '[0,1]' . Is there any thing wrong?

But when i ran it with 1 gpu, every thing is ok.

from univl.

I have no idea about this bug now. I tested on P40, P100, and V100, and all of them work well. Can you tell me your GPUs' version and pytorch's version?

from univl.

I have no idea about this bug now. I tested on P40, P100, and V100, and all of them work well. Can you tell me your GPUs' version and pytorch's version?

Python version: 3.7 (64-bit runtime)
Tesla V100-PCIE-32GB
CUDA runtime version: 10.0.130
torch==1.8.1

from univl.

Feel free to reopen if any progress on this issue

from univl.