Out-Of-Memory regardless of setting batch_size about transmatcher HOT 10 CLOSED

What is the GPU card you used? I was using V100 with 32GB memory without issues, but I did not check those with lower memory. You can try lower the batch size, --num_trans_layers, --dim_feedforward, and --neck.

from transmatcher.

I used 8 GPUs of V-100 Tesla 16GB but when I set to batch_size of 8, it still has an out-of-memory error.

from transmatcher.

I used only one V100, with a batch size of 64. It seems that you should be able to run it with batch size of 8 in one of your GPU.

from transmatcher.

How many GB memory for your V100? Is it 32 GB? I just have 16GB and it says that 9.0 GB is required.

from transmatcher.

Yes, as I said, 32GB.

from transmatcher.

I see. So, we are unable to run the code with lower memory even multiple GPUs and small batch size. But it is weird that the efficient model can not be trained with batch size of 1. Could you show me where to update the code to make it run with lower memory? Thanks

from transmatcher.

I'm not sure. I still feel strange that you cannot run the code with batch size of 8 in one V100 (16GB), since I'm able to run with batch size of 64 in one V100 (32GB). Did you try one GPU?

For further suggestions, please see the below.

What is the GPU card you used? I was using V100 with 32GB memory without issues, but I did not check those with lower memory. You can try lower the batch size, --num_trans_layers, --dim_feedforward, and --neck.

from transmatcher.

I have encountered this issue and it seems that you need to change the test_batch smaller. Because the default size of the test batch is 256.

from transmatcher.

Thanks. Yes, could be this reason. I thought @haithanhp encountered this during training.

from transmatcher.

Thanks for your help. I used the server with larger GPU memory. So, the issue is no longer there.

from transmatcher.