Multi-GPU about vit-pytorch HOT 6 CLOSED

How to do multi-GPU training?
Now only one GPU get utilised

How did you solve this issue?

from vit-pytorch.

@kamalkraj @chanhoeom
Hello, I meet the same issue, have you solved it?

from vit-pytorch.

@TitaniumOne Checkout PyTorch distributed training
Pytorch multiprocess launcher script, torch.distributed.launch

from vit-pytorch.

Same problem even though using torch.distributed.launch, codes are shown below

CUDA_VISIBLE_DEVICES=0,1,2,3 python3 -m torch.distributed.launch --nproc_per_node=4 train.py xxx

It is weird that all the trainning are running on the first GPU. Then if the batch size is increased, OOM error is reported.

I was running the program in a docker container.

Any one knows what's wrong?

from vit-pytorch.

same error here

from vit-pytorch.

this might help:

#58

from vit-pytorch.