How to get performance gain using multigpu training about convnet HOT 3 OPEN

• You cannot train same layer on two CPU or cannot gain speed with two different adjacent layers on two separate GPU. The only way to gain speed on two or more GPU if e.g you NN architecture is split e.g. like this architecture: http://benanne.github.io/images/imagenet.png where you can take advantage from two GPU cards. There are many more situations where you can train sort of parallel architectures on multiple GPU e.g train two separate input layers that at some point will join each other.

from convnet.

Thank you, I will try to split the model.

from convnet.

I tested the model splitting with imagenet data on two K40 GPU boards. The 20-minibatch (each minibatch has 128 images of size 224x224) training time is 32.2 seconds for the AlexNet [Alex Krizhevsky NIPS 2012] using single GPU, while the 20-minibatch training time is 21.1 seconds if I split the model into two GPU memories following the model structure of AlexNet. I do get improvement in efficiency. But if I use the same structure with same batch size in training, Caffe can achieve around 19 seconds per 20 minibatches even on single GPU. Anyone else tested it? Tell me if my testing results are wrong.

from convnet.