GPipe vs mesh? about mesh HOT 3 CLOSED

@matthewygf let's hope that Google releases the image classification sample soon and also any basic documentation on how to use GPipe. Right now it seems to be deeply integrated with sequence models only.

from mesh.

@denfromufa

it is open sourced here:
https://github.com/tensorflow/lingvo/blob/master/lingvo/core/gpipe.py

i might not be correct, but seems to me with Mesh, it is solved by constructing an ILP, i.e. finding an optimal set of layout with a number of constrains.

where gpipe, you defined your op that get separated manually.

from mesh.

From what I understand, GPipe is actually pipelining, that is model sequentialism + data parallelism.
Basically, you place each layer on a GPU sequentially, and one GPU #1 is done with batch #1 you feed batch #2. This is illustrated in the Figure 2.c of their paper.

Mesh, however, is true model parallelism in the sense that you really define a distributed operation say the convolution, and distribute it on different GPUs. So you won't be suffering from the bubble of GPipe, that is, except when communicating, all your GPUs will be in use.

from mesh.