Train models on multiple GPUs about deepcell-tf HOT 2 CLOSED

While the code changes for horovod are pretty simple, it involves running our docker container with the mpi process. It is my understanding that this will work well for .py scripts, but it does not seem to be useful with jupyter notebooks (the notebook server is duplicated on each GPU). There seems to be jupyter support, but it may not be worth the extra effort if we can just get multi_gpu_model working.

This thread uses a wrapper class that exposes the original model's save/load functions. Currently, this solution works on the sample training with minor edits; using ModelCheckpoint(..., save_only_weights=True,...)

from deepcell-tf.

Fixed by #101

from deepcell-tf.