Performance Differences between Tensorflow and Pytorch about firedup HOT 8 CLOSED

Has there been any followup on this? This is an amazing library....would hate to go back to Tensorflow...

from firedup.

@kashif makes sense :) I do appreciate how easy to follow this set of code is. Translating from TF to PyTorch is a big deal and appreciate so much that you've done it 👍

from firedup.

thanks @karanchahal so can you kindly run with cpu 1 to make sure I am not doing anything wrong with the MPI versions of Adam, and then the other big difference is how the Dense layers are initialized in Tensorflow (is it glorot? dont remember) and how they are initialized in Pytorch... Perhaps also try to run it for a bit more epochs etc.

I should rename this project to something else, so that both tensorflow and pytorch versions can exist together which might help too...

thanks again for your help!

from firedup.

Hey guys, as to weight initializing, OpenAI-Baselines has custom numpy-based orthogonal initializers. I haven't tested them in firedup thoroughly yet but having them seems to help, at least we can leave the different init methods between pytorch and tensorflow.

https://github.com/openai/baselines/blob/master/baselines/a2c/utils.py

from firedup.

Thanks @chutaklee I think pytorch also has orthogonal_ which I could test with. Great work BTW on your fork!

from firedup.

@windweller thanks! I havent had time to try out the orthogonal initialisation... if you want to give it a go that would be great, otherwise I will try in a week or so.

from firedup.

@kashif Ahhh I'm mostly trying to extend the library to do something else...not trying to get to the top of the benchmark. ChutakLee also emailed me explaining why it's difficult to do a direct performance comparison.

A slightly more relevant question -- I notice that both Spinningup and Firedup are not trained on GPU right? I guess it's hard to allocate GPU intelligently with the OpenMPI tool...

from firedup.

@windweller right so the networks here are more on the small size so I thought it would be better to just keep everything simple and not introduce gpu/cpu device and scale the training by using more mpi workers... It would not be too hard to add a device flag and then on the mpi side we copy to cpu and do the broadcast and then copy back or use pytorch's distributed api...

from firedup.