Some confusion about Tensorflow Distributed Training（The amount of synchronized data is much larger than the theoretical value） about tensorflow HOT 4 OPEN

@certainly-cyber,
tf.contrib.distribute.CollectiveAllReduceStrategy is not available and it was the deprecated API. The CollectiveAllReduceStrategy is exported as MultiWorkerMirroredStrategy.

https://www.tensorflow.org/api_docs/python/tf/distribute/MultiWorkerMirroredStrategy

Also I suspect you are using the code which was related tensorflow 1.x version which is not actively supported now. Kindly convert the code for the latest version and use the tensorflow v2.15 or v2.16. Thank you!

from tensorflow.

I understand what you mean, but this API used to be available...right？
Due to some reasons, I may not be able to migrate and use the relevant version of TF2. In fact, the above code can run successfully and obtain good training results using TF1.14 and CollectiveAllReduceStrategy API. The only point I am confused about is why there is a much larger communication volume between workers than the theoretical value.
Looking forward to your reply, and have a nice day~

from tensorflow.

@certainly-cyber,
The API tf.contrib.distribute.CollectiveAllReduceStrategy is not available and as it is part of the deprecated tf.contrib
CollectiveAllReduceStrategy is MultiWorkerMirrorStrategy. CollectiveAllReduceStrategy is a name we used in the implementation. Please refer to documentations of MultiWorkerMirrorStrategy.

Also CollectiveAllReduceStrategy reduces computation overhead by distributing it across workers, there's additional overhead associated with transferring data between workers. This overhead can include serialization, deserialization, and network latency, all of which can contribute to a larger communication volume than the theoretical minimum. Thank you!

from tensorflow.

Okay, I got it. Thank you again for your answer~
By the way, if I have adopted MultiWorkerMirroredStrategy, can we accurately calculate (or roughly estimate) the amount of data that needs to be synchronized? I have already know how to use model parameters for estimation, but I think it's not accurate enough:) Such like what the format of the synchronized message will be like, or what is the proportion of overhead information? Thank you!

from tensorflow.