joker221b Goto Github PK

all-reducible-signsgd

Distributed Data Parallel training is a well known solution to the problem of large scale Machine Learning training. A significant portion of training time is spent in communication between machines for the purpose of gradient synchronization. Current research hints at various ways in which we can optimize communication between machines - Gradient compression , topK, reduced precision, ternGrad to name a few. In this paper, we propose an All-Reducible implementation for signSGD gradient synchronization and show that it achieves 2-4x speed up when compared to the State of the Art 1-bit Adam implementation.

gnn_compression