SendRecv Time about nccl-tests HOT 2 OPEN

The reported time is the time of the NCCL group call, i.e.

ncclGroupStart();
ncclRecv(...); // from prev rank
ncclSend(...); // to next rank
ncclGroupEnd();

It is not a ping-pong test, it's more like a single ring connecting previous rank and next rank.

from nccl-tests.

@sjeaugey Thank you, assuming the alpha-beta cost model, would you agree that the following accurately describes the total time per rank $t_r$ and ideal reported time $t_R$? $$t_r = \alpha_{r-1, r} + n \cdot\beta_{r-1, r} = \frac{RTT}{2}$$ $$t_R = \max_{r \in W} t_r$$ where $\alpha_{ij}$ and $\beta_{ij}$ denote latency and bandwidth of sending from $i$ to $j$ and $n$ is data size and $W$ is process world.

That is, for GPU 0, $t_0 = \alpha_{1, 0} + n \cdot\beta_{1, 0}$ since both GPU 0 and 1 send simultaneously through the ring, and GPU 0 will have to wait until it receives from GPU 1, and vice versa.

from nccl-tests.