Running NCCL test on multiple nodes about nccl-tests HOT 6 CLOSED

You would probably need to clean the previous build first via make clean before building again with MPI.

from nccl-tests.

Did you try compiling them with MPI=1 ?

make MPI=1 MPI_HOME=/path/to/mpi CUDA_HOME=/path/to/cuda NCCL_HOME=/path/to/nccl
mpirun -H host1,host2 -np 2 ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 4

from nccl-tests.

Did you try compiling them with MPI=1 ?

make MPI=1 MPI_HOME=/path/to/mpi CUDA_HOME=/path/to/cuda NCCL_HOME=/path/to/nccl
mpirun -H host1,host2 -np 2 ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 4

Got it, thanks~

from nccl-tests.

Did you try compiling them with MPI=1 ?

make MPI=1 MPI_HOME=/path/to/mpi CUDA_HOME=/path/to/cuda NCCL_HOME=/path/to/nccl
mpirun -H host1,host2 -np 2 ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 4

I tried this but it seems that this command just ran all_reduce_perf on two host seperatly and there wasn't communication between nodes?

Here's my command:
mpirun -H 10.251.154.148,10.251.154.143 -np 2 ./all_reduce_perf -b 8 -e 128M -f 128 -g 2
(They are two nodes and each of them has 2 GPUs)
and part of the output:
Out of bounds values : 0 OK
Avg bus bandwidth : 3.94145
Out of bounds values : 0 OK
Avg bus bandwidth : 3.94156

And if I run it on single node, the average bus bandwidth is the same.
command:
mpirun -H 10.251.154.148 ./all_reduce_perf -b 8 -e 128M -f 128 -g 2
result:
Out of bounds values : 0 OK
Avg bus bandwidth : 3.95192

from nccl-tests.

You would probably need to clean the previous build first via make clean before building again with MPI.

Thanks this solved my problem.

from nccl-tests.

You would probably need to clean the previous build first via make clean before building again with MPI.

it helps me a lot. thank you

from nccl-tests.