Comments (6)
AddyLaddy
commented on August 20, 2024
In the Nvidia provided containers the MPI=1 built tests are usually called:
/opt/nccl-tests/build/all_reduce_perf_mpi
Did you build /opt/nccl-tests/build/all_reduce_perf yourself using MPI=1 ?
from nccl-tests.
FrankLeeeee
commented on August 20, 2024
I encountered the same problem on H100 as well. I built nccl-tests with OpenMPI and did not use Docker.
from nccl-tests.
jianh619
commented on August 20, 2024
In the Nvidia provided containers the MPI=1 built tests are usually called:
/opt/nccl-tests/build/all_reduce_perf_mpiDid you build
/opt/nccl-tests/build/all_reduce_perfyourself using MPI=1 ?
Yes , I build the image myself , compiling nccl test with MPI=1
BTW , is there official container provided by Nvidia ? Coudl you let me know where I can get the download link?
from nccl-tests.
FrankLeeeee
commented on August 20, 2024
I solved this issue by using OpenMPI 4.1 instead. I originally built nccl-tests with openmpi 5.0 but it runs separately on each node. After switching to OpenMPI 4.1 and rebuilding it, it works as expected now.
from nccl-tests.
kiskra-nvidia
commented on August 20, 2024
Yes , I build the image myself , compiling nccl test with MPI=1
It sure looks like, for whatever reason, either your MPI compilation or your MPI installation does not work as expected.
Does a simple MPI "hello world" type program work correctly (you know, one that would report the rank and size of MPI_COMM_WORLD from each launched process)?
Can you verify if your all_reduce_perf actually uses MPI? Say, check with ldd if it links with the MPI library:
ldd all_reduce_perf | grep mpi
Or check with nm if it has any MPI symbols:
nm all_reduce_perf | grep MPI
BTW , is there official container provided by Nvidia ? Coudl you let me know where I can get the download link?
Docker container nvidia/cuda:12.2.2-devel-ubuntu22.04 contains NCCL 2.19.3. A number of containers in Nvidia's NGC catalog (https://catalog.ngc.nvidia.com/) contain NCCL as well. I believe TensorRT does (https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorrt), and PyTorch (https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch). Also https://catalog.ngc.nvidia.com/orgs/nvidia/containers/hpc-benchmarks, https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nvhpc...
from nccl-tests.
jianh619
commented on August 20, 2024
Thanks guys , it should be some reason for compilation , rebuild the image , works now .
from nccl-tests.
Related Issues (20)
- Rank Assignment Issue under four containers on two different servers. HOT 8
- Test NCCL failure common.cu:959 'internal error - please report this issue to the NCCL developers / ' HOT 9
- 1 GiB headroom might be too small
- how to support One Device per Process? HOT 4
- NCCL WARN Cannot use cuda/gdr transports as part of specified UCX_TLS HOT 5
- mpirun all_reduce_perf hang with multi-device test
- alltoall_perf: each rank is only sending to half of the other ranks HOT 14
- stepbytes (increment size) argument does not support 1M notation HOT 1
- Test fail caused by ibvwrap.c:160 NCCL WARN Call to ibv_modify_qp failed with error Connection timed out. HOT 2
- has nvswitch, but uses 0 nvls channels HOT 3
- 2 Nodes nccl-test with mpi hangs HOT 1
- what is cu:990 error? how to solve this problem? HOT 5
- Test NCCL failure common.cu:997 'internal error HOT 9
- NCCL Tree allreduce test cannot reach the theoretical bus bandwidth on 2 nodes with 4 nics
- all_reduce_perf core dumped on 4 L20 HOT 21
- What's multi-allreduce ? HOT 1
- How do we comprehend the factor between algBw and busBw? HOT 5
- 2 Node Nccl Test don’t work HOT 7
- how to calculate the tree based allreduce ib bw?
- all_reduce algo factor for NVLink SHARP In network reductions
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nccl-tests.