Comments (3)
kwen2501
commented on August 20, 2024
Hi, have you tried a clean build from a clean clone of nccl-tests?
Also, are you building on some specific platforms like ARM? I was wondering if the #pragma weak feature could have been deprecated in some newer compiler toolchains. Ref: https://developer.arm.com/documentation/dui0491/i/Compiler-specific-Features/-pragma-weak-symbol---pragma-weak-symbol1---symbol2
Could you please see if the following change fixes the issue? Thanks!
--- a/src/all_reduce.cu
+++ b/src/all_reduce.cu
@@ -106,9 +106,7 @@ testResult_t AllReduceRunTest(struct threadArgs* args, int root, ncclDataType_t
return testSuccess;
}
-struct testEngine allReduceEngine = {
+struct testEngine ncclTestEngine = {
AllReduceGetBuffSize,
AllReduceRunTest
};
-
-#pragma weak ncclTestEngine=allReduceEngine
from nccl-tests.
AmericanEnglish
commented on August 20, 2024
@kwen2501 This did indeed fix the issue for compiling all_reduce.cu. I am not compiling for arm. I am compiling for an Intel Skylake Xeon Gold and an Intel Ivybridge Xeon. I then run into issues compiling the other cu files due to the same problems. I assume that make this change everywhere will most likely fix the problem.
from nccl-tests.
AmericanEnglish
commented on August 20, 2024
Applying your suggested change too all cu files now allows for everything to compile.
[barajasc@gpunode102 nccl-tests]$ make -j 16
make -C src build
make[1]: Entering directory `/users/barajasc/tests/nccl/nccl-tests/src'
Compiling all_reduce.cu > ../build/all_reduce.o
Compiling common.cu > ../build/common.o
Compiling all_gather.cu > ../build/all_gather.o
Compiling broadcast.cu > ../build/broadcast.o
Compiling reduce_scatter.cu > ../build/reduce_scatter.o
Compiling reduce.cu > ../build/reduce.o
Compiling alltoall.cu > ../build/alltoall.o
Linking ../build/all_reduce.o > ../build/all_reduce_perf
Linking ../build/all_gather.o > ../build/all_gather_perf
Linking ../build/broadcast.o > ../build/broadcast_perf
Linking ../build/reduce_scatter.o > ../build/reduce_scatter_perf
Linking ../build/reduce.o > ../build/reduce_perf
Linking ../build/alltoall.o > ../build/alltoall_perf
make[1]: Leaving directory `/users/barajasc/tests/nccl/nccl-tests/src'
from nccl-tests.
Related Issues (20)
- hypercube out-of-bound errors with single-proc + `gpus-per-thread=4`, not with multi-proc + `gpus-per-thread=1` HOT 1
- nccl-tests result is only a half of ib_write_bw
- NCCL Test Does not work with GID 3 or GID 1, but it works fine for GID 0
- NCCL Test hang when the number of nodes goes beyond 18, and CPU usage is very high HOT 2
- all_reduce_perf between NVLINK connected H100 PCIe GPUs lower than A100 SXM4 GPUs
- undefined reference to ncclRedOpDestroy HOT 2
- busbw exceeds network bandwidth (2 nodes, 16 gpus, 100Gbps intel NIC, no NVSwitch) - what algorithm is used? HOT 5
- How to explain Bus Bandwidth in Allreduce Operation?
- How is the maximum number of bytes for all_reduce operation calculated? HOT 3
- Interaction between NCCL_IB_SL and NCCL_IB_ADAPTIVE_ROUTING
- Multi node test hang phenomenon HOT 2
- Performance lack of NCCL Test HOT 5
- Why getBw don't have access to agg_iters ? HOT 1
- Test NCCL failure common.cu:961 'internal error - please report this issue to the NCCL developers / ' HOT 7
- cputime
- misc/ibvwrap.cc:278 NCCL WARN Call to ibv_reg_mr_iova2 failed with error Cannot allocate memory HOT 2
- make failed, error -- unsupported GNU version! gcc versions later than 11 are not supported!
- Test NCCL failure common.cu:954 'unhandled cuda error HOT 1
- The network bandwidth in the alltoall_perf test failed to meet expectations HOT 3
- Differences problems in performance data of HGX A800 single server N GPUs nccl testing
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nccl-tests.