amdresearch / hpcfund Goto Github PK

View Code? Open in Web Editor NEW

10.0 5.0 4.0 120 KB

AMD HPC Research Fund Cloud

Home Page: https://amdresearch.github.io/hpcfund/

amdgpu hpc research

hpcfund's People

Contributors

Stargazers

Watchers

Forkers

fengjixuchui zstreet87

hpcfund's Issues

launching HIP compiler

I am trying to run 4 openmm simulations simultaneously in parallel on the same node, one at each GPU -since mi1004x has 4 and mi1008x has 8- using this loop
for i in seq 0 3
do
mkdir $i
cd $i
python3 $execname $i &> slurmout.run.$SLURM_JOB_ID &
cd -
done
where $i runs over GPU index and $execname is the python file.
The issue is getting this error when I make the loop runs till 3 or more
"openmm.OpenMMException: Error launching HIP compiler: 32512"

The job works well when the max number in the loop is 2.
Also this happens whatever the partition I am using (devel and mi1004x and mi1008x ) either in interactive job or using bash file.

Installing CP2K using Spack: Failure

Hi Tom,
I have been looking for install CP2K using the spack framework:
# Example: For building CP2K 7.1 with ELPA, AOCC and AOCL
$ spack install [email protected]+elpa %gcc ^amdfftw ^amdscalapack ^amdblis ^amdlibflame ^elpa+openmp ^openmpi fabrics=auto
as described in https://www.amd.com/en/developer/zen-software-studio/applications/spack/hpc-applications-cp2k.html

However, it has been unsuccessful. There are a lot of errors including

==> Installing dbcsr-2.6.0-7rd7h7jk4pelsdksdnvl3rqj6fw4wcta [56/60]
==> No binary for dbcsr-2.6.0-7rd7h7jk4pelsdksdnvl3rqj6fw4wcta found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/c6/c67b02ff9abc7c1f529af446a9f01f3ef9e5b0574f220259128da8d5ca7e9dc6.tar.gz
==> No patches needed for dbcsr
==> dbcsr: Executing phase: 'cmake'
==> Error: ProcessError: Command exited with status 1:
    '/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/cmake-3.27.9-ynrsqwcupq2ikk6csf4ng2titzr4y7bm/bin/cmake' '-G' 'Ninja' '-DCMAKE_INSTALL_PREFIX:STRING=/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/dbcsr-2.6.0-7rd7h7jk4pelsdksdnvl3rqj6fw4wcta' '-DCMAKE_BUILD_TYPE:STRING=Release' '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF' '-DCMAKE_POLICY_DEFAULT_CMP0090:STRING=NEW' '-DCMAKE_FIND_USE_PACKAGE_REGISTRY:BOOL=OFF' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON' '-DCMAKE_INSTALL_RPATH:STRING=/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/dbcsr-2.6.0-7rd7h7jk4pelsdksdnvl3rqj6fw4wcta/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/dbcsr-2.6.0-7rd7h7jk4pelsdksdnvl3rqj6fw4wcta/lib64;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdblis-4.1-hs3ndn6uhnrdxoxj3whtiv4tnz3si5yf/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/gcc-runtime-12.2.0-ceuxgepwdijlfdvrvpng2hm3m5dvkl77/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdlibflame-4.1-qdyxlhwb7z4uc4ztnvjlkxcdrplxw453/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libxsmm-1.17-vapy2yelns7wg46wh2edwrmqaolwlslv/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/hwloc-2.9.1-hlmqbvatvllspftaoj7raciiwbutwww6/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libpciaccess-0.17-4xzi2qwgsqunkhno7dnezrp5trvc2dey/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libxml2-2.10.3-reszpc7xtpw32yxf5ekgg3n2kmcl2wqi/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libiconv-1.17-spt7q67r2gmfjyyugip7x3ikvxcivpga/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/xz-5.4.6-yty3u44c53cyymr3pd7qhpy4u46eqbhf/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/zlib-ng-2.1.5-3gdh45yzusvjqhpowvu5wo2ssql6fv2w/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/ncurses-6.4-ajopbihnxse2p5ta2k3kvhkzwot6t3ch/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libevent-2.1.12-yh6zdkoiwcjmqevw2slmsf3fgq5nrhss/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/numactl-2.0.14-2i24pnwedxml6beiwgclddrn5m5q3yrv/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/pmix-5.0.1-nrlwrv6olzxacpsndherq6iwyzuxbf64/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openssl-3.2.1-jo5rzom5ha4cy24slpxwyfb56g7tq65t/lib64' '-DCMAKE_PREFIX_PATH:STRING=/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/py-fypp-3.1-mrkqaj2yqjeaet26v6m3iqnx7umjfttc;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/pmix-5.0.1-nrlwrv6olzxacpsndherq6iwyzuxbf64;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/numactl-2.0.14-2i24pnwedxml6beiwgclddrn5m5q3yrv;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libevent-2.1.12-yh6zdkoiwcjmqevw2slmsf3fgq5nrhss;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/hwloc-2.9.1-hlmqbvatvllspftaoj7raciiwbutwww6;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libpciaccess-0.17-4xzi2qwgsqunkhno7dnezrp5trvc2dey;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/ninja-1.11.1-v46lrig2qrqdxqoofgxjmpadqaneahvj;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libxsmm-1.17-vapy2yelns7wg46wh2edwrmqaolwlslv;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdlibflame-4.1-qdyxlhwb7z4uc4ztnvjlkxcdrplxw453;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/cmake-3.27.9-ynrsqwcupq2ikk6csf4ng2titzr4y7bm;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdblis-4.1-hs3ndn6uhnrdxoxj3whtiv4tnz3si5yf;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openssl-3.2.1-jo5rzom5ha4cy24slpxwyfb56g7tq65t;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libxml2-2.10.3-reszpc7xtpw32yxf5ekgg3n2kmcl2wqi;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/zlib-ng-2.1.5-3gdh45yzusvjqhpowvu5wo2ssql6fv2w;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/xz-5.4.6-yty3u44c53cyymr3pd7qhpy4u46eqbhf;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/ncurses-6.4-ajopbihnxse2p5ta2k3kvhkzwot6t3ch;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/pkgconf-1.9.5-7mn7yjokyok3sfsi3264nmwiykwhqrv7;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libiconv-1.17-spt7q67r2gmfjyyugip7x3ikvxcivpga;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/gcc-runtime-12.2.0-ceuxgepwdijlfdvrvpng2hm3m5dvkl77' '-DUSE_SMM=libxsmm' '-DUSE_MPI:BOOL=ON' '-DUSE_OPENMP:BOOL=ON' '-DWITH_C_API:BOOL=ON' '-DBLAS_FOUND=true' '-DBLAS_LIBRARIES=/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdblis-4.1-hs3ndn6uhnrdxoxj3whtiv4tnz3si5yf/lib/libblis.so' '-DLAPACK_FOUND=true' '-DLAPACK_LIBRARIES=/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdlibflame-4.1-qdyxlhwb7z4uc4ztnvjlkxcdrplxw453/lib/libflame.so' '-DBUILD_SHARED_LIBS:BOOL=ON' '-DWITH_EXAMPLES:BOOL=OFF' '/tmp/jzapatar/spack-stage/spack-stage-dbcsr-2.6.0-7rd7h7jk4pelsdksdnvl3rqj6fw4wcta/spack-src'

1 error found in build log:
     37    -- Found BLAS: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdblis-4.1-hs3ndn6uhnrdxoxj3whtiv4tnz3si5yf/lib/libblis.so
     38    -- A library with LAPACK API found.
     39    -- Found Python: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/python-3.11.7-ylyj7xfqigor5esuzrejby2i6z6ygbhs/bin/python3.11 (found version "3.11.7") found components: Interpreter
     40    -- Could NOT find MPI_C (missing: MPI_C_WORKS)
     41    -- Could NOT find MPI_CXX (missing: MPI_CXX_WORKS)
     42    -- Could NOT find MPI_Fortran (missing: MPI_Fortran_WORKS)
  >> 43    CMake Error at /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/cmake-3.27.9-ynrsqwcupq2ikk6csf4ng2titzr4y7bm/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
     44      Could NOT find MPI (missing: MPI_C_FOUND MPI_CXX_FOUND MPI_Fortran_FOUND C
     45      CXX Fortran)
     46    Call Stack (most recent call first):
     47      /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/cmake-3.27.9-ynrsqwcupq2ikk6csf4ng2titzr4y7bm/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
     48      /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/cmake-3.27.9-ynrsqwcupq2ikk6csf4ng2titzr4y7bm/share/cmake-3.27/Modules/FindMPI.cmake:1837 (find_package_handle_standard_args)
     49      CMakeLists.txt:188 (find_package)

See build log for details:
  /tmp/jzapatar/spack-stage/spack-stage-dbcsr-2.6.0-7rd7h7jk4pelsdksdnvl3rqj6fw4wcta/spack-build-out.txt

==> Warning: Skipping build of cp2k-2024.1-mglxwu5yt7pybwrwal5b63oll6t4vszo since dbcsr-2.6.0-7rd7h7jk4pelsdksdnvl3rqj6fw4wcta failed
==> Installing amdfftw-4.1-a6m65e7wv2vh2csinhcm7znkjbtafqts [57/60]
==> No binary for amdfftw-4.1-a6m65e7wv2vh2csinhcm7znkjbtafqts found: installing from source
==> Using cached archive: /work1/zapata/jzapatar/software/spack/var/spack/cache/_source-cache/archive/f1/f1cfecfcc0729f96a5bd61c6b26f3fa43bb0662d3fff370d4f73490c60cf4e59.tar.gz
==> Ran patch() for amdfftw
==> amdfftw: Executing phase: 'autoreconf'
==> amdfftw: Executing phase: 'configure'
==> Error: ProcessError: Command exited with status 1:
    './../configure' '--prefix=/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdfftw-4.1-a6m65e7wv2vh2csinhcm7znkjbtafqts' '--enable-amd-opt' '--enable-dynamic-dispatcher' '--enable-mpi' '--enable-amd-mpifft' '--enable-shared' '--enable-openmp' '--disable-threads' '--disable-amd-fast-planner' '--disable-amd-mpi-vader-limit' '--disable-static' 'AMD_ARCH=znver3' '--enable-sse2' '--enable-avx' '--enable-avx2' '--disable-avx512'

1 error found in build log:
     87    checking how to run the C preprocessor... /work1/zapata/jzapatar/software/spack/lib/spack/env/gcc/gcc -E
     88    checking for ocamlbuild... no
     89    checking for mpicc... /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/bin/mpicc
     90    checking for MPI_Init... no
     91    checking for MPI_Init in -lmpi... no
     92    checking for MPI_Init in -lmpich... no
  >> 93    configure: error: could not find mpi library for --enable-mpi

See build log for details:
  /tmp/jzapatar/spack-stage/spack-stage-amdfftw-4.1-a6m65e7wv2vh2csinhcm7znkjbtafqts/spack-build-out.txt

==> Installing amdscalapack-4.1-jtzxfedvdp3z4ov63gfrljb3yn46btqw [58/60]
==> No binary for amdscalapack-4.1-jtzxfedvdp3z4ov63gfrljb3yn46btqw found: installing from source
==> Using cached archive: /work1/zapata/jzapatar/software/spack/var/spack/cache/_source-cache/archive/b2/b2e51c3604e5869d1faaef2e52c92071fcb3de1345aebb2ea172206622067ad9.tar.gz
==> No patches needed for amdscalapack
==> amdscalapack: Executing phase: 'cmake'
==> Error: ProcessError: Command exited with status 1:
    '/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/cmake-3.27.9-ynrsqwcupq2ikk6csf4ng2titzr4y7bm/bin/cmake' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:STRING=/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdscalapack-4.1-jtzxfedvdp3z4ov63gfrljb3yn46btqw' '-DCMAKE_BUILD_TYPE:STRING=Release' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF' '-DCMAKE_POLICY_DEFAULT_CMP0090:STRING=NEW' '-DCMAKE_FIND_USE_PACKAGE_REGISTRY:BOOL=OFF' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON' '-DCMAKE_INSTALL_RPATH:STRING=/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdscalapack-4.1-jtzxfedvdp3z4ov63gfrljb3yn46btqw/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdscalapack-4.1-jtzxfedvdp3z4ov63gfrljb3yn46btqw/lib64;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdblis-4.1-hs3ndn6uhnrdxoxj3whtiv4tnz3si5yf/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/gcc-runtime-12.2.0-ceuxgepwdijlfdvrvpng2hm3m5dvkl77/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdlibflame-4.1-qdyxlhwb7z4uc4ztnvjlkxcdrplxw453/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/hwloc-2.9.1-hlmqbvatvllspftaoj7raciiwbutwww6/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libpciaccess-0.17-4xzi2qwgsqunkhno7dnezrp5trvc2dey/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libxml2-2.10.3-reszpc7xtpw32yxf5ekgg3n2kmcl2wqi/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libiconv-1.17-spt7q67r2gmfjyyugip7x3ikvxcivpga/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/xz-5.4.6-yty3u44c53cyymr3pd7qhpy4u46eqbhf/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/zlib-ng-2.1.5-3gdh45yzusvjqhpowvu5wo2ssql6fv2w/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/ncurses-6.4-ajopbihnxse2p5ta2k3kvhkzwot6t3ch/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libevent-2.1.12-yh6zdkoiwcjmqevw2slmsf3fgq5nrhss/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/numactl-2.0.14-2i24pnwedxml6beiwgclddrn5m5q3yrv/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/pmix-5.0.1-nrlwrv6olzxacpsndherq6iwyzuxbf64/lib;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openssl-3.2.1-jo5rzom5ha4cy24slpxwyfb56g7tq65t/lib64' '-DCMAKE_PREFIX_PATH:STRING=/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/pmix-5.0.1-nrlwrv6olzxacpsndherq6iwyzuxbf64;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/numactl-2.0.14-2i24pnwedxml6beiwgclddrn5m5q3yrv;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libevent-2.1.12-yh6zdkoiwcjmqevw2slmsf3fgq5nrhss;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/hwloc-2.9.1-hlmqbvatvllspftaoj7raciiwbutwww6;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libpciaccess-0.17-4xzi2qwgsqunkhno7dnezrp5trvc2dey;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdlibflame-4.1-qdyxlhwb7z4uc4ztnvjlkxcdrplxw453;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/cmake-3.27.9-ynrsqwcupq2ikk6csf4ng2titzr4y7bm;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdblis-4.1-hs3ndn6uhnrdxoxj3whtiv4tnz3si5yf;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openssl-3.2.1-jo5rzom5ha4cy24slpxwyfb56g7tq65t;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libxml2-2.10.3-reszpc7xtpw32yxf5ekgg3n2kmcl2wqi;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/zlib-ng-2.1.5-3gdh45yzusvjqhpowvu5wo2ssql6fv2w;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/xz-5.4.6-yty3u44c53cyymr3pd7qhpy4u46eqbhf;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/ncurses-6.4-ajopbihnxse2p5ta2k3kvhkzwot6t3ch;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/libiconv-1.17-spt7q67r2gmfjyyugip7x3ikvxcivpga;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/gmake-4.4.1-nhnzvk525rck4ckiik5bqyiccm7btra4;/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/gcc-runtime-12.2.0-ceuxgepwdijlfdvrvpng2hm3m5dvkl77' '-DBUILD_SHARED_LIBS:BOOL=ON' '-DBUILD_STATIC_LIBS:BOOL=OFF' '-DLAPACK_FOUND=true' '-DLAPACK_INCLUDE_DIRS=/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdlibflame-4.1-qdyxlhwb7z4uc4ztnvjlkxcdrplxw453/include' '-DLAPACK_LIBRARIES=/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdlibflame-4.1-qdyxlhwb7z4uc4ztnvjlkxcdrplxw453/lib/libflame.so' '-DBLAS_LIBRARIES=/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdblis-4.1-hs3ndn6uhnrdxoxj3whtiv4tnz3si5yf/lib/libblis.so' '-DCMAKE_C_FLAGS:STRING=' '-DCMAKE_Fortran_FLAGS=-fallow-argument-mismatch' '-DENABLE_ILP64:BOOL=OFF' '-DUSE_F2C:BOOL=OFF' '-DLAPACK_LIBRARIES=/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/amdlibflame-4.1-qdyxlhwb7z4uc4ztnvjlkxcdrplxw453/lib/libflame.so' '-DLAPACK_FOUND=true' '-DCMAKE_C_COMPILER=/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/bin/mpicc' '-DCMAKE_Fortran_COMPILER=/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/bin/mpif90' '/tmp/jzapatar/spack-stage/spack-stage-amdscalapack-4.1-jtzxfedvdp3z4ov63gfrljb3yn46btqw/spack-src'

84 errors found in build log:
     17     -- The C compiler identification is GNU 12.2.0
     18     -- The Fortran compiler identification is GNU 12.2.0
     19     -- Detecting C compiler ABI info
     20     -- Detecting C compiler ABI info - failed
     21     -- Check for working C compiler: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/bin/mpicc
     22     -- Check for working C compiler: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/bin/mpicc - broken
  >> 23     CMake Error at /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/cmake-3.27.9-ynrsqwcupq2ikk6csf4ng2titzr4y7bm/share/cmake-3.27/Modules/CMakeTestCCompiler.cmake:67 (message):
     24       The C compiler
     25     
     26         "/work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/bin/mpicc"
     27     
     28       is not able to compile a simple test program.
     29     

     ...

     40         /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/cmake-3.27.9-ynrsqwcupq2ikk6csf4ng2titzr4y7bm/bin/cmake -E cmake_link_script CMakeFiles/cmTC_4302f.dir/link.txt --verbose=1
     41         /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/bin/mpicc -rdynamic CMakeFiles/cmTC_4302f.dir/testCCompiler.c.o -o cmTC_4302f
     42         /usr/bin/ld: warning: libucp.so.0, needed by /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so, not found (try using -rpath or -rpath-link)
     43         /usr/bin/ld: warning: libucs.so.0, needed by /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so, not found (try using -rpath or -rpath-link)
     44         /usr/bin/ld: warning: libucm.so.0, needed by /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so, not found (try using -rpath or -rpath-link)
     45         /usr/bin/ld: warning: libuct.so.0, needed by /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so, not found (try using -rpath or -rpath-link)
  >> 46     /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucm_test_external_events'
  >> 47         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_rkey_release'
  >> 48         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_atomic_fetch_nb'
  >> 49         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucp_worker_flush_nb'
  >> 50         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_ep_get_address'
  >> 51         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_worker_destroy'
  >> 52         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_worker_release_address'
  >> 53         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_query_components'
  >> 54         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_md_mkey_pack'
  >> 55         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_rkey_destroy'
  >> 56         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_release_component_list'
  >> 57         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucm_set_external_event'
  >> 58         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_get_version'
  >> 59         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucp_worker_get_address'
  >> 60         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_tag_recv_nbx'
  >> 61         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_config_release'
  >> 62         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_tag_probe_nb'
  >> 63         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_atomic_post'
  >> 64         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucs_async_context_create'
  >> 65         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucp_disconnect_nb'
  >> 66         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_cleanup'
  >> 67         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_tag_recv_nb'
  >> 68         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_worker_create'
  >> 69         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_iface_set_am_handler'
  >> 70         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_iface_open'
  >> 71         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_ep_create'
  >> 72         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucm_test_events'
  >> 73         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_md_config_read'
  >> 74         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_md_close'
  >> 75         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_component_query'
  >> 76         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_rkey_unpack'
  >> 77         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_dt_destroy'
  >> 78         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucp_context_print_info'
  >> 79         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucp_rkey_pack'
  >> 80         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_iface_get_device_address'
  >> 81         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_worker_destroy'
  >> 82         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_iface_get_address'
  >> 83         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_config_read'
  >> 84         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_tag_send_sync_nb'
  >> 85         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucp_request_release'
  >> 86         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_tag_send_nb'
  >> 87         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucm_vm_munmap'
  >> 88         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_md_open'
  >> 89         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_tag_send_nbx'
  >> 90         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_request_free'
  >> 91         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_request_cancel'
  >> 92         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucs_status_string'
  >> 93         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucp_ep_flush_nb'
  >> 94         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_ep_connect_to_ep'
  >> 95         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_iface_query'
  >> 96         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_rkey_buffer_release'
  >> 97         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_tag_msg_recv_nb'
  >> 98         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_iface_close'
  >> 99         /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_ep_destroy'
  >> 100        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_worker_create'
  >> 101        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_worker_query'
  >> 102        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_release_tl_resource_list'
  >> 103        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_md_iface_config_read'
  >> 104        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_get_nbi'
  >> 105        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_config_release'
  >> 106        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucp_mem_map'
  >> 107        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_ep_destroy'
  >> 108        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_request_test'
  >> 109        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_ep_query'
  >> 110        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucs_async_context_destroy'
  >> 111        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_md_mem_dereg'
  >> 112        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_request_check_status'
  >> 113        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_context_query'
  >> 114        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucp_mem_query'
  >> 115        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucp_worker_fence'
  >> 116        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_worker_progress'
  >> 117        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_dt_create_generic'
  >> 118        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_init_version'
  >> 119        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `ucp_mem_unmap'
  >> 120        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_put_nbi'
  >> 121        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libmpi.so: undefined reference to `ucp_ep_rkey_unpack'
  >> 122        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_md_query_tl_resources'
  >> 123        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_md_mem_reg'
  >> 124        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_ep_create'
  >> 125        /usr/bin/ld: /work1/zapata/jzapatar/software/spack/opt/spack/linux-rocky9-zen3/gcc-12.2.0/openmpi-5.0.2-yuagm6al2hwcptzzvfplnk6l56wbk4y4/lib/libopen-pal.so.80: undefined reference to `uct_md_query'
  >> 126        collect2: error: ld returned 1 exit status
  >> 127        gmake[1]: *** [CMakeFiles/cmTC_4302f.dir/build.make:99: cmTC_4302f] Error 1
     128        gmake[1]: Leaving directory '/tmp/jzapatar/spack-stage/spack-stage-amdscalapack-4.1-jtzxfedvdp3z4ov63gfrljb3yn46btqw/spack-build-jtzxfed/CMakeFiles/CMakeScratch/TryCompile-5xZ5Ro'
  >> 129        gmake: *** [Makefile:127: cmTC_4302f/fast] Error 2
     130    
     131    
     132    
     133    
     134    
     135      CMake will not be able to correctly generate this project.

See build log for details:
  /tmp/jzapatar/spack-stage/spack-stage-amdscalapack-4.1-jtzxfedvdp3z4ov63gfrljb3yn46btqw/spack-build-out.txt

==> Warning: Skipping build of elpa-2023.11.001-patched-wefperxof6qb3sbdy2nhooe7u7enmhgq since amdscalapack-4.1-jtzxfedvdp3z4ov63gfrljb3yn46btqw failed
==> Error: cp2k-2024.1-mglxwu5yt7pybwrwal5b63oll6t4vszo: Package was not installed
==> Error: Installation request failed.  Refer to reported errors for failing package(s).

A similar pattern applies to the Quantum Espresso package:
# Example for building Quantum-Espresso with AOCC and AOCL.
$ spack install quantum-espresso+elpa+openmp+scalapack+mpi hdf5=parallel %aocc ^amdfftw ^amdscalapack ^amdblis threads=openmp ^amdlibflame ^[email protected] ^openmpi fabrics=auto
https://www.amd.com/en/developer/zen-software-studio/applications/spack/quantum-espresso.html

Because we are no experienced on that kind of installation, please:
Can you check whatever we are doing wrong?
Can this type of installing takes advantage of GPU resources in the hpcfund architecture?

cannot submit jobs to mi1008x

Hi,

The jobs I submitted to mix1008x got into queue with the reason "resource". However, i checked that there are only 4 mi1008x nodes being used. Do you know what might be the issue?

Singularity module

Is it possible to install singularity as a module on the cluster? It would help run containers for our MD program, OpenMM.

some nodes in mi1008x are down

Hi, some nodes in mi1008x are down this morning. I think only 10 can be used right now.

2815415 Segmentation fault

I got this error
/path/execnew.sh: line 3: 2815415 Segmentation fault (core dumped) python3 /path/run.py $1
When I use sbatch to run a job where I use 4 GPUs for my job.
#!/bin/bash -l
#SBATCH --job-name=example
#SBATCH --nodes=1
#SBATCH --export=ALL
#SBATCH --time=24:00:00
#SBATCH -p mi1004x

I think there is nothing wrong with my script because this error happens only with one node of mi1004x which is t006-002.
Also I get the same error with some of mi1008x and sometimes another error with only the last GPU in the mi1008x that I can't reproduce now. I should mention that the same script was working well till a month ago and it works well with some nodes, this is why i think it is not a script problem.

Inquiry of increasing the job time limit

Hi! Thanks for providing the HPC resources!

I would like to know if you can increase the job time limit from 24 to 96 hours because when I pre-train a model, it often takes more than one day to finish the whole experiment. Right now, I need to resume my previous training occasionally, which is really inconvenient. I would really appreciate it if you could increase the slurm job time limit. Thank you!

Running multiple scripts on a node

I am attempting to run multiple copies of a PyTorch python script in parallel e.g. one copy for each of the 4 gpus on a mi1004x node. I am currently attempting to do this by:

opening an interactive shell on devel / mi1004x with salloc
using a helper script which uses subprocess.Popen() to run an non-blocking instance of the training script
within the training script, I set os.envrion['CUDA_VISIBLE_DEVICES'] to the GPU id I want to use.

This seems to work for running up to 3 jobs in parallel, but when trying to run a 4th job (which should be possible as there is a 4th GPU free), I encounter the following error: Check failed: ret == 0 (11 vs. 0)Thread tf_Compute creation via pthread_create() failed. Could it be that I need to add a certain flag(s) during step 1. so that each traning script has adequate CPU resources?

Tensorflow/Keras

It would be very useful for us if you may provide Tensorflow/Keras wth GPU support

Experiments on some nodes are 10x slower than other

Hi,

I just launched a 4-node experiment on mi1008x (t006-[009-010],t007-[009-010]) and found that my experiment ran significantly slower (more than 10 times) than before. Then I ran the exact same experiment on another 4 node (t004-007,t006-007,t008-[007,009]) and the speed is the same as before. I haven't experience this issue before. I'm wondering if there's something wrong with the nodes in (t006-[009-010],t007-[009-010]). Thanks!

Internet down at HPC?

It seems that there are some internet issues in HPC. I cannot load the model from huggingface today. Got the error "OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like facebook/esm2_t33_650M_UR50D is not the path to a directory containing a file named config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode.". The code worked fine yesterday.

Profiling the node(s)

Hi,

I need to profile the microarchitecture for some HPC applications. I aim to profile microarchitectural events such as cache hit/miss rate. Based on my understanding, I should use the AMD uProf profiler. Would you please let me know if we have access to this profiler in the HPC cloud or not, and if yes, how I can access it?

Thank you in advance

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.