Plenty of time was wasted when calculating the propagators about quda HOT 15 CLOSED

Not sure whether this is a QUDA issue then. @bjoo thoughts?

from quda.

It seems more likely to be an issue from Chroma instead of QUDA. Low invertQuda / (initQuda - endQuda) means that Chroma takes too much time to run.
Maybe you could check if you have set --enable-openmp while configuring QDPXX.

from quda.

Thanks for the comments. I did set -DQDP_USE_OPENMP=ON when CMaking QDPXX.

from quda.

The CMakeLists.txt in qdpxx/devel branch has something wrong. QDP_USE_OMP_THREADS macro is not actually set. Using autoconf to do the configuration should be fine.
If you want CMake to work, you could add set(QDP_USE_OMP_THREADS ${QDP_USE_OPENMP}) just under this line.

from quda.

from quda.

Thank @SaltyChiang and @bjoo for the valuable advice. I have taken some time and tried the change. Adding the QDP_USE_OMP_THREADS setting in qdpxx/devel CMakeLists.txt solve the problem. I can see QDP use OpenMP threading and improvement of efficiency. Although the default -DCHROMA_ENABLE_OPENMP is OFF, OMP still works. I think I should mention this issue in qdpxx project later.

from quda.

Closing this here as not a QUDA issue. @wittscien if you create a follow-up issue for QDPXX or Chroma feel free to link it here.

from quda.

from quda.

I see, thanks @bjoo.

from quda.

@SaltyChiang A follow-up -- while I find one of the machines works fine (great improvement in efficiency) after adding QDP_USE_OMP_THREADS setting in qdpxx/devel CMakeLists.txt, there is no improvement in efficiency on another machine (The log now prints QDP use OpenMP threading though). I installed them in the same way and used the same sbatch script. Do you know any possible issues that cause the problem?

from quda.

@wittscien could the issue be related to binding? Are the threads being allocated to their own cores? We have instructions for creating job binding scripts here in case this helps. (Look at how we use the CPU_REORDER variable to set numactl.)

from quda.

@maddyscientist Looks like setting numactl does not improve the efficiency. And OMP_NUM_THREADS has not a bit of help. I use only one V100 card. Seems very weird.

from quda.

@wittscien No idea now. Maybe you should write a simple program to check if the machine could handle OpenMP normally.

from quda.

from quda.

Thank you for the comments. Yes, I have exported OMP_NUM_THREADS, and the output of a Chroma program prints on the first line QDP use OpenMP threading. We have x threads as expected @bjoo. But it just does not do it honestly -- if I use top to show the processes, I see only one thread with 100% CPU usage. I wrote a simple code and it does allocate the specified threads @maddyscientist.

from quda.