Available runtime processors is not really number of available processors about scala-native HOT 8 CLOSED

I think JVM respect tasksets:

$ cat test.sc
println(Runtime.getRuntime().availableProcessors())

$ scala-cli run test.sc
Compiling project (Scala 3.4.1, JVM (21))
Compiled project (Scala 3.4.1, JVM (21))
12

$ taskset -c 0-0 scala-cli run test.sc
1

$ taskset -c 0-0 scala-cli run --native test.sc
** compiler output snipped **
12

from scala-native.

@amsen20 That particular method is suppose to indicate the number of cores or hyperthreads and should match what the JVM reports. OMP looks like you need to set the number of cores OMP will use. Glancing at the docs (not an OpenMP user) shows that you can also set an env var to do what you are doing with taskset above. There could be a problem if our method doesn't match what the JVM report. I guess that would be a good job for scala-cli. Thanks for bringing this to our attention.

Ref: https://curc.readthedocs.io/en/latest/programming/OpenMP-C.html

from scala-native.

For reference this is how nproc (the GNU coreutils command to get the number of available processors) implement this query with linux/bsd APIs.

Note that if you look at num_processors (the main function) at the end of the file, nproc also follows OpenMP environment variables, but we don't have to (and the JVM doesn't either).

from scala-native.

aI took some time looking at this Issue. The "taskset" JVM/SN comparison helped a lot.

• I can not read the nproc source because that is almost certainly GPL licensed.

• Just to make sure I am on the correct page, by "availableProcessors" we are talking about
  "number of logical processes (threads)" not "number of hardware cores". The Java Runtime.getavaibleProcessors()
  description is particularly silent there.

It appears to me that this Issue divides into three parts:

1. Windows -- I have no expertise there, so any PR would need to use the current code until someone
   with windows expertise & testing system could fix it.

2. BSD systems -- These are the easier path to code. The core/logical_processor distinction is keenest here.
   The time consuming part here is figuring out the "compile only if used" condition compilation of the
   .c file. I think the pieces are in place to do that, just not fixed in my brain or fingers yet.

   • I can test macOS easily. In a pinch I can test FreeBSD.

   • I have no access to NetBSD. I hate leaving untested
     code active, so I propose to fail the compilation on NetBSD. It ought to work and be easy for a NetBSD
     champion to remove the guard

3. Linux -- The simple case where number of configured (logical) CPUs is less than 1025 should be
   comparatively simple. The difficulty comes from the <= 1024 restriction. Conditional compilation only on
   use is a, I believe surmountable, thorn in the path here.

   • Getting the number of hardware cores would be more difficult: reading & parsing files, which might not exist, depending
     on how the was configured.

   • I propose to fallback to current code if the sched_getaffinity() code returns -1 & errno EINVAL (invalid argument)
     This restriction would lead to a more timely PR.

   • amsen20: would this be useful to you until a PR handling > 1024 logical processors could be created & tested?

4. Does anybody have access to a recent Linux system with (much) greater than 1024 logical processors?
   I can write code intended to handle such but I have learned through private & public shame & hard experience
   not to release untested code.

   I have not checked, but I think coming up with such a large number of processors might not be feasable
   on Amazon EC cloud.

Thunderstorms & power failures coming, gotta go.

from scala-native.

Unless someone has evidence to the contrary, I think OpenMP environment should be handled
by the application, not javalib. Consensus?

from scala-native.

In my use case I am not even near using 1024 threads, so resolving it for <=1024 hyper threads until a PR comes would be great for me.

from scala-native.

amsen20,

Does PR #3969 give you what you want/need? It is Linux only, with nProcessors <= 1024.

Please advise. I can make additional changes if this baselevel does not solve this Issue (for Linux).

Thank you, once again, for finding & reporting this defect.

Lee

from scala-native.

Yeah, it works for my use case well.
Thanks.

from scala-native.