Cray MPI launches as singletons on frontier about flux-core HOT 10 CLOSED

I think we used to always insert the path to Flux's libpmi first in LD_LIBRARY_PATH, but I can't remember why we stopped.

It cause problems on El Cap where Cray's libpmi2.so needs to be found before ours, or else 😿 See #5041

from flux-core.

I didn't think of that!

So maybe prepend ours in the simple pmi plugin and un-prepend in the cray-pals plugin.

I think that makes sense.

from flux-core.

plugins are called in the order they were loaded, so builtin plugins always come first currently.

from flux-core.

Makes sense to me! I think we used to always insert the path to Flux's libpmi first in LD_LIBRARY_PATH, but I can't remember why we stopped.

Another alternative would be a shell option to specifically do this, e.g. -o force-pmi-path or something. That would be easy to implement in a Lua plugin, with the drawback that we're currently not able to check fo invalid or not consumed shell options, so a typo would go undetected.

from flux-core.

Well we could also prepend to LD_LIBRARY_PATH by default again, and add a plugin to flux-coral2 or on the Cray machines that removes it. (e.g. implement the workaround on the one machine that's broken, not the other way around).

I'm just throwing ideas out there though.

from flux-core.

I do remember getting yelled at in mattermost for the audacity of modifying LD_LIBRARY_PATH. However, I also think there was extreme irony in that situation. 😃

from flux-core.

Hmm if we add to LD_LIBRARY_PATH in the pmi shell plugin and subtract in the flux-coral2 shell plguin, will the ordering work out? Which comes first, built-ins or add-ons?

from flux-core.

On second thought, if we only set LD_LIBRARY_PATH when the non-default -o pmi=pmi2 option is specified, then we probably don't need to do anything in cray-pals. All subtracting from LD_LIBRARY_PATH in cray-pals would do is change who wins when both plugins are loaded. Does that even matter?

from flux-core.

It guess it would be more convenient for users if we prepended to LD_LIBRARY_PATH by default so MPI works in the common case without the user needing to know if they need to specify -o pmi=pmi2 or why.

I guess this is equivalent to how Slurm works, only they achieve it by installing libpmi into the standard path.

Just a thought.

from flux-core.

Well when you put it that way it does make sense.

from flux-core.