mca_pml_ob1_recv_frag_callback_match occasional segfault about ompi HOT 9 CLOSED

You may already know this, but be aware that SchedMD changed the srun cmd line in 23.11 - you might need to make some adjustments.

from ompi.

rebuilt slurm and openmpi to use pmi 4.2.9 and dropped the PMIX_MCA_gds=hash setting . Ran ~3000 hello_world jobs in this environment without seeing any core dumps.

from ompi.

Just to be clear: you originally said you ran 3000 jobs with mpirun using PMIx 5.0.2 and saw no problems. So I'm assuming your last test refers to executing with srun and not mpirun - yes??

I fail to see a connection between PMIx and ob1/recv being caught in a segfault - we don't have anything to do with the MPI message exchange. Likewise, it's hard to see what srun has to do with it, so I have no idea what to suggest. Given everything you have encountered across the two issue reports, I suspect there is something more fundamentally borked in this system.

from ompi.

My apologies - blasted github had me logged into a different account when I wrote the above note. Sigh.

from ompi.

no worries - thanks for taking a look at this for me.

Yep, the new testing using an srun launch with a PMIx 4.2.9 based slurm/openmpi did not see any core dumps in ~3000 runs. I'll stick with this new setup for now since things seem happier.

If you can think of any env variables I can set to provide more debug information, please let me know and I can give them a try and report back what I find.

from ompi.

Gave this some thought - given that things work fine under mpirun but fail under srun, I'm inclined to think there is some problem in the Slurm-PMIx integration when using PMIx 5.x. I know nothing about debugging Slurm, so I would really encourage you to file a ticket with SchedMD. At the very least, they should be made aware of the situation in case others encounter it.

It still feels to me like there is something else in your environment causing the problem (and the PMIx change being just a canary or flat out red herring), but minus more info, I have no idea how to pursue it.

from ompi.

one last note to add here before closing this one out and turning my focus to the Slurm/SchedMD side of the house. Two interesting things:

1. adding strace in front of the hello_world_mpi application buries/hides/avoids the issue
2. removing options using cgroups from the slurm.conf appears buries/hides/avoids the issue

Turns out that I had disabled cgroups in my testing area earlier and forgotten about it. My comments above about PMIx impacting this issue should be ignored. Much more likely the change in my slurm configuration in my test environment that changed the launch behavior.

from ompi.

@bhendersonPlano If this issue is not in OMPI rather SLURM or PMIX can you please file in corresponding community and close here?

from ompi.

I've started a thread on the slurm-users mailing list - hopefully someone will chime in there.

I'll close this one out as it does not appear to be an OpenMPI issue.

from ompi.