flux-imp: Warning: pam_open_session: Cannot make/remove an entry for the specified session about flux-core HOT 15 CLOSED

Ok, I commented out pam_loginuid.so from the flux PAM stack and then the job seems to run successfully (unless my tests are racing with other changes you're trying?)

from flux-core.

There may be more detail in the system logs. I can take look. What were the nodes assigned to the job?

from flux-core.

I think fluke6 was assigned in both cases. It looks like the prolog and epilog worked (reporting status=0) but the imp+shell did not.

My searches for logs have so far been fruitless...

from flux-core.

It looks like the prolog and epilog worked (reporting status=0) but the imp+shell did not.

I think the IMP is the only thing that is trying to call pam_open_session(3), so that makes sense. I haven't seen this particular error before - I'm wondering if it is some specific module in the stack or something internal to PAM that isn't working when run inside one of the systemd transient units.

from flux-core.

I'm not doing anything right now.

from flux-core.

Also, FYI if we don't use pam_loginuid.so the loginuid appears to be that of the flux user, so we might have to figure this out:

 grondo@fluke108:~$ flux run cat /proc/self/loginuid
505

from flux-core.

Well I think what's happening is the pam_loginuid.so is being run twice: once from systemd.user:

$ cat /etc/pam.d/systemd-user
# [snip irrelevant stuff]
# Used by systemd --user instances.

account  include system-auth

session  required pam_selinux.so close
session  required pam_selinux.so nottys open
session  required pam_loginuid.so
session  include system-auth
session  optional pam_systemd.so

and once from flux:

$ cat /etc/pam.d/flux
auth            required        pam_localuser.so
account         required        pam_unix.so
session         required        pam_limits.so
session         required        pam_loginuid.so

Once set, the loginuid is inherited by child processes and cannot be changed. So on our systems, it's getting set for the systemd user instance running as flux and then the IMP inherits this and can't reset it to the end user and fails.

It seems like what we want is some way for the flux user to skip over the pam_loginuid.so line in the systemd-user pam config.

from flux-core.

Good sleuthing, I was only halfway there and was still trying to figure out what was setting the initial loginuid.

from flux-core.

If we don't use user instances for anything except Flux on our clusters, then we might be able to comment out pam_loginuid.so from that pam stack as a workaround for now.

from flux-core.

Just for completeness, the reason the loginuid can't be changed after it is set is due to the following setting:

# auditctl -s | grep loginuid
loginuid_immutable 1 locked

Changing this would also allow the loginuid to be modified, but this is a security setting so unlikely to be changed.

from flux-core.

Maybe we could use pam_succeed_if to skip pam_loginuid.so for user flux:
(If you haven't tried this already)

# Used by systemd --user instances.

account  include system-auth

session  required pam_selinux.so close
session  required pam_selinux.so nottys open
session  required [default=1 success=ignore] pam_succeed_if.so user=flux
session  required pam_loginuid.so
session  include system-auth
session  optional pam_systemd.so

The flux user systemd instance would have to be restarted

from flux-core.

Hah nice one! I'll try it.

from flux-core.

oops, the config above is wrong, the pam_succeed_if line should be:

session  [default=1 success=ignore] pam_succeed_if.so user=flux

from flux-core.

@grondo verified this works on fluke, at least for avoiding the error that is the subject of this issue. Possibly sys admins will want to take a closer look at it to make sure it is kosher with respect to security requirements.

from flux-core.

I guess we can close this one since sdexec is now working on all of fluke with the proposed PAM change.

from flux-core.