report oom kills about flux-core HOT 5 CLOSED

The OOM killer will terminate tasks with SIGKILL. If a task exited with wait status indicating this signal, that could trigger a check in the memory.events file, or alternately, if the shell is separately watching the memory.events file via inotify, a flag could be set which would trigger an error message that tasks terminated with SIGKILL may have been victims of the oom killer.

from flux-core.

I think the shell (assuming it's not the target of the OOM) can simply do the following

• get the shell's cgroup root from /proc/<pid>/cgroup (it will contain 0::path, where path is relative to /sys/fs/cgroup)
• read the oom_kill count from memory.events
• if nonzero, log something like memory cgroup out of memory: process killed

This could be checked in the shell task.exit callback, although if multiple ranks were killed, I'm not sure if we can reliably conclude that the exiting task was the target of the oom, unless there were some way to retain the task's cgroup memory.events post exit...

from flux-core.

I wonder if it would work to use something like inotify or otherwise to watch the memory.events file and report an OOM kill as it happens?

unless there were some way to retain the task's cgroup memory.events post exit...

Is there a separate memory.events per pid?

from flux-core.

Is there a separate memory.events per pid?

Yes. I was worried about racy access since the child process is immediately reaped by the libev child watcher, but maybe there is a non racy way to consume all the events via inotify or even just by holding it open and reading it before closing it. Good thought! I'll try some experiments.

from flux-core.

Is there a separate memory.events per pid?

Er. I misspoke. The task /proc/<pid>/cgroup points to the same cgroup dir as the shell. (Which makes sense I guess)

from flux-core.