hwloc-ps core dump on Altix, with CPU set interactions about hwloc HOT 8 CLOSED

Imported from trac issue 45. Created by cs on 2011-06-27T19:13:13, last modified: 2012-12-22T03:26:33

• cs attached hwloc-gather-topology.out.output on 2011-06-27 19:15:12
• cs attached hwloc-gather-topology.out.tar.bz2 on 2011-06-27 19:16:01
• sthibaul attached patch on 2011-06-28 10:17:16

from hwloc.

Trac comment by sthibaul on 2011-06-28 10:14:26:

From the tarball it seems that only PU P#2 is allowed, what processor(s) are in the test cpuset?

Currently, hwloc-ps does not have the --whole-system option, and defaults to not showing the whole system, only what is available. In your case that'd mean not showing the processes which are not within the current cpuset. With a --whole-system option, such processes would be shown. Is that OK in your case?

from hwloc.

Trac comment by sthibaul on 2011-06-28 10:18:05:

Could you check whether the behaviour with the attached patch matches your expectation? (without --whole-system option, adding support for --whole-system is another matter)

from hwloc.

Trac comment by cs on 2011-06-28 14:24:06:

The CPU set information is in the attached hwloc-gather-topology tar file, but here's how
it was created:
{{{
cs@altix-02$ sudo cpuset -c /test
cpu_exclusive
mem_exclusive
cpus 2
mems 1
cpuset created.
}}}

from hwloc.

Trac comment by cs on 2011-06-28 14:45:44:

Replying to [comment:3 sthibaul]:

Could you check whether the behaviour with the attached patch matches your expectation? (without --whole-system option, adding support for --whole-system is another matter)

The patch appears to simply the code to loop:
{{{
(gdb) break 144
Breakpoint 1 at 0x4000000000002bf2: file hwloc-ps.c, line 144.
(gdb) run
Starting program: /usr/local/bin/hwloc-ps

Breakpoint 1, main (argc=1, argv=0x60000fffffffb1a8) at hwloc-ps.c:144
144 if (obj) {
(gdb) next
145 hwloc_obj_type_snprintf(type, sizeof(type), obj, 1);
(gdb)
146 idx = logical ? obj->logical_index : obj->os_index;
(gdb)
147 if (idx == (unsigned) -1)
(gdb)
150 printf("%s%s:%u", first ? "" : " ", type, idx);
(gdb)
151 hwloc_bitmap_andnot(remaining, remaining, obj->cpuset);
(gdb) print type
$1 = "Machine\000\000À\002\000\000\000\000 Ьÿÿÿ\017\000`", '\0' <repeats 24 times>, "(\f\000\000\000\000\000 ¸ï\003\000\000\000\000 "
(gdb) print idx
$2 = 0
(gdb) print remaining
$3 = 0x6000000000009da0
(gdb) print *remaining
$4 = {ulongs_count = 1, ulongs_allocated = 8, ulongs = 0x600000000000b120,
infinite = 0}
(gdb) print obj->cpuset
$5 = 0x60000000000092f0
(gdb) print *obj->cpuset
$6 = {ulongs_count = 1, ulongs_allocated = 8, ulongs = 0x6000000000009310,
infinite = 0}
(gdb) next
152 first = 0;
(gdb)
140 while (!hwloc_bitmap_iszero(remaining)) {
(gdb) print remaining
$7 = 0x6000000000009da0
(gdb) print *remaining
$8 = {ulongs_count = 1, ulongs_allocated = 8, ulongs = 0x600000000000b120,
infinite = 0}
(gdb) next
143 hwloc_obj_t obj = hwloc_get_first_largest_obj_inside_cpuset(topology, remaining);
(gdb) print topology
$9 = 0x6000000000008010
(gdb) print *topology
$10 = {nb_levels = 3, next_group_depth = 0, level_nbobjects = {1, 2, 1,
0 <repeats 125 times>}, levels = {0x60000000000090d0, 0x600000000000ab70,
0x600000000000acc0, 0x0 <repeats 125 times>}, flags = 0, type_depth = {0,
0, 1, -1, -1, -1, 2, -1, -1}, ignored_types = {HWLOC_IGNORE_TYPE_NEVER,
HWLOC_IGNORE_TYPE_NEVER, HWLOC_IGNORE_TYPE_NEVER, HWLOC_IGNORE_TYPE_NEVER,
HWLOC_IGNORE_TYPE_NEVER, HWLOC_IGNORE_TYPE_NEVER, HWLOC_IGNORE_TYPE_NEVER,
HWLOC_IGNORE_TYPE_KEEP_STRUCTURE, HWLOC_IGNORE_TYPE_NEVER},
is_thissystem = 1, is_loaded = 1, pid = 0,
set_thisproc_cpubind = 0x200000000003ca38 <local+7656>,
get_thisproc_cpubind = 0x200000000003c868 <local+7192>,
set_thisthread_cpubind = 0x200000000003c878 <local+7208>,
get_thisthread_cpubind = 0x200000000003c888 <local+7224>,
set_proc_cpubind = 0x200000000003ca28 <local+7640>,
get_proc_cpubind = 0x200000000003c858 <local+7176>, set_thread_cpubind = 0,
get_thread_cpubind = 0,
get_thisproc_last_cpu_location = 0x200000000003c8a8 <local+7256>,
get_thisthread_last_cpu_location = 0x200000000003c8b8 <local+7272>,
get_proc_last_cpu_location = 0x200000000003ca58 <local+7688>,
set_thisproc_membind = 0, get_thisproc_membind = 0,
set_thisthread_membind = 0x200000000003c8e8 <local+7320>,
get_thisthread_membind = 0x200000000003c8f8 <local+7336>,
set_proc_membind = 0, get_proc_membind = 0,
set_area_membind = 0x200000000003c8c8 <local+7288>, get_area_membind = 0,
alloc = 0x200000000003c828 <local+7128>,
alloc_membind = 0x200000000003c8d8 <local+7304>,
free_membind = 0x200000000003c808 <local+7096>, support = {
discovery = 0x6000000000009070, cpubind = 0x6000000000009090,
membind = 0x60000000000090b0}, os_distances = {{nbobjs = 0, indexes = 0x0,
objs = 0x0, distances = 0x0}, {nbobjs = 0, indexes = 0x0, objs = 0x0,
distances = 0x0}, {nbobjs = 2, indexes = 0x6000000000009700,
objs = 0x60000000000096c0, distances = 0x60000000000096e0}, {nbobjs = 0,
indexes = 0x0, objs = 0x0, distances = 0x0}, {nbobjs = 0, indexes = 0x0,
objs = 0x0, distances = 0x0}, {nbobjs = 0, indexes = 0x0, objs = 0x0,
distances = 0x0}, {nbobjs = 0, indexes = 0x0, objs = 0x0,
distances = 0x0}, {nbobjs = 0, indexes = 0x0, objs = 0x0,
distances = 0x0}, {nbobjs = 0, indexes = 0x0, objs = 0x0,
distances = 0x0}}, backend_type = HWLOC_BACKEND_SYSFS, backend_params = {
sysfs = {root_path = 0x0, root_fd = -1}, synthetic = {arity = {0, 0,
4294967295, 0 <repeats 125 times>}, type = {
HWLOC_OBJ_SYSTEM <repeats 128 times>}, id = {0 <repeats 128 times>},
depth = {0 <repeats 128 times>}}}}
(gdb) next

Breakpoint 1, main (argc=1, argv=0x60000fffffffb1a8) at hwloc-ps.c:144
144 if (obj) {
(gdb) print obj
$11 = 0x0
(gdb) next
140 while (!hwloc_bitmap_iszero(remaining)) {
(gdb) print remaining
$12 = 0x6000000000009da0
(gdb) print *remaining
$13 = {ulongs_count = 1, ulongs_allocated = 8, ulongs = 0x600000000000b120,
infinite = 0}
(gdb) next
143 hwloc_obj_t obj = hwloc_get_first_largest_obj_inside_cpuset(topology, remaining);
(gdb) print *topology
$14 = {nb_levels = 3, next_group_depth = 0, level_nbobjects = {1, 2, 1,
0 <repeats 125 times>}, levels = {0x60000000000090d0, 0x600000000000ab70,
0x600000000000acc0, 0x0 <repeats 125 times>}, flags = 0, type_depth = {0,
0, 1, -1, -1, -1, 2, -1, -1}, ignored_types = {HWLOC_IGNORE_TYPE_NEVER,
HWLOC_IGNORE_TYPE_NEVER, HWLOC_IGNORE_TYPE_NEVER, HWLOC_IGNORE_TYPE_NEVER,
HWLOC_IGNORE_TYPE_NEVER, HWLOC_IGNORE_TYPE_NEVER, HWLOC_IGNORE_TYPE_NEVER,
HWLOC_IGNORE_TYPE_KEEP_STRUCTURE, HWLOC_IGNORE_TYPE_NEVER},
is_thissystem = 1, is_loaded = 1, pid = 0,
set_thisproc_cpubind = 0x200000000003ca38 <local+7656>,
get_thisproc_cpubind = 0x200000000003c868 <local+7192>,
set_thisthread_cpubind = 0x200000000003c878 <local+7208>,
get_thisthread_cpubind = 0x200000000003c888 <local+7224>,
set_proc_cpubind = 0x200000000003ca28 <local+7640>,
get_proc_cpubind = 0x200000000003c858 <local+7176>, set_thread_cpubind = 0,
get_thread_cpubind = 0,
get_thisproc_last_cpu_location = 0x200000000003c8a8 <local+7256>,
get_thisthread_last_cpu_location = 0x200000000003c8b8 <local+7272>,
get_proc_last_cpu_location = 0x200000000003ca58 <local+7688>,
set_thisproc_membind = 0, get_thisproc_membind = 0,
set_thisthread_membind = 0x200000000003c8e8 <local+7320>,
get_thisthread_membind = 0x200000000003c8f8 <local+7336>,
set_proc_membind = 0, get_proc_membind = 0,
set_area_membind = 0x200000000003c8c8 <local+7288>, get_area_membind = 0,
alloc = 0x200000000003c828 <local+7128>,
alloc_membind = 0x200000000003c8d8 <local+7304>,
free_membind = 0x200000000003c808 <local+7096>, support = {
discovery = 0x6000000000009070, cpubind = 0x6000000000009090,
membind = 0x60000000000090b0}, os_distances = {{nbobjs = 0, indexes = 0x0,
objs = 0x0, distances = 0x0}, {nbobjs = 0, indexes = 0x0, objs = 0x0,
distances = 0x0}, {nbobjs = 2, indexes = 0x6000000000009700,
objs = 0x60000000000096c0, distances = 0x60000000000096e0}, {nbobjs = 0,
indexes = 0x0, objs = 0x0, distances = 0x0}, {nbobjs = 0, indexes = 0x0,
objs = 0x0, distances = 0x0}, {nbobjs = 0, indexes = 0x0, objs = 0x0,
distances = 0x0}, {nbobjs = 0, indexes = 0x0, objs = 0x0,
distances = 0x0}, {nbobjs = 0, indexes = 0x0, objs = 0x0,
distances = 0x0}, {nbobjs = 0, indexes = 0x0, objs = 0x0,
distances = 0x0}}, backend_type = HWLOC_BACKEND_SYSFS, backend_params = {
sysfs = {root_path = 0x0, root_fd = -1}, synthetic = {arity = {0, 0,
4294967295, 0 <repeats 125 times>}, type = {
HWLOC_OBJ_SYSTEM <repeats 128 times>}, id = {0 <repeats 128 times>},
depth = {0 <repeats 128 times>}}}}
(gdb) print *remaining
$15 = {ulongs_count = 1, ulongs_allocated = 8, ulongs = 0x600000000000b120,
infinite = 0}
(gdb) next

Breakpoint 1, main (argc=1, argv=0x60000fffffffb1a8) at hwloc-ps.c:144
144 if (obj) {
(gdb) print obj
$16 = 0x0
(gdb) next
140 while (!hwloc_bitmap_iszero(remaining)) {
(gdb) next
143 hwloc_obj_t obj = hwloc_get_first_largest_obj_inside_cpuset(topology, remaining);
...
}}}

from hwloc.

Trac comment by bgoglin on 2011-06-29 02:10:44:

The while loop cannot terminate because this process is bound (at least partially) outside of the cpuset. We need a break when obj is NULL, and we need to decide what to display with these processes. Two solutions:

• restrict the process binding to its intersection with the current topology before displaying anything and ignore the process if the intersection is NULL
• when displaying objects (without -c), append a special object string such as "outside" when part of the binding is outside of the current topology (could be the case only when -a is given)

from hwloc.

Trac comment by bgoglin on 2011-07-04 07:58:18:

(In [3537]) Fix crash in hwloc-ps (and improve lstopo --ps) when running inside a Linux cpuset
and showing processes from other cpusets.

Restrict the process binding to the currently available topology.
If people want to see process bindings outside of the current administrative
restriction, they should use --whole-system.

Thanks to "cs" trac user (don't know how to get his real name).

This commit fixes #45

from hwloc.

Trac comment by bgoglin on 2011-07-04 10:08:21:

(In [3540]) Backport trunk commit r3537 into v1.2 branch:

• Fix crash in hwloc-ps (and improve lstopo --ps) when running inside a Linux cpuset
  and showing processes from other cpusets.
  Restrict the process binding to the currently available topology.
  If people want to see process bindings outside of the current administrative
  restriction, they should use --whole-system (available in v1.3+).
  Thanks to Carl Smith for reporting the problem.
  This commit fixes #45.

from hwloc.