assertion with syscall tracepoint about ply HOT 9 CLOSED

Thanks for the detailed report!

I made the assumption that the stated field type would correspond to the native C type for the particular architecture. I.e. if a field is int somefield, then somefield is of type int. This appears to be wrong though, as int can apparently have a size of both 4 (e.g. common_pid) and 8 (e.g. dfd) on the same platform. Unexpected! Are you running this on an x86_64 machine?

I will have to look in to how the kernel determines what type to write in the format-file of the tracepoint.

If you want a quickfix, try to disable the entire block of code attached to the /* Find all basic scalars. */-comment in tracepoint_parse_type. This way, ply will always fallback to use types of the stated size.

from ply.

Hi,

Yes, I was running this on x86_64 for evaluating at the moment but planning to use it in aarch64.
Your quickfix works as expected, so thank you for this information.

And I managed to cross compile ply for aarch64 with some hint in wkz#19, but it didn't work like x86 as below.

root:/tmp# ply -d 'tracepoint:syscalls/sys_enter_openat { printf("%v\n", comm); }'
info: creating kallsyms cache
ERR:-22
root:/tmp# ply -d 'tracepoint:syscalls/sys_enter_open { printf("%v\n", comm); }'
error: tracepoint:syscalls/sys_enter_open: Unable to parse tracepoint at /sys/kernel/debug/tracing/events/syscalls/sys_enter_open
ply: active

Then can I assume that your quickfix works only for x86_64?
Or my test environment for aarch64 may not be configured properly for running eBPF program.

Test environment:
kernel version: 4.9.140

root:~# zgrep BPF /proc/config.gz
# CONFIG_CGROUP_BPF is not set
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
# CONFIG_NETFILTER_XT_MATCH_BPF is not set
# CONFIG_NET_CLS_BPF is not set
# CONFIG_NET_ACT_BPF is not set
# CONFIG_BPF_JIT is not set
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
# CONFIG_TEST_BPF is not set

from ply.

No, there shouldn't be anything architecture specific in the workaround. The error indicates that it ply is not able to read the tracepoint's format file.

Have you mounted debugfs and tracefs? (/sys/kernel/debug and /sys/kernel/debug/tracing)

Is your kernel built with tracepoint support?

from ply.

Yes, actually we are using ftrace and LTTng trace often.

It's strange that why there is no sys_enter_open in my target.

root:~# ls /sys/kernel/debug/tracing/events/syscalls/sys_enter_open*
/sys/kernel/debug/tracing/events/syscalls/sys_enter_open_by_handle_at:
enable  filter  format  id  trigger

/sys/kernel/debug/tracing/events/syscalls/sys_enter_openat:
enable  filter  format  id  trigger

But there is sys_enter_openat, but ply return ERR -22(EINVAL??).

root:~# cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_openat/format
name: sys_enter_openat
ID: 453
format:
        field:unsigned short common_type;       offset:0;       size:2; signed:0;
        field:unsigned char common_flags;       offset:2;       size:1; signed:0;
        field:unsigned char common_preempt_count;       offset:3;       size:1; signed:0;
        field:int common_pid;   offset:4;       size:4; signed:1;

        field:int __syscall_nr; offset:8;       size:4; signed:1;
        field:int dfd;  offset:16;      size:8; signed:0;
        field:const char * filename;    offset:24;      size:8; signed:0;
        field:int flags;        offset:32;      size:8; signed:0;
        field:umode_t mode;     offset:40;      size:8; signed:0;

print fmt: "dfd: 0x%08lx, filename: 0x%08lx, flags: 0x%08lx, mode: 0x%08lx", ((unsigned long)(REC->dfd)), ((unsigned long)(REC->filename)), ((unsigned long)(REC->flags)), ((unsigned long)(REC->mode))

root:~# ply -d 'tracepoint:syscalls/sys_enter_openat { printf("%v\n", comm); }'
info: creating kallsyms cache
ERR:-22

But kprobe works as expected.

root:/tmp# ply -d 'kprobe:schedule { @[stack] = count(); }'
info: creating kallsyms cache
ply: active
^Cply: deactivating

@:
{
        schedule
        __mutex_lock_slowpath+248
        mutex_lock+80
        clk_prepare_lock+72
        clk_unprepare+44
        tegra_spi_runtime_suspend+72
        pm_generic_runtime_suspend+60
        __rpm_callback+120
        rpm_callback+56
        rpm_suspend+252
        pm_runtime_work+128
        process_one_work+488
        worker_thread+88
        kthread+236
        ret_from_fork+16
 }: 1
{
        schedule
        wait_for_common+200
        wait_for_completion_timeout+44
        usb_start_wait_urb+212
        usb_control_msg+188
        wq_netinfo+160
        process_one_work+488
        worker_thread+88
        kthread+236
        ret_from_fork+16
 }: 1
...

from ply.

I think most modern libcs always translate open to openat. Older architectures still need a kernel side open for older binaries, but if open was never shipped with a certain arch, you could cut it from the kernel ABI. Maybe that is why it is not there.

Looking closer at your output I see that I was wrong in my previous reply. If it had to do with the format file you should have gotten a "Unable to parse tracepoint at ..." message. So it must be something else.

Unfortunately I think the only way forward from here is to pull out gdb and debug it from there. I can't make any promises on when I will have the time to do that. But I can try to guide you through it if you're feeling up for it.

from ply.

Unfortunately the test target doesn't have binaries with GPL at the moment, so I can't run gdb.
But I expect I can have another aarch64 board for testing in a few months and I'm free to use some GPL binaries with it, then I'll revisit this issue again with that.

from ply.

IANAL, but you're free to run whatever bits you want on your device as long as you're not distributing it to someone else.

Just FYI, you do as you wish of course 😄

Anyways, I'll leave the issue open until then.

from ply.

Right. This HW is for commercial purpose, so doesn't have required debug packages with the original setup.
But one issue is that this is common to my team and other members are using it for different purpose at the same time, so it's little bit tricky to modify something for testing.
I expect I can get a board like Raspberry PI for my own purpose, then I'll try it again. 😃

from ply.

I had the same issue and commenting out the block attached to /* Find all basic scalars. */ fixed the problem.

from ply.