iovisor / ply Goto Github PK
View Code? Open in Web Editor NEWThis project forked from wkz/ply
Dynamic Tracing in Linux
License: GNU General Public License v2.0
This project forked from wkz/ply
Dynamic Tracing in Linux
License: GNU General Public License v2.0
Hi,
This should maybe be two issues but I'm posting them as one for now because maybe the root cause is the same i.e. user error on my part.
I've added ply to buildroot and have basic scripts like the snippet below working on x86_64. I'm using kernel 5.4. buildroot is building that kernel alongside the toolchain so I don't think this is a problem with the kernel headers and toolchain being out of sync.
kprobe:vfs_read
{
@[caller] = count();
}
I'm having problems getting wildcards and quantize() working.
if I change vfs_read
to vfs_*
in the above snippet it looks like attaching the probes fails:
openat(AT_FDCWD, "/sys/kernel/debug/tracing/kprobe_events", O_RDWR|O_CREAT|O_APPEND, 0666) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
write(7, "p:ply195/p610b50_vfs_fadvise vfs"..., 3119) = -1 EINVAL (Invalid argument)
close(7) = 0
close(6) = 0
close(5) = 0
munmap(0x7fef69ff4000, 3425280) = 0
close(4) = 0
fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x5, 0x1), ...}) = 0
ioctl(1, TCGETS, {B9600 opost isig icanon echo ...}) = 0
write(1, "ERR:-22\n", 8ERR:-22
) = 8
exit_group(1) = ?
+++ exited with 1 +++
If I change the above snippet to this:
#!/usr/bin/env ply
kretprobe:vfs_read
{
@["rdsz"] = quantize(retval);
}
I get a huge wall of output like this:
from 72 to 75: R0=inv28 R1=inv1 R2=inv0 R6=ctx(id=0,off=0,imm=0) R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mmmmmmmm fp-144=mmmmmmmm fp-152=mmmmmmmm fp-160=mmmmmmmm fp-168=mmmmmmmm fp-176=mmmmmmmm fp-184=mmmmmmmm fp-192=mmmmmmmm fp-200=mmmmmmmm fp-208=mmmmmmmm fp-216=mmmmmmmm fp-224=mmmmmmmm fp-232=mmmmmmmm fp-240=mmmmmmmm fp-248=mmmmmmmm fp-256=mmmmmmmm fp-264=mmmmmmmm fp-272=mmmmmmmm
75: (67) r0 <<= 2
76: (bf) r1 = r10
77: (07) r1 += -264
78: (0f) r1 += r0
last_idx 78 first_idx 69
regs=1 stack=0 before 77: (07) r1 += -264
regs=1 stack=0 before 76: (bf) r1 = r10
regs=1 stack=0 before 75: (67) r0 <<= 2
regs=1 stack=0 before 72: (b5) if r1 <= 0x1 goto pc+2
regs=1 stack=0 before 69: (b5) if r1 <= 0x3 goto pc+2
R0_rw=invP28 R1_rw=inv(id=0,umin_value=1,umax_value=68719476735,var_off=(0x0; 0xfffffffff)) R2=inv0 R6=ctx(id=0,off=0,imm=0) R10=fp0 fp-8_r=mmmmmmmm fp-16_r=mmmmmmmm fp-24_r=mmmmmmmm fp-32_r=mmmmmmmm fp-40_r=mmmmmmmm fp-48_r=mmmmmmmm fp-56_r=mmmmmmmm fp-64_r=mmmmmmmm fp-72_r=mmmmmmmm fp-80_r=mmmmmmmm fp-88_r=mmmmmmmm fp-96_r=mmmmmmmm fp-104_r=mmmmmmmm fp-112_r=mmmmmmmm fp-120_r=mmmmmmmm fp-128_r=mmmmmmmm fp-136_r=mmmmmmmm fp-144_r=mmmmmmmm fp-152_r=mmmmmmmm fp-160_r=mmmmmmmm fp-168_r=mmmmmmmm fp-176_r=mmmmmmmm fp-184_r=mmmmmmmm fp-192_r=mmmmmmmm fp-200_r=mmmmmmmm fp-208_r=mmmmmmmm fp-216_r=mmmmmmmm fp-224_r=mmmmmmmm fp-232_r=mmmmmmmm fp-240_r=mmmmmmmm fp
ERR:-28
This seems to be output from the bpf verification in the kernel? I'm pretty new to bpf so I don't think I'll have much hope of debugging that. If posting the complete output/the code ply generates would be helpful I can do that.
Thanks,
Daniel
I've been trying to get the trace
provider working with a predicate, but running the example script gives me this error:
error(4): syntax error, unexpected '{'
Here's the script:
#!/usr/bin/env ply
trace:raw_syscalls/sys_exit / ret() < 0 /
{
@[comm()].count()
}
I am working on a 32 bit powerpc (ppc440) and all the examples I run give EINVAL (-22).
As far as I can tell the kernel header version that I am using is what is running. The --with-kerneldir does not seem to have any effect.
I am using the most recent code.
Thanks,
Neil
# cat /proc/version
Linux version 5.10.2-debug (oe-user@oe-host) (arm-resy-linux-gnueabi-gcc (GCC) 10.2.0, GNU ld (GNU Binutils) 2.35.0.20200730) #1 SMP Mon Dec 21 12:30:08 UTC 2020
# cat /proc/cpuinfo
processor : 0
model name : ARMv7 Processor rev 10 (v7l)
BogoMIPS : 6.00
Features : half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x2
CPU part : 0xc09
CPU revision : 10
processor : 1
model name : ARMv7 Processor rev 10 (v7l)
BogoMIPS : 6.00
Features : half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x2
CPU part : 0xc09
CPU revision : 10
processor : 2
model name : ARMv7 Processor rev 10 (v7l)
BogoMIPS : 6.00
Features : half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x2
CPU part : 0xc09
CPU revision : 10
processor : 3
model name : ARMv7 Processor rev 10 (v7l)
BogoMIPS : 6.00
Features : half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x2
CPU part : 0xc09
CPU revision : 10
Hardware : Freescale i.MX6 Quad/DualLite (Device Tree)
Revision : 0000
Serial : 0000000000000000
# ply -T
Verifying kernel config (/proc/config.gz)... OK
Ensuring that debugfs is mounted... OK
Verifying kprobe... OK
Verifying tracepoint... OK
# ply -v
ply 2.1.1-14-ge25c913 (linux-version:330240~5.10.0)
# ply 'kretprobe:vfs_read { @["size"] = quantize(retval); }'
ply: active
^Cply: deactivating
@:
{ size }:
< 0 1 ┤▏ │
[ 0, 1] 111 ┤████████████████▎ │
[ 2, 3] 19 ┤██▊ │
...
[ 16, 31] 2 ┤▎ │
[ 32, 63] 66 ┤█████████▊ │
...
[ 512, 1k) 5 ┤▊ │
...
[ 4k, 8k) 14 ┤██ │
root@multi-v7-ml:~/projects/ply/scripts#
# ply 'kprobe:do_sys_open { printf("%v(%v): %s\n", comm, uid, str(arg1)); }'
ply: active
executed find and cat
^Cply: deactivating
# ./opensnoop.ply
ply: active
cd /
find | grep linux
^Cply: deactivating
path:
# ./execsnoop.ply
ERR:-2
# ./net-rx.ply
ERR:-2
Can you please advise on the test cases?
In case you want me to run more tests, or different things please let me know.
# ply -c 'kprobe:do_sys_open { i = 42; printf("%d\n", i); }'
1 probe active
42
42
42
42
42
42
42
42
42
42
42
42
42
42
42
^Cde-activating probes
# ply -c 'kprobe:do_sys_open { i = -42; printf("%d\n", i); }'
error(1): syntax error, unexpected '-'
ply: provider/tracepoint.c:198: tracepoint_parse: Assertion `offs == type_offsetof(t, t->sou.fields[n - 1].name)' failed.
I've tried this workaround, but it didn't work:
/* Find all basic scalars. */
/* if (!strcmp(str, "char")) */
/* return explicit_sign ? (sign ? &t_schar : &t_uchar) : &t_char; */
/* else if (!strcmp(str, "short")) */
/* return explicit_sign ? (sign ? &t_sshort : &t_ushort) : &t_short; */
/* else if (!strcmp(str, "int")) */
/* return explicit_sign ? (sign ? &t_sint : &t_uint) : &t_int; */
/* else if (!strcmp(str, "long")) */
/* return explicit_sign ? (sign ? &t_slong : &t_ulong) : &t_long; */
/* else if (!strcmp(str, "long long")) */
/* return explicit_sign ? (sign ? &t_sllong : &t_ullong) : &t_llong; */
Format:
~# cat /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/format
name: sys_enter
ID: 19
format:
field:unsigned short common_type; offset:0; size:2; signed:0;
field:unsigned char common_flags; offset:2; size:1; signed:0;
field:unsigned char common_preempt_count; offset:3; size:1; signed:0;
field:int common_pid; offset:4; size:4; signed:1;
field:unsigned short common_migrate_disable; offset:8; size:2; signed:0;
field:unsigned short common_padding; offset:10; size:2; signed:0;
field:long id; offset:16; size:4; signed:1;
field:unsigned long args[6]; offset:20; size:24; signed:0;
print fmt: "NR %ld (%lx, %lx, %lx, %lx, %lx, %lx)", REC->id, REC->args[0], REC->args[1], REC->args[2], REC->args[3], REC->args[4], REC->args[5]
I've added some logs:
~# ply 'tracepoint:raw_syscalls/sys_enter { @[pid, comm] = count(); }'
name: common_type, offs: 0, type_offsetof: 0
name: common_flags, offs: 2, type_offsetof: 2
name: common_preempt_count, offs: 3, type_offsetof: 3
name: common_pid, offs: 4, type_offsetof: 4
name: common_migrate_disable, offs: 8, type_offsetof: 8
name: common_padding, offs: 10, type_offsetof: 10
name: id, offs: 16, type_offsetof: 12
ply: provider/tracepoint.c:198: tracepoint_parse: Assertion `offs == type_offsetof(t, t->sou.fields[n - 1].name)' failed.
Aborted (core dumped)
# ply -c 'kprobe:vfs_read { @start[pid()] = nsecs(); } kretprobe:vfs_read / @start[pid()] / { @us.quantize((nsecs() - @start[pid()]) / 1000); @start[pid()] = 0; }'
2 probes active
^Cde-activating probes
[...]
@us:
0 103 ┤█▏ │
1 748 ┤████████ │
[ 2, 3] 811 ┤████████▋ │
[ 4, 7] 727 ┤███████▊ │
[ 8, 15] 540 ┤█████▊ │
[ 16, 31] 30 ┤▍ │
[ 32, 63] 6 ┤▏ │
[ 64, 127] 5 ┤ │
[ 128, 255] 2 ┤ │
[ 256, 511] 0 ┤ │
[ 512, 1k) 6 ┤▏ │
[ 1k, 2k) 2 ┤ │
[ 2k, 4k) 2 ┤ │
[ 4k, 8k) 6 ┤▏ │
[ 8k, 16k) 2 ┤ │
[ 16k, 32k) 8 ┤▏ │
[ 32k, 64k) 2 ┤ │
[ 64k, 128k) 0 ┤ │
[128k, 256k) 2 ┤ │
[256k, 512k) 0 ┤ │
[512k, 1M) 1 ┤ │
ASCII graph is nice (and I like how -A is available), but could it use the full width?
My current understanding is that ply doesn't support uprobes yet. Is that planned, or is anyone working on it?
Looking at the providers, the support to add say profiling provide in kprobes was wonderfully concise. Might uprobes be similar, or is it a quite different undertaking? Apologies in advance for my lack of understanding there.
root@vm_compiler-s8:~/ply# ply -dS 'k:schedule { exit(0); }'
-- globals
int stdbuf{u32}
k:schedule
{}void
bwritevoid
ctxvoid __bpf *
stdbufint{u32}
:structstruct :anon_0x24058e0
0u64
:structstruct :anon_0x2405880
0int
-- locals
void __bpf *ctx
-- ir
0 movq r2, r1
;; >pre {}()
;; >pre bwrite()
;; >pre ctx()
;; >post ctx()
;; >pre stdbuf()
;; >post stdbuf()
;; >pre :struct()
1 stw [bp - 0x4], #0x0
;; >pre <0>
;; >post <0>
;; >pre :struct()
;; >pre <0>
;; >post <0>
;; >post :struct()
2 stw [bp - 0x8], #0x0
;; >post :struct()
3 stq [bp - 0x10], #0x0
;; >post bwrite()
4 movq r1, r2
5 ldmap r2, stdbuf
6 ldw r0, #0x0
7 movw r3, #-0x1
8 movq r4, bp
9 addq r4, #-0x10
10 movq r5, #0x10
11 call perf_event_output
;; >post {}()
12 exit
error: could not link map to queue
error: unable to create buffer 'stdbuf'
ERR:-22
Hi,
First of all, thank you for your work.
I have a problem, maybe it's something stupid, but the first example is still failing for me:
$ ./configure --with-kerneldir=/usr/src/linux-4.10_p3-pf/
$ make
$ ./src/ply -v
ply-v1-beta1(9e810b1) (linux-version:264704~4.10.0)
$ uname -a
Linux erthalion 4.10.0-pf3 #2 SMP Sat Jul 15 20:07:25 CEST 2017 x86_64 Intel(R) Core(TM) i5-4210U CPU @ 1.70GHz GenuineIntel GNU/Linux
$ ./src/ply -d -c 'kprobe:SyS_*{ @[func()].count(); }'
dbg memlock_uncap : unlimited memlock
dbg annotate_script : ok
dbg map_setup : @: type:1 ksize:0x8 vsize:0x8 nelem:0x400
dbg compile_probe : kprobe:SyS_*
ERR probe_load : kprobe:SyS_* : Invalid argument
ERR probe_load : no output from kernel verifier
ERR probe_load : was ply built against the running kernel?
$ ./src/ply -D -d -c 'kprobe:SyS_*{ @[func()].count(); }'
dbg memlock_uncap : unlimited memlock
symtable:
@(kprobe:SyS_*) (type:map/int size:0x8 loc:stack/-0x8)
dbg annotate_script : ok
ast:
`-> <script> (type:script/none size:0x0 loc:nowhere)
`-> kprobe:SyS_* (type:probe/none size:0x0 loc:nowhere)
`-> <method> (type:method/none size:0x0 loc:nowhere)
`-> @ (type:map/int size:0x8 loc:stack/-0x8)
`-> <rec> (type:rec/rec size:0x8 loc:stack/-0x10)
`-> <auto>.func (type:call/int size:0x8 loc:stack/-0x10)
`-> 0x10 (type:int/none size:0x0 loc:virtual)
`-> method.count (type:call/int size:0x8 loc:nowhere)
dbg compile_probe : kprobe:SyS_*
0: mov ctx, r1
dmp compile_post : > func (call/int/0x8)
1: mov r0, #0x0
2: stdw [sp - 0x10], r0
3: mov r1, sp
4: add r1, #-0x10
5: mov r2, #0x8
6: mov r3, ctx
7: add r3, #0x80
8: call probe_read
dmp compile_post : < func (call/int/0x8)
dmp compile_post : > <rec> (rec/rec/0x8)
dmp compile_post : < <rec> (rec/rec/0x8)
dmp compile_post : > @ (map/int/0x8)
9: mov r0, #0x0
10: stdw [sp - 0x8], r0
11: lddw r1, r1
12: ldw r0, #0x0
13: mov r2, sp
14: add r2, #-0x10
15: call map_lookup_elem
16: jeq r0, #0x0, +5
17: mov r1, sp
18: add r1, #-0x8
19: mov r2, #0x8
20: mov r3, r0
21: call probe_read
dmp compile_post : < @ (map/int/0x8)
dmp compile_post : > count (call/int/0x8)
22: lddw r0, [sp - 0x8]
23: add r0, #0x1
24: stdw [sp - 0x8], r0
dmp compile_post : < count (call/int/0x8)
dmp compile_post : > <method> (method/none/0)
25: lddw r1, r1
26: ldw r0, #0x0
27: mov r2, sp
28: add r2, #-0x10
29: mov r3, sp
30: add r3, #-0x8
31: mov r4, #0x0
32: call map_update_elem
dmp compile_post : < <method> (method/none/0)
33: mov r0, #0x0
34: exit
Any idea what am I doing wrong?
Hi there !
I have a problem with ply
. When I try to execute some of your README examples, I have an error code returned :
apitronix@cuillere: ~/Documents/Sources/PLY $ sudo ply 'kretprobe:SyS_read { @["size"] = quantize(retval); }'
ERR:-22
apitronix@cuillere: ~/Documents/Sources/PLY $ sudo ply 'kprobe:SyS_read / arg2 > 1024 / { @[pid] = quantize(arg2); }'
ERR:-2
apitronix@cuillere: ~/Documents/Sources/PLY $ sudo ply 'kprobe:SyS_* { @[caller] = count(); }'
ERR:-22
apitronix@cuillere: ~/Documents/Sources/PLY $ sudo ply 'kprobe:SyS_* { @[comm, pid] = count(); }'
ERR:-22
I think this problem is similar to the issue #7 , but I'm not sure. So I tried the Brendan's solution:
apitronix@cuillere: ~/Documents/Sources/PLY $ ./autogen.sh # This is fine
apitronix@cuillere: ~/Documents/Sources/PLY $ ./configure --with-kerneldir=/usr/linux
# This option don't exist anymore, then I'm not sure to use it right...
apitronix@cuillere: ~/Documents/Sources/PLY $ make # This is fine
apitronix@cuillere: ~/Documents/Sources/PLY $ sudo make install # This is fine too
Some additional information about the system:
OS version:
Linux cuillere 5.3.11-100.fc29.x86_64 #1 SMP Tue Nov 12 20:41:25 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Ply version:
ply 7978c1a (linux-version:328459~5.3.11)
Some way to run code (eg, printf()s) at the start of the program. Useful for printing headings. awk has BEGIN, which is even used in the ply source already:
[...]
autom4te.cache/output.1:BEGIN {
config.status:ac_cs_awk_cr=`$AWK 'BEGIN { print "a\rb" }' </dev/null 2>/dev/null`
config.status:echo 'BEGIN {' >"$ac_tmp/subs1.awk" &&
config.status:BEGIN {
configure:ac_cs_awk_cr=`$AWK 'BEGIN { print "a\rb" }' </dev/null 2>/dev/null`
configure:echo 'BEGIN {' >"$ac_tmp/subs1.awk" &&
configure:BEGIN {
Makefile: $(AWK) 'BEGIN { files["."] = "" } { files[$$2] = files[$$2] " " $$1; \
Makefile: BEGIN { nonempty = 0; } \
Makefile.in: $(AWK) 'BEGIN { files["."] = "" } { files[$$2] = files[$$2] " " $$1; \
Makefile.in: BEGIN { nonempty = 0; } \
[...]
# ./opensnoop.ply
ERR compile_pred : predicate @path was not in a register as expected
source:
#!/usr/bin/env ply
/*
* opensnoop trace file opens.
*/
kprobe:do_sys_open
{
# should just save ptr, pending issue #13
@path[pid()] = mem(arg(1), "128s");
}
kretprobe:do_sys_open / @path[pid()] /
{
if (retval() > 0) {
fd = retval();
errno = 0;
} else {
fd = -1;
errno = -1 * retval();
}
printf("%-6d %-16s %4d %3d %s\n", pid(), comm(), fd, errno, @path[pid()]);
@path[pid()] = nil;
}
Hi,
I got an assertion with sys_enter_openat tracepoint but not with sys_enter_open as below.
root@osboxes:/home/sgkim/work/tmp/ply# ply -d 'tracepoint:syscalls/sys_enter_openat { printf("%v", comm); }'
ply: provider/tracepoint.c:197: tracepoint_parse: Assertion `offs == type_offsetof(t, t->sou.fields[n - 1].name)' failed.
Aborted
root@osboxes:/home/sgkim/work/tmp/ply# ply -d 'tracepoint:syscalls/sys_enter_open { printf("%v", comm); }'
ply: active
^Cply: deactivating
I think it is because the offset of dfd is not matching 12 = previous offset(8) + size of __syscall_nr(4) but the offset of "filename" is padded in sys_enter_open case.
root@osboxes:/sys/kernel/debug/tracing# cat events/syscalls/sys_enter_openat/format
name: sys_enter_openat
ID: 607
format:
field:unsigned short common_type; offset:0; size:2; signed:0;
field:unsigned char common_flags; offset:2; size:1; signed:0;
field:unsigned char common_preempt_count; offset:3; size:1; signed:0;
field:int common_pid; offset:4; size:4; signed:1;
field:int __syscall_nr; offset:8; size:4; signed:1;
field:int dfd; offset:16; size:8; signed:0;
field:const char * filename; offset:24; size:8; signed:0;
field:int flags; offset:32; size:8; signed:0;
field:umode_t mode; offset:40; size:8; signed:0;
print fmt: "dfd: 0x%08lx, filename: 0x%08lx, flags: 0x%08lx, mode: 0x%08lx", ((unsigned long)(REC->dfd)), ((unsigned long)(REC->filename)), ((unsigned long)(REC->flags)), ((unsigned long)(REC->mode))
root@osboxes:/sys/kernel/debug/tracing# cat events/syscalls/sys_enter_open/format
name: sys_enter_open
ID: 609
format:
field:unsigned short common_type; offset:0; size:2; signed:0;
field:unsigned char common_flags; offset:2; size:1; signed:0;
field:unsigned char common_preempt_count; offset:3; size:1; signed:0;
field:int common_pid; offset:4; size:4; signed:1;
field:int __syscall_nr; offset:8; size:4; signed:1;
field:const char * filename; offset:16; size:8; signed:0;
field:int flags; offset:24; size:8; signed:0;
field:umode_t mode; offset:32; size:8; signed:0;
print fmt: "filename: 0x%08lx, flags: 0x%08lx, mode: 0x%08lx", ((unsigned long)(REC->filename)), ((unsigned long)(REC->flags)), ((unsigned long)(REC->mode))
This is tested with the commit aa5b9ac and kernel version 5.0.0-23, so could you have a look on this?
I tried to check it by myself, but have no idea how to fix it without breaking other cases.
Hello,
fuzzing ply_fparse has revealed an issue that causes a segmentation fault. It can also be reproduced by running ply with the generated file directly. The file is attached below, as an attachment to this issue, as it contains unprintable characters.
crash.zip
I've reproduced the crash by running src/ply/.libs/ply with the file. Valgrind's output is shown below:
==1716== Invalid write of size 1
==1716== at 0x483CFF4: strncpy (vg_replace_strmem.c:550)
==1716== by 0x4873FA1: node_string (node.c:201)
==1716== by 0x4870641: yylex (lexer.l:65)
==1716== by 0x4867C0D: yyparse (grammar.c:1504)
==1716== by 0x487AE22: ply_fparse (libply.c:148)
==1716== by 0x10A809: main (ply.c:194)
==1716== Address 0x4a82d80 is 0 bytes after a block of size 0 alloc'd
==1716== at 0x483BB65: calloc (vg_replace_malloc.c:760)
==1716== by 0x4873F5D: xcalloc (utils.h:65)
==1716== by 0x4873F5D: node_string (node.c:200)
==1716== by 0x4870641: yylex (lexer.l:65)
==1716== by 0x4867C0D: yyparse (grammar.c:1504)
==1716== by 0x487AE22: ply_fparse (libply.c:148)
==1716== by 0x10A809: main (ply.c:194)
After analyzing the issue, the cause seems to be the statement len = strlen(data) - 2;
. In this case strlen(data) is 1, so subtracting 2 causes an underflow, which then causes the strncpy in node.c:201 to write outside of the bounds of the unquoted buffer.
This patch seems to fix the crash and if it is appropriate I can open a pull request with it.
diff --git a/src/libply/node.c b/src/libply/node.c
index 5a14a07..8885639 100644
--- a/src/libply/node.c
+++ b/src/libply/node.c
@@ -195,7 +195,10 @@ struct node *node_string(const struct nloc *loc, char *data)
if (data[0] == '"') {
char *unquoted;
- len = strlen(data) - 2;
+ len = strlen(data);
+ if (len > 1 && data[len - 1] == '"') {
+ len -= 2;
+ }
unquoted = xcalloc(1, len + 1);
strncpy(unquoted, data + 1, len);
However, I'm not sure if that is appropriate and whether a fix in lexer.l would work better, as I'm not sure why the regex in lexer.l:65 matched the single quote as a string.
If additional information about the crash is required, or if I should change something in the patch, please let me know.
Regards,
Juraj
When making the man pages, 'ronn' is required, which I didn't have:
Making all in man
make[2]: Entering directory 'HOME/ply/man'
ronn <ply.1.ronn >ply.1
/bin/sh: ronn: command not found
Makefile:520: recipe for target 'ply.1' failed
Easiest thing for me was to install ruby-ronn, at least on debian.
arg(0) could be the return value
Hi, I failed to run kprobe command in ply due to the KERNEL_VERSION_CODE checking failed. The kernel image version code actually is 265829, but the ply version printed out is 200960. How can I include the correct version.h file in ply configure command? Currently the configuration command I use is "./configure --host=aarch64-linux-gnu --prefix=/home/data/bpf/ply/install CPPFLAGS="-I /home/data/stpu/kernel/linux/include/uapi -I /home/data/stpu/kernel/linux/include/". Thanks.
I am having some trouble getting ply going on an arm chromebook.
Linux localhost 4.19.184 #29 SMP PREEMPT Mon Apr 12 17:19:19 AEST 2021 aarch64 GNU/Linux
I compiled using the gentoo ebuild: https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-util/ply
Which uses ply-2.1.1 + an arm patch: https://github.com/iovisor/ply/commit/1bc183af8703023e377f92716ecd3f339ffffd11.patch
# ply 'kprobe:do_sys_open { printf("%v(%v): %s\n", comm, uid, str(arg1)); }'
error: could not link map to queue
error: unable to create buffer 'stdbuf'
ERR:-22
Which looks like this issue: #48
I am not sure the best testcase to run, nothing works that I can see.
Here is the strace snippet:
openat(AT_FDCWD, "/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 3
read(3, "0-3\n", 8192) = 4
close(3) = 0
access("k:schedule { exit(0); }", R_OK) = -1 ENOENT (No such file or directory)
getpid() = 19458
stat64("/var/tmp/ply-ksyms", {st_mode=S_IFREG|0644, st_size=192, ...}) = 0
openat(AT_FDCWD, "/var/tmp/ply-ksyms", O_RDWR) = 3
mmap2(NULL, 192, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0xead4c000
stat64("/proc", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
stat64("/var/tmp/ply-ksyms", {st_mode=S_IFREG|0644, st_size=192, ...}) = 0
ugetrlimit(RLIMIT_MEMLOCK, {rlim_cur=64*1024, rlim_max=64*1024}) = 0
prlimit64(0, RLIMIT_MEMLOCK, {rlim_cur=RLIM64_INFINITY, rlim_max=RLIM64_INFINITY}, NULL) = 0
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=4, map_flags=0, inner_map_fd=0, map_name="", map_ifindex=0}, 48) = 4
perf_event_open({type=PERF_TYPE_SOFTWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=PERF_COUNT_SW_BPF_OUTPUT, ...}, -1, 0, -1, 0) = 5
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=4, key=0xffe63d34, value=0x106f57e4, flags=BPF_ANY, ...}, 48) = -1 EINVAL (Invalid argument)
write(2, "error: could not link map to que"..., 35error: could not link map to queue
Let me know what else I can try
I built ply cfc56c7 against Linux 4.8.4_1, but it doesn't seem to like the bpf code:
% sudo strace ./src/ply -c 'kprobe:SyS_open { $opens.count() }'
...
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_HASH, key_size=8, value_size=8, max_entries=512}, 48) = 3
open("/sys/kernel/debug/tracing/kprobe_events", O_RDWR|O_CREAT|O_APPEND, 0666) = 4
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=29, insns=0x1315558, license="GPL", log_level=1, log_size=131072, log_buf=0x611780, kern_version=262415}, 48) = -1 EINVAL (Invalid argument)
Hello,
the file attached below causes a segmentation fault. It was found by fuzzing ply_fparse and can be reproduced by running ply with the file as an argument.
nullptr.zip
The issue seems to be in node_append in node.c:263. When the file is parsed, tail is NULL, so tail->prev is a NULL pointer dereference. Returning NULL if tail is NULL seems to fix the issue and ply then successfully reports an error with the following output:
[root@fuzz ply]# ./src/ply/.libs/ply ./nullptr
1: error: syntax error, unexpected IDENT, expecting ';'
ERR:-22
Regards,
Juraj
Hello,
the following file, found by fuzzing ply, attached below, causes ply to segfault when it is ran with the file.
nullptr.zip
The issue seems to be caused by a null dereference in type.h in type_base. As struct type *t is NULL, accessing t->ttype accesses invalid memory.
The following patch seems to fix the issue:
diff --git a/src/libply/built-in/memory.c b/src/libply/built-in/memory.c
index b3780e6..2dbe056 100644
--- a/src/libply/built-in/memory.c
+++ b/src/libply/built-in/memory.c
@@ -261,6 +261,7 @@ static int struct_deref_type_infer(const struct func *func, struct node *n)
if (t->ttype != T_POINTER) {
_ne(n, "%N is not a pointer (type '%T').\n",
sou, sou->sym->type);
+ return -EINVAL;
}
t = type_base(t->ptr.type);
The unpatched version produces the following output, so I guess that exiting after the issue has been detected should work as a fix:
[root@fuzz ply]# ./src/ply/.libs/ply ./nullptr
1: error: unknown token
<input>:1:3-9: error: 2 is not a pointer (type 'int').
Segmentation fault (core dumped)
Regards,
Juraj
I'm trying to build ply on an in-house enterprise distribution which has a backported GCC but is still using a very old YACC (version 1.9 20070509); your parser doesn't compile with that version of YACC:
% make
Making all in src
make[1]: Entering directory `/home/build/pg/ply/src'
YACC lang/parse.c
/home/build/pg/ply/src/lang/parse.y:20.21-27: syntax error, unexpected identifier
make[1]: *** [lang/parse.c] Error 1
make[1]: Leaving directory `/home/build/pg/ply/src'
make: *** [all-recursive] Error 1
Do you know what version of YACC I would need to backport in order to get this to build?
ply 'kprobe:do_sys_open { printf("opened:%s\n", str(arg1));}'
[1] 41739
error: unable to load kprobe:do_sys_open, errno:22
error: no output from kernel bpf verifier, retry with debugging enabled
ERR:-22
ply 'kprobe:watchdog_timer_fn { @[stack] = count(); }'
error: unable to load kprobe:watchdog_timer_fn, errno:22
error: no output from kernel bpf verifier, retry with debugging enabled
ERR:-22
Hi there,
Before we can clear map with assignment 'nil', seems now the latest code base doesn't support 'nil' anymore. So e.g. below program cannot work.
kprobe:kmem_cache_alloc_node {
s[cpu,kpid] = stack();
}
kretprobe:kmem_cache_alloc_node {
@[retval] = s[cpu,kpid];
s[cpu,kpid] = nil;
}
kprobe:kmem_cache_free {
@[arg1] = nil;
}
If I change 'nil' to 0, the program can run successfully. But the assignment for the sentence "@[retval] = s[cpu,kpid]" also cannot work as expected, I observe the result as below:
{ -281474254472384 }:
{ -281474254472000 }:
{ -281474231713792 }:
{ -281474177160960 }:
{ -281474148187360 }:
{ -281474147794048 }:
{ -281474147622912 }:
{ -281474132892352 }:
{ -281474132891456 }:
{ -281474132887872 }:
{ -281474132887424 }:
{ -281474132886528 }:
Thanks for the suggestion!
Use-case: I'm trying to trace long page faults.
#!/usr/sbin/ply
kprobe:handle_mm_fault
{
@lat[pid()] = nsecs();
}
kretprobe:handle_mm_fault
{
start = @lat[pid()]; # Fetch the time at handle_mm_fault() entry.
if (start == 0) {
return; # Don't proveed if we don't have a start time.
}
lat = nsecs() - start;
limit = 1000000;
if (lat > limit) {
printf("%16s(%5d): handle_mm_fault took more than %dus (%dus)", comm(), pid(), limit / 1000, lat / 1000);
printf("%v\n\n", stack());
}
@lat[pid()] = nil; # Clear map.
}
This produces output like the following:
git(191300): handle_mm_fault took more than 1000us (9121us)
kretprobe_trampoline
trace_page_fault+0x28
git(191300): handle_mm_fault took more than 1000us (1524us)
kretprobe_trampoline
trace_page_fault+0x28
cron(191662): handle_mm_fault took more than 1000us (1028us)
kretprobe_trampoline
trace_page_fault+0x28
zsh(190053): handle_mm_fault took more than 1000us (2037us)
kretprobe_trampoline
__wake_up_parent
trace_page_fault+0x28
__wake_up_parent
__put_user_4+0x1c
schedule_tail+0x40
ret_from_fork+0x8
chrome(72654): handle_mm_fault took more than 1000us (3971us)
kretprobe_trampoline
trace_page_fault+0x28
nvim(190453): handle_mm_fault took more than 1000us (2483us)
kretprobe_trampoline
trace_page_fault+0x28
__clear_user+0x21
load_elf_binary+0x12e6
search_binary_handler+0xb8
search_binary_handler+0xa0
do_execveat_common.isra.37+0x5aa
sys_execve+0x35
kretprobe_trampoline
return_from_SYSCALL_64
Sometimes the stacktrace looks like its rooted at kernel entry (return_from_SYCALL_64
), but most of the time I only see the kretprobe_trampoline
and trace_page_fault
.
I'm running a Linux 4.9.0 kernel. Looking for trace_page_fault
, I only find assembly:
#ifdef CONFIG_TRACING
ENTRY(trace_page_fault)
ASM_CLAC
pushl $trace_do_page_fault
jmp error_code
END(trace_page_fault)
#endif
Could that be what's messing with the stack creation?
I tried running the same with trace(8)
from bcc-tools and it appears that this is not ply's fault:
$ sudo trace -K -I linux/mm.h 'r::handle_mm_fault(struct vm_area_struct *vma, unsigned long address, unsigned int flags) "%d", retval'
46540 46540 chrome handle_mm_fault 0
kretprobe_trampoline+0x0 [kernel]
page_fault+0x28 [kernel]
46540 46540 chrome handle_mm_fault 0
kretprobe_trampoline+0x0 [kernel]
page_fault+0x28 [kernel]
Strangely enough, the referenced function is called trace_fault
here, instead of trace_page_fault
. Does ply
have custom code to avoid printing any trace_
prefix?
Another question/suggestions: perhaps ply
should filter out the kretprobe_trampoline
stuff (unless in debugging mode)?
So I guess this is just a heads up. I notice the `page_fault
Final note: non-kretprobe tracing has saner stacks:
$ sudo trace -K -I linux/mm.h 'p::handle_mm_fault(struct vm_area_struct *vma, unsigned long address, unsigned int flags) "%d", address
45283 45283 chrome handle_mm_fault 72798280
handle_mm_fault+0x1 [kernel]
__do_page_fault+0x25c [kernel]
page_fault+0x28 [kernel]
45283 45283 chrome handle_mm_fault 72798280
handle_mm_fault+0x1 [kernel]
__do_page_fault+0x25c [kernel]
page_fault+0x28 [kernel]
Although I didn't expect it to be this short, it makes ~sense.
Thanks for building a great low-dependency tool, its code is a joy to read (the parts I've seen, at least).
Hi,
Ply looks like an awesome tool, and my embedded software team is very excited to try/use it.
We successfully cross-compiled the tool for arm-linux using arm-linux-androideabi-clang, and we pushed the tool to our embedded device.
The tool compiled correctly. i.e. the following two commands work on our arm embedded device:
./ply
error: no input
ply - Dynamic tracing utility
Usage:
ply [options] <ply-text>
ply [options] <ply-file>
Options:
-c COMMAND Run COMMAND in a shell, exit upon completion.
-d Enable debug output.
-e Exit after compiling.
-h Print usage message and exit.
-k Keep going in face of trace buffer overruns.
-S Show generated BPF.
-v Print version information.
./ply -v
ply 15c164b (linux-version:328450~5.3.2)
However, we are unable to run a basic ply test:
ply 'kprobe:do_sys_open { printf("%v(%v): %s\n", comm, uid, str(arg1)); }' -d
When running the above command, we get the following error:
info: creating kallsyms cache
warning: unable to create kallsyms cache: No such file or directory
error: unable to load kprobe:do_sys_open, errno:22
warning: was ply built against the running kernel?
ERR:-22
In ply 1.x, there was a configure param --with-kerneldir that devs could use to set a custom kernel distro (other than kernel version on host build machine).
In 2.1.1 it seems this option is unrecognized as part of the configure process.
configure: WARNING: unrecognized options: --with-kerneldir
How can we compile ply 2.1.1 for the kernel version used on our embedded device, rather than the kernel version of our local development linux machines?
We are using kernel v4.14 on embedded device, but version 4.15 on our local dev ubuntu machines.
Thanks,
Eric
# ./opensnoop.ply
ERR static_post : node:<auto>.mem (type:call/none size:0x0 loc:nowhere) : Invalid argument
ERR annotate_script : static type inference failed
code:
#!/usr/bin/env ply
/*
* opensnoop trace file opens.
*/
kprobe:do_sys_open
{
@path[pid()] = arg(1);
}
kretprobe:do_sys_open / @path[pid()] /
{
if (retval() > 0) {
fd = retval();
errno = 0;
} else {
fd = /* minus */ 1;
errno = /* minus */ retval();
}
printf("%-6d %-16s %4d %3d %s\n", pid(), comm(), fd, errno,
mem(@path[pid()], "128s"));
@path[pid()] = nil;
}
(Ignore the "/* minus */", which are pending #11 .)
This code is pretty nice BTW.
vfs_caches_init() can't be traced on my machine. It's in /proc/kallsyms, but it is not in /sys/kernel/debug/tracing/available_filter_functions. This is what happens with bcc:
# ./bcc/tools/trace.py vfs_caches_init
check dmesg output for possible cause
Failed to attach BPF to kprobe
dmesg | tail -1
[ 6679.602959] Could not insert probe at vfs_caches_init+0: -22
# ./bcc/tools/funccount.py 'vfs_*'
Tracing 49 functions for "vfs_*"... Hit Ctrl-C to end.
^C
FUNC COUNT
vfs_fstat 1
vfs_getattr_nosec 1
vfs_getattr 1
vfs_read 32
vfs_open 40
vfs_write 48
Which is ok. funccount is skipping it entirely, and trace is trying (I'm not sure it should) but failing.
With ply:
# ply -c 'kprobe:vfs_caches_init { printf("hi\n"); }'
ERR probe_event_id : "kprobes/p_vfs_caches_init_0" : No such file or directory
# ply -c 'kprobe:vfs* { @[func()].count(); }'
ERR probe_event_id : "kprobes/p_vfs_caches_init_early_0" : No such file or directory
ERR probe_event_id : "kprobes/p_vfs_caches_init_0" : No such file or directory
#
The problem is the second invocation. Maybe ply should just skip probes not in available_filter_functions?
I want to package ply
for ALT Linux. But, most basic eamples I try failing with different errors. What this could be?
5.4.51-std-def-alt1:~# cat > count-syscallt.ply
#!/usr/bin/env ply
kprobe:SyS_*
{
@syscalls[caller] = count();
}
5.4.51-std-def-alt1:~# ply count-syscallt.ply
0 movq v0, r1
;; >pre {}()
;; >pre =()
;; >pre []()
;; >pre @syscalls()
;; >post @syscalls()
;; >pre caller()
;; >pre .()
;; >pre u*()
;; >pre regs()
;; >pre ctx()
;; >post ctx()
;; >post regs()
;; >post u*()
;; >pre "rip"
;; >post "rip"
;; >post .()
1 movq r3, v0
2 addq r3, #0x80
3 movq r2, #0x8
4 movq r1, bp
5 addq r1, #-0x10
6 call probe_read
;; >post caller()
7 ldq r0, [bp - 0x10]
8 stq [bp - 0x18], r0
;; >post []()
;; >pre count()
;; >post count()
9 ldq r0, [bp - 0x8]
10 addq r0, #0x1
11 stq [bp - 0x8], r0
;; >post =()
ply: ir.c:389: ir_emit_sym_to_stack: Assertion `0' failed.
Aborted
5.4.51-std-def-alt1:~#
5.4.51-std-def-alt1:~# ply 'kretprobe:vfs_read { @["size"] = quantize(retval); }'
0 movq v0, r1
;; >pre {}()
;; >pre =()
;; >pre []()
;; >pre @()
;; >post @()
;; >pre "size"
;; >post "size"
1 stw [bp - 0x108], #0x657a6973
2 stw [bp - 0x104], #0x0
;; >post []()
;; >pre quantize()
;; >pre retval()
;; >pre .()
;; >pre u*()
;; >pre regs()
;; >pre ctx()
;; >post ctx()
;; >post regs()
;; >post u*()
;; >pre "rax"
;; >post "rax"
;; >post .()
3 movq r3, v0
4 addq r3, #0x50
5 movq r2, #0x8
6 movq r1, bp
7 addq r1, #-0x110
8 call probe_read
;; >post retval()
9 ldq v1, [bp - 0x110]
;; >post quantize()
10 movq r0, #0x0
11 movq r1, v1
12 movq r2, r1
13 rshq r2, #0x20
14 jeq r2, #0x0, +2
15 addq r0, #0x20
16 movq r1, r2
17 jle r1, #0xffff, +2
18 addq r0, #0x10
19 rshq r1, #0x10
20 jle r1, #0xff, +2
21 addq r0, #0x8
22 rshq r1, #0x8
23 jle r1, #0xf, +2
24 addq r0, #0x4
25 rshq r1, #0x4
26 jle r1, #0x3, +2
27 addq r0, #0x2
28 rshq r1, #0x2
29 jle r1, #0x1, +2
30 addq r0, #0x1
31 rshq r1, #0x1
32 lshq r0, #0x2
33 movq r1, bp
34 addq r1, #-0x100
35 addq r1, r0
36 movq r0, #0x1
37 stw r1, r0
;; >post =()
ply: ir.c:389: ir_emit_sym_to_stack: Assertion `0' failed.
Aborted
5.4.51-std-def-alt1:~#
5.4.51-std-def-alt1:~# ply 'kprobe:dev_queue_xmit { @[stack] = count(); }'
0 movq v0, r1
;; >pre {}()
;; >pre =()
;; >pre []()
;; >pre @()
;; >post @()
;; >pre stack()
;; >pre ctx()
;; >post ctx()
;; >pre :stackmap()
;; >post :stackmap()
;; >post stack()
1 movq r1, v0
2 ldmap r2, :stackmap
3 ldw r0, #0x0
4 movq r3, #0x0
5 call get_stackid
6 stw [bp - 0xc], r0
;; >post []()
;; >pre count()
;; >post count()
7 ldq r0, [bp - 0x8]
8 addq r0, #0x1
9 stq [bp - 0x8], r0
;; >post =()
ply: ir.c:389: ir_emit_sym_to_stack: Assertion `0' failed.
Aborted
5.4.51-std-def-alt1:~#
5.4.51-std-def-alt1:~# cat execsnoop.ply
#!/sbin/ply
kprobe:SyS_execve {
execs[kpid] = str(arg0, 48);
}
kretprobe:SyS_execve {
printf("(%4u) %v %3ld\n", uid, execs[kpid], retval);
}
5.4.51-std-def-alt1:~# ply execsnoop.ply
error: unable to load kretprobe:SyS_execve, errno:13
error: no output from kernel bpf verifier, retry with debugging enabled
ERR:-13
5.4.51-std-def-alt1:~#
ply 'kprobe:do_sys_open { printf("%v(%v): %s\n", comm, uid, str(arg1)); }'
this example works!Is it possible to do this? It'd be nice to have this, so that I can implement execsnoop with return code check. Currently, I can only do stuff like this:
kprobe:sys_execve {
x = 0;
printf("%s: ", mem(arg(0), "128s"));
unroll (10) {
if (mem(arg(1) + x*sizeof("p"), "p")) {
printf("%s ", mem(mem(arg(1) + x*sizeof("p"), "p"), "128s"));
x=x+1;
}
}
printf("\n");
}
# ply -c 'kprobe:SyS_read { $bytes.quantize(arg(2)); }'
1 probe active
^Cde-activating probes
$bytes:
[ 0, 1] 698
[ 64, 128) 31
[ 128, 256) 2
[ 512, 1k) 109
[ 1k, 2k) 18
[ 4k, 8k) 28
[ 16k, 32k) 3
This is a good terse output for power-of-2 histograms, however, I think it should be optional, and the default should be something more easily readable, like:
value : count distribution
0 -> 1 : 3 | |
2 -> 3 : 0 | |
4 -> 7 : 211 |********** |
8 -> 15 : 0 | |
16 -> 31 : 0 | |
32 -> 63 : 0 | |
64 -> 127 : 1 | |
128 -> 255 : 800 |**************************************|
Note that zero rows are included, as well as an ASCII distribution. (The ASCII distribution would make less sense if the zero rows were elided). Optionally, the heading "value" could be changed. Eg, in this case, to "bytes".
This could be accomplished a number of ways. Here's one suggestion:
# emits the raw output (as it does today):
ply -c 'kprobe:SyS_read { $bytes.quantize(arg(2), raw=true); }'
# emits the sample output above:
ply -c 'kprobe:SyS_read { $bytes.quantize(arg(2)); }'
# emits the sample output, but with the value column renamed:
ply -c 'kprobe:SyS_read { $bytes.quantize(arg(2), value="bytes"); }'
I've opted against just adding optional arguments to quantize (eg, "quantize(arg(2), true, "bytes"). I think positional arguments like those should be for the common use cases. In cases where the argument would be uncommonly used, a key=value approach may be easier for users to learn and remember, since it self documents.
kernel version :5.7
I want to get variable ’duration‘ value in watchdog_timer_fn, how i should do ?
such as:
ply 'kprobe:watchdog_timer_fn+ offset {xxxxx}'
Can you provide a full command for me ?
Thanks.
Hi there,
When I test syscall-fail.ply script, it reports error:
root@linaro-developer:~/ply/scripts# ply syscall-fail.ply
error(6): syntax error, unexpected '{'
I tried to modify for ply script, but have no luck to fix it. Could you check for this?
typo: %s instead of %d:
# ply -c 'kprobe:do_sys_open { printf("pid:%s\n", pid()); }'
1 probe active
pid:s
pid:?
pid:?_
pid:s
pid:?
pid:s
pid:?
pid:s
[...]
I suppose it is doing what I told it to do, but it would be nice if it threw an error or warning.
Mistyping arg(0) as arg0:
# ply -c 'kprobe:SyS_open { printf("open %s\n", arg0); }'
ERR probe_load : kprobe:SyS_open : Permission denied
ERR probe_load : output from kernel bpf verifier:
0: (bf) r9 = r1
1: (62) *(u32 *)(r10 +0) = 0
invalid stack off=0 size=4
Same as:
# ply -c 'kprobe:SyS_open { printf("open %s\n", foo); }'
ERR probe_load : kprobe:SyS_open : Permission denied
ERR probe_load : output from kernel bpf verifier:
0: (bf) r9 = r1
1: (62) *(u32 *)(r10 +0) = 0
invalid stack off=0 size=4
Should say something like: "foo" unknown.
Hello,
the files attached below cause a segmentation fault in ply. The crashes were found by fuzzing ply_fparse, like #55 and the crashes can be reproduced the same way, by feeding the files into ply.
crashes.zip
The first input, in the invalidptr1 file causes a null pointer dereference in node_nloc_valid. However, checking whether n is NULL in node_vfprintxf doesn't prevent all the crashes.
The second file invalidptr2 still causes the same crash, and bypasses the NULL pointer check, as n in that case takes a value of 1.
If any additional information is needed regarding the crash, please let me know.
Regards,
Juraj
Maybe I'm doing something wrong:
(root) /mnt/src/ply # ply -c 'kprobe:SyS_* { @syscalls[func()].count() }'
ERR probe_load : kprobe:SyS_* : Invalid argument
ERR probe_load : no output from kernel verifier
ERR probe_load : was ply built against the running kernel?
(root) /mnt/src/ply # ply -c 'kprobe:tcp_sendmsg { @names[func()].count() }'
ERR probe_load : kprobe:tcp_sendmsg : Invalid argument
ERR probe_load : no output from kernel verifier
ERR probe_load : was ply built against the running kernel?
(root) /mnt/src/ply # uname -a
Linux bgregg-xenia 4.9.0-rc5-virtual #1 SMP Fri Nov 18 22:01:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
(root) /mnt/src/ply # ply -v
ply-v1-beta1(e2cc48e) (linux-version:263203~4.4.35)
(root) /mnt/src/ply # ply -D -c 'kprobe:tcp_sendmsg { @names[func()].count() }'
symtable:
@names(kprobe:tcp_sendmsg) (type:map/int size:0x8 loc:stack/-0x8)
ast:
`-> <script> (type:script/none size:0x0 loc:nowhere)
`-> kprobe:tcp_sendmsg (type:probe/none size:0x0 loc:nowhere)
`-> <method> (type:method/none size:0x0 loc:nowhere)
`-> @names (type:map/int size:0x8 loc:stack/-0x8)
`-> <rec> (type:rec/rec size:0x8 loc:stack/-0x10)
`-> <auto>.func (type:call/int size:0x8 loc:stack/-0x10)
`-> 0x10 (type:int/none size:0x0 loc:virtual)
`-> method.count (type:call/int size:0x8 loc:nowhere)
0: mov ctx, r1
dmp compile_post : > func (call/int/0x8)
1: mov r0, #0x0
2: stdw [sp - 0x10], r0
3: mov r1, sp
4: add r1, #-0x10
5: mov r2, #0x8
6: mov r3, ctx
7: add r3, #0x80
8: call probe_read
dmp compile_post : < func (call/int/0x8)
dmp compile_post : > <rec> (rec/rec/0x8)
dmp compile_post : < <rec> (rec/rec/0x8)
dmp compile_post : > @names (map/int/0x8)
9: mov r0, #0x0
10: stdw [sp - 0x8], r0
11: lddw r1, r1
12: ldw r0, #0x0
13: mov r2, sp
14: add r2, #-0x10
15: call map_lookup_elem
16: jeq r0, #0x0, +5
17: mov r1, sp
18: add r1, #-0x8
19: mov r2, #0x8
20: mov r3, r0
21: call probe_read
dmp compile_post : < @names (map/int/0x8)
dmp compile_post : > count (call/int/0x8)
22: lddw r0, [sp - 0x8]
23: add r0, #0x1
24: stdw [sp - 0x8], r0
dmp compile_post : < count (call/int/0x8)
dmp compile_post : > <method> (method/none/0)
25: lddw r1, r1
26: ldw r0, #0x0
27: mov r2, sp
28: add r2, #-0x10
29: mov r3, sp
30: add r3, #-0x8
31: mov r4, #0x0
32: call map_update_elem
dmp compile_post : < <method> (method/none/0)
33: mov r0, #0x0
34: exit
I found this works, with the kernel headers package only (Ubuntu Yakkety):
$ git clone https://github.com/iovisor/ply
$ cd ply
$ ./autogen.sh
$ ./configure --with-kerneldir=/usr/src/linux-headers-4.8.0-30-generic
$ make
Making all in src
make[1]: Entering directory '/root/ply/src'
LEX lang/lex.c
YACC lang/parse.c
updating lang/parse.h
HEADERS /usr/src/linux-headers-4.8.0-30-generic(x86_64)
make -C /usr/src/linux-headers-4.8.0-30-generic ARCH=x86_64 INSTALL_HDR_PATH=/root/ply/src/.kernel headers_install
make[2]: Entering directory '/usr/src/linux-headers-4.8.0-30-generic'
CHK include/generated/uapi/linux/version.h
make[3]: *** No rule to make target 'arch/x86/entry/syscalls/syscall_32.tbl', needed by 'arch/x86/entry/syscalls/../../include/generated/asm/syscalls_32.h'. Stop.
arch/x86/Makefile:191: recipe for target 'archheaders' failed
make[2]: *** [archheaders] Error 2
make[2]: Leaving directory '/usr/src/linux-headers-4.8.0-30-generic'
Makefile:1089: recipe for target '.kernel/include/linux/version.h' failed
make[1]: *** [.kernel/include/linux/version.h] Error 2
make[1]: Leaving directory '/root/ply/src'
Makefile:402: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1
$ vi src/Makefile
[...]
.kernel/include/linux/version.h:
@echo " HEADERS /usr/src/linux-headers-4.8.0-30-generic(x86_64)"
# make -C /usr/src/linux-headers-4.8.0-30-generic $(kernelenv) headers_install
[...]
$ make
$ sudo make install
$ ply -c 'kprobe:schedule { @cs[comm()].count(); }'
1 probe active
^Cde-activating probes
@cs:
gmain 1
watchdog/0 1
ply 3
kworker/0:1 4
kworker/u2:2 4
rcu_sched 5
iscsid 7
kworker/u2:0 10
sshd 10
swapper/0 18
So... Do we need the headers_install?
I noticed trace_pipe was full of tracepoint events after using ply; I think the problem is that it's enabling all tracepoints:
/sys/kernel/debug/tracing # for f in `find . -name enable`; do read i < $f; (( i )) && echo $f $i; done
[...]
./events/xen/xen_mmu_set_domain_pte/enable 1
./events/xen/xen_mmu_set_pte_at/enable 1
./events/xen/xen_mmu_pte_clear/enable 1
./events/xen/xen_mmu_set_pmd/enable 1
./events/xen/xen_mmu_pmd_clear/enable 1
./events/xen/xen_mmu_set_pud/enable 1
./events/xen/xen_mmu_set_pgd/enable 1
./events/xen/xen_mmu_pud_clear/enable 1
./events/xen/xen_mmu_pgd_clear/enable 1
./events/xen/xen_mmu_ptep_modify_prot_start/enable 1
./events/xen/xen_mmu_ptep_modify_prot_commit/enable 1
./events/xen/xen_mmu_alloc_ptpage/enable 1
./events/xen/xen_mmu_release_ptpage/enable 1
./events/xen/xen_mmu_pgd_pin/enable 1
./events/xen/xen_mmu_pgd_unpin/enable 1
./events/xen/xen_mmu_flush_tlb_all/enable 1
./events/xen/xen_mmu_flush_tlb/enable 1
./events/xen/xen_mmu_flush_tlb_single/enable 1
./events/xen/xen_mmu_flush_tlb_others/enable 1
./events/xen/xen_mmu_write_cr3/enable 1
./events/xen/xen_cpu_write_ldt_entry/enable 1
./events/xen/xen_cpu_write_idt_entry/enable 1
./events/xen/xen_cpu_load_idt/enable 1
./events/xen/xen_cpu_write_gdt_entry/enable 1
./events/xen/xen_cpu_set_ldt/enable 1
./events/xen/enable 1
./events/vsyscall/emulate_vsyscall/enable 1
./events/vsyscall/enable 1
./events/raw_syscalls/sys_enter/enable 1
./events/raw_syscalls/sys_exit/enable 1
./events/raw_syscalls/enable 1
/sys/kernel/debug/tracing # for f in `find . -name enable`; do read i < $f; (( i )) && echo $f $i; done|wc
1804 3608 79468
seems the same. like both lightweight and the samilar syntax
Hi,
Can I use ply
for i586
architecture? When I try to compile ply
I see the follow error:
| make all-recursive
| make[1]: Entering directory `/local/workspace/ply/2.1.0-r2'
| Making all in include
| make[2]: Entering directory `/local/workspace/ply/2.1.0-r2/include'
| make[2]: Nothing to be done for `all'.
| make[2]: Leaving directory `/local/workspace/ply/2.1.0-r2/include'
| Making all in src
| make[2]: Entering directory `/local/workspace/ply/2.1.0-r2/src'
| Making all in libply
| make[3]: Entering directory `/local/workspace/ply/2.1.0-r2/src/libply'
| make all-am
| make[4]: Entering directory `/local/workspace/ply/2.1.0-r2/src/libply'
| make[4]: *** No rule to make target `arch/i586.c', needed by `arch/libply_la-i586.lo'. Stop.
| make[4]: Leaving directory `/local/workspace/ply/2.1.0-r2/src/libply'
| make[3]: *** [all] Error 2
| make[3]: Leaving directory `/local/workspace/ply/2.1.0-r2/src/libply'
| make[2]: *** [all-recursive] Error 1
| make[2]: Leaving directory `/local/workspace/ply/2.1.0-r2/src'
| make[1]: *** [all-recursive] Error 1
| make[1]: Leaving directory `/local/workspace/ply/2.1.0-r2'
| make: *** [all] Error 2
| autoreconf: make failed with exit status: 2
Hello,
First, thanks for the great piece of software. I can confirm it works on (armv7l) raspberry pi zero, with kernel 5.10.
I am trying to do static linking to try it on embedded system with read-only filesystem.
I am building with:
and getting following linking error. Any idea how to build ply as a static library?
...
Making all in ply
make[3]: Entering directory '/home/pi/ply/ply/src/ply'
CC ply-ply.o
cd /home/pi/ply/ply/src/ply && \
/usr/bin/ld -r -b binary -o /home/pi/ply/ply/src/ply/self-test.o self-test.sh
CCLD ply
/usr/bin/ld: ../libply/.libs/libply.a(libply_la-provider.o): in function `provider_get':
provider.c:(.text+0xb8): undefined reference to `__start_providers'
/usr/bin/ld: provider.c:(.text+0xbc): undefined reference to `__stop_providers'
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile:388: ply] Error 1
make[3]: Leaving directory '/home/pi/ply/ply/src/ply'
make[2]: *** [Makefile:356: all-recursive] Error 1
make[2]: Leaving directory '/home/pi/ply/ply/src'
make[1]: *** [Makefile:457: all-recursive] Error 1
make[1]: Leaving directory '/home/pi/ply/ply'
make: *** [Makefile:368: all] Error 2
I compiled and installed ply, but it keeps throwing probe_load errors when I try to run.
$ sudo ply -d -c 'kretprobe:SyS_read{ @.quantize(retval()); }'
dbg memlock_uncap : unlimited memlock
dbg annotate_script : ok
dbg map_setup : @: type:1 ksize:0x10 vsize:0x8 nelem:0x400
dbg compile_probe : kretprobe:SyS_read
ERR probe_load : kretprobe:SyS_read : Invalid argument
ERR probe_load : no output from kernel verifier
ERR probe_load : was ply built against the running kernel?
Is there some obvious step I am missing?
I keep hitting the same error for some of the other examples in the README
as well.
Hi,
I'm trying to build ply for arm64 target but somehow I'm unable to do it:
I used configure like this:
./configure --target=arm64 --with-kerneldir=/home/kedaru/proj/base/linux/
But I still get x86_64 ply file after a make. How do I compile ply executable for arm64 target ?
Thanks.
# ply -c 'kprobe:SyS_open { printf("open %s\n", arg(0)); }'
1 probe active
open pr??
open ?q??
open ?!A
open pr??
open ?q??
Ok, I need to use mem(). This ticket is to explore either:
A) not needing mem(). Previous command would just work.
B) a shortcut, like:
# ply -c 'kprobe:SyS_open { printf("open %s\n", str(arg(0))); }'
where str() could be:
#define str(p) (mem(p, "128s"))
Could also make the 128 a global tunable.
hi, I enabled ply in our embedded target board, and can successfully run commands like:
ply 'tracepoint:irq/irq_handler_entry { @[data->irq] = count(); }'
ply 'kretprobe:vfs_read { @[pid] = quantize(retval); }'
ply 'tracepoint:sched/sched_switch { @[kstack] = count();}'
But when I want to print syscall's argument, it can't be printed out. For example:
ply 'kprobe:do_sys_open { printf("opened:%s\n", str(arg1));}'_
then I opened a file. But only prints out:
opened
The str(arg1) seems empty...
Also met below errors when I tried to run commands in https://www.slideshare.net/brendangregg/linux-4x-tracing-performance-analysis-with-bccbpf
_sh-4.4# ply 'kprobe:do_sys_open { printf("%s %ld\n", comm, nsecs()); }'
<input>:1:46-51: error: type of symbol 'nsecs' is unknown
<input>:1:21-54: error: type of symbol ':struct' is unknown
<input>:1:21-54: error: type of symbol ':struct' is unknown
sh-4.4# ply 'kretprobe:vfs_read { @[func()] = quantize(retval); }'
<input>:1:21-22: error: type of symbol '@' is unknown
<input>:1:23-27: error: type of symbol 'func' is unknown
<input>:1:22-30: error: type of symbol ':struct' is unknown
Is it because func() and nsecs() are not available in current ply? Are there any alternative functions?Thanks.
This is my first time trying to use ply
, so I apologize if this is a self induced issue, but I'm trying to execute the following command (example taken from the man page):
$ sudo ply -c 'kprobe:SyS_* / !strcmp(execname, "dd") / { @[func()].count() }'
ERR static_post : node:<auto>.strcmp (type:call/none size:0x0 loc:nowhere) : Invalid argument
ERR annotate_script : static type inference failed
Any clues as to what might be causing the error? and/or maybe point me to some docs/source code so I can try to read up on how this is supposed to work?
I built ply
from source, and am using an Ubuntu 17.04 system:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 17.04
Release: 17.04
Codename: zesty
$ uname -a
Linux ubuntu 4.10.0-19-generic #21-Ubuntu SMP Thu Apr 6 17:04:57 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ ply -v
ply-v1-beta1(9e810b1) (linux-version:264712~4.10.8)
I had to download Ubuntu's kernel sources, and use the --with-kerneldir
option to workaround #7
I use the following instructions to compile the installation and then copy the files in the build directory to the board.
./autogen.sh
./configure --host=aarch64-linux-gnu --prefix=~/build
make
make install
After the files in the lib directory are copied to the /lib64 directory of the development board, I try to run ply in sbin directory. However, the following error message is displayed. Is the cross compilation chain incorrect?
ply: line 3: syntax error: unexpected "("
Hi, I've tried to run the test but it failed like below
ply$ cd test/
test$ ls
Makefile README.md test.conf test.sh test-wrapper.sh
test$ make
mkdir -p work/basis-ply
mkdir -p cache
wget -O cache/armv5-eabi--glibc--bleeding-edge-2020.08-1.tar.bz2 https://toolchains.bootlin.com/downloads/releases/toolchains/armv5-eabi/tarballs/armv5-eabi--glibc--bleeding-edge-2020.08-1.tar.bz2
--2021-09-22 12:41:46-- https://toolchains.bootlin.com/downloads/releases/toolchains/armv5-eabi/tarballs/armv5-eabi--glibc--bleeding-edge-2020.08-1.tar.bz2
Resolving toolchains.bootlin.com (toolchains.bootlin.com)... 51.254.149.199
Connecting to toolchains.bootlin.com (toolchains.bootlin.com)|51.254.149.199|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 121154542 (116M) [application/x-bzip2]
Saving to: ‘cache/armv5-eabi--glibc--bleeding-edge-2020.08-1.tar.bz2’
cache/armv5-eabi--glibc--bleedi 100%[=======================================================>] 115.54M 168KB/s in 11m 0s
2021-09-22 12:52:47 (179 KB/s) - ‘cache/armv5-eabi--glibc--bleeding-edge-2020.08-1.tar.bz2’ saved [121154542/121154542]
tar -C cache -maxf cache/armv5-eabi--glibc--bleeding-edge-2020.08-1.tar.bz2
cd work/basis-ply cache/armv5-eabi--glibc--bleeding-edge-2020.08-1 && PATH=/home/namhyung/.local/bin:/home/namhyung/.cargo/bin:/home/namhyung/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/namhyung/go/bin:/home/namhyung/project/ply/test/cache/armv5-eabi--glibc--bleeding-edge-2020.08-1/bin ../../../configure CFLAGS="-Wall -Wextra -Werror" --host=arm-linux --prefix=
/bin/sh: line 1: cd: too many arguments
make: *** [Makefile:151: work/basis-ply/Makefile] Error 1
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.