Provide easy way to run `contrib` script from CLI about drgn HOT 13 OPEN

Oh boy, I did not realize that you would be interested in IPython magics! I have some groundwork for that which I could share! I will try to do it Monday :)

from drgn.

They even have nice decorators where one can define arguments of a magic command:
https://ipython.readthedocs.io/en/stable/api/generated/IPython.core.magic_arguments.html#IPython.core.magic_arguments.magic_arguments

from drgn.

Or we can do something simpler like:

class Contrib(object):
    @property
    def ps(self):
        print('call execscript(contrib/ps.py)')

    @property
    def vmstat(self):
        print('call execscript(contrib/vmstat.py)')

cmd = Contrib()

where will then use something like this in the interactive session:

>>> cmd.ps
...
>>> cmd.vmstat

what do you think? Note the properties can be created automatically based on the content of contrib folder.

from drgn.

I love the idea of higher-level commands in drgn. However, I don't want to use contrib for this. I really want to keep contrib as a dumping ground with almost no rules, but if we expose it in a first-class manner like this, users will have expectations.

There is also a technical issue: the contrib directory is not installed at all when installing drgn from PyPI. At best, some of the distro packages install it in /usr/share, but in general we can't rely on it being installed.

Instead of using contrib, I'd prefer to have some sort of infrastructure for defining commands as part of the main drgn package and have the usual requirements of kernel version support, unit tests, style, etc. I think IPython/Jupyter integration is a really promising direction for drgn, so I'd lean towards doing this via the IPython interface that you pointed out, but that would take some initial groundwork.

from drgn.

That being said, once this infrastructure exists, it'd be great to "promote" the contrib scripts you've added to "real" commands.

from drgn.

I love the idea of higher-level commands in drgn. However, I don't want to use contrib for this. I really want to keep contrib as a dumping ground with almost no rules, but if we expose it in a first-class manner like this, users will have expectations.

Sure, makes sense.

There is also a technical issue: the contrib directory is not installed at all when installing drgn from PyPI. At best, some of the distro packages install it in /usr/share, but in general we can't rely on it being installed.

Yep!

Instead of using contrib, I'd prefer to have some sort of infrastructure for defining commands as part of the main drgn package and have the usual requirements of kernel version support, unit tests, style, etc.

I like the idea. Note the scripts are tested pretty well, so I don't expect any big fallout and when it comes to style, it's already formated by mypy and black.

I think IPython/Jupyter integration is a really promising direction for drgn, so I'd lean towards doing this via the IPython interface that you pointed out, but that would take some initial groundwork.

Great, I prefer the IPython interface, where one can define arguments for the magic commands. Looking forward to @brenns10's groundwork..

from drgn.

It is not Monday, but better late than never.

Here is a starting point which you can run simply as a drgn script directly (e.g. drgn -c vmcore -s vmlinux my_ipython_runner.py).

There are several weird oddities with IPython - sometimes you can skip using % for the magic, but sometimes it's required. The object formatting is decidedly annoying. Overriding an IPython built-in magic is... bordering on impossible. And the setup steps (which you can see at the bottom of the file) are a bit annoying to say the least. I ended up abandoning the IPython based approach because I realized that most folks I work with that do vmcore debugging, they have some Python experience but not much IPython. Making the transition from crash commands to drgn is already difficult because you need to brush up on Python. Adding in the complexity of IPython and the strange syntax would have added too many roadblocks.

I think I later came up with a crash lookalike repl, with a feature to switch between a normal Python REPL and the crash shell. This would have been nice, but honestly playing a game of catch-up where you reimplement all of the crash commands wasn't something I have time for. My coworkers, if they need crash commands, will use crash. We use drgn for things that are difficult in crash.

$ python -m drgn -c ~/archive/drgn-tools-testing/t1/vmcore -s ~/archive/drgn-tools-testing/t1/vmlinux-5.4.17-2136.304.4.1.el8uek.x86_64.bak my_ipython_runner.py
warning: could not get debugging information for:
kernel modules (could not read depmod: open: /lib/modules/5.4.17-2136.304.4.1.el8uek.x86_64/modules.dep.bin: No such file or directory)

Welcome to the IPython drgn
You can run python code or commands in the same prompt.

In [1]: bt -c 0
#0  native_safe_halt (./arch/x86/include/asm/irqflags.h:61:1)
#1  arch_safe_halt (./arch/x86/include/asm/paravirt.h:144:2)
#2  default_idle (arch/x86/kernel/process.c:579:2)
#3  arch_cpu_idle (arch/x86/kernel/process.c:570:2)
#4  default_idle_call (kernel/sched/idle.c:94:3)
#5  cpuidle_idle_call (kernel/sched/idle.c:154:3)
#6  do_idle (kernel/sched/idle.c:264:4)
#7  cpu_startup_entry (kernel/sched/idle.c:356:3)
#8  rest_init (init/main.c:453:2)
#9  arch_call_rest_init (init/main.c:575:2)
#10 start_kernel (init/main.c:786:2)
#11 x86_64_start_reservations (arch/x86/kernel/head64.c:490:2)
#12 x86_64_start_kernel (arch/x86/kernel/head64.c:471:2)
#13 secondary_startup_64+0xb6/0xb6 (arch/x86/kernel/head_64.S:241)

In [2]: %set 1

In [3]: bt
#0  context_switch (kernel/sched/core.c:3852:2)
#1  __schedule (kernel/sched/core.c:5082:8)
#2  schedule (kernel/sched/core.c:5157:3)
#3  schedule_hrtimeout_range_clock (kernel/time/hrtimer.c:2185:3)
#4  schedule_hrtimeout_range (kernel/time/hrtimer.c:2239:9)
#5  ep_poll (fs/eventpoll.c:1925:8)
#6  do_epoll_wait (fs/eventpoll.c:2333:10)
#7  __do_sys_epoll_wait (fs/eventpoll.c:2343:9)
#8  __se_sys_epoll_wait (fs/eventpoll.c:2340:1)
#9  __x64_sys_epoll_wait (fs/eventpoll.c:2340:1)
#10 do_syscall_64 (arch/x86/entry/common.c:296:14)
#11 entry_SYSCALL_64+0x1a8/0x2aa (arch/x86/entry/entry_64.S:179)

In [4]: prog["jiffies_64"]
Out[4]: (u64)4298890678

In [5]: prog["slab_caches"]
Out[5]:
(struct list_head){
        .next = (struct list_head *)0xffff8b831d972260,
        .prev = (struct list_head *)0xffff8b8007c02060,
}

from drgn.

Just to comment further on the question of scripts and commands. I also like the idea of interactive commands. For interactive debugging, it's a lot easier to do dmesg | less or bt -c 5 than to run open("/tmp/dmesg.txt").write(get_dmesg(prog)); os.system("less /tmp/dmesg.txt"); os.unlink("/tmp/dmesg.txt") or do prog.stack_trace(per_cpu(prog["current_task"], 5)).

On the other hand, what is the correct "command language" for drgn? Should it emulate crash? There's already @pfactum's crush which attempts something close to that. I've only dabbled with it but it seems good, and it has a lot of groundwork already laid; I wouldn't want to start parallel efforts. Then there's sdb, which is another CLI. As a crash user, I didn't want to learn another CLI paradigm for debugging (on top of crash and drgn APIs), so I haven't really used it, but it seems to have a reasonably extensive library of functionality too.

So if we adopt a command set that's similar to one of those (or others I don't know about), then we may be able to share some effort (if the authors are willing), but we'd also alienate the other CLIs built on drgn. If we come up with our own "official" command set, then we'd duplicate a fair amount of effort, and maybe also still alienate all the other maintainers?

I don't know where I'm going with this long comment beyond to say (a) I would love an interactive command system, but (b) it seems outside of the scope of drgn. I like the diversity of options, and I could see an IPython/Jupyter based app being an interesting option as well.

And right now, despite the diversity of drgn-based tools, I find that when I've needed an interactive command system, I've just opened up crash in a tmux pane right next to drgn. Anyway, just my two cents, I know I'm not really providing any clarity or conclusion here.

As a side note, as a library user, I wouldn't want to see an IPython dependency added on drgn. That would make it much harder to install (right now you can just copy a wheel to wherever you want it, and do a pip install). I suppose it would be totally fine as an optional dependency (e.g. drgn[ipython]).

from drgn.

This honestly feels like a good topic for a hypothetical "kernel debugging" track at LPC 2023?

from drgn.

It is not Monday, but better late than never.

Heh ;) Thank you sharing of what you came up to.

Here is a starting point which you can run simply as a drgn script directly (e.g. drgn -c vmcore -s vmlinux my_ipython_runner.py).

First, let me make a small disclaimer. I'm not a crash user and I have hardly any experience with debugging real-world crashes. What I know is there's an established crash tool with its own REPL syntax and a community of folks who are familiar with it. On other hand, the tool is quite legacy, the original author left the project and from what I've seen, the C implementation of all the commands is full of boilerplate code and much more complicated than a Python implementation. That's why I thought it would not be so difficult to mimic the basic crash commands and maybe that can replace crash in the future.

Back to the CLI. I don't have a strong opinion here. Maybe IPython can provide a reasonable replacement for the known crash-like REPR. IPython provides ¹ a pretty nice interface when it comes to post-processing of line based data:

In [53]: ps = !ps -ax
In [53]: len(ps)
Out[53]: 498
In [56]: ps.grep('ssh')
Out[56]: 
[' 1326 ?        Ss     0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups',
 ' 3645 ?        S      0:00 /usr/bin/ssh-agent -D -a /run/user/1000/keyring/.ssh',
 '13806 ?        Ss     0:00 sshd: marxin [priv]',
 '13812 ?        S      0:00 sshd: marxin@pts/0',
 '26192 ?        Ss     0:00 sshd: marxin [priv]',
 '26198 ?        S      0:00 sshd: marxin@pts/4',
 '26691 ?        Ss     0:00 sshd: marxin [priv]',
 '26697 ?        S      0:00 sshd: marxin@pts/1',
 '27063 ?        Ss     0:00 sshd: marxin [priv]',
 '27070 ?        S      0:00 sshd: marxin@pts/2',
 '27655 ?        Ss     0:00 sshd: marxin [priv]',
 '27661 ?        S      0:00 sshd: marxin@pts/3']
In [61]: ps.grep('ssh').fields(3, 0).sort()
Out[61]: 
['0:00 1326',
 '0:00 13806',
 '0:00 13812',
 '0:00 26192',
 '0:00 26198',
 '0:00 26691',
 '0:00 26697',
 '0:00 27063',
 '0:00 27070',
 '0:00 27655',
 '0:00 27661',
 '0:00 3645']
In [65]: print(ps.grep('ssh').fields(3, 0).sort().n)
0:00 1326
0:00 13806
0:00 13812
0:00 26192
0:00 26198
0:00 26691
0:00 26697
0:00 27063
0:00 27070
0:00 27655
0:00 27661
0:00 3645

There are several weird oddities with IPython - sometimes you can skip using % for the magic, but sometimes it's required. The object formatting is decidedly annoying. Overriding an IPython built-in magic is... bordering on impossible. And the setup steps (which you can see at the bottom of the file) are a bit annoying to say the least. I ended up abandoning the IPython based approach because I realized that most folks I work with that do vmcore debugging, they have some Python experience but not much IPython.

Well, I see the difference in between Python REPL and IPython quite small and yes, people will need to get more familiar with %-syntax.

Making the transition from crash commands to drgn is already difficult because you need to brush up on Python. Adding in the complexity of IPython and the strange syntax would have added too many roadblocks.

The question is if we should stick to the crash commands? A solution may be a post-processing of drgn-commands with what I showed in the aforementioned demo combined with python snippets (even multi-line snippets can be easily written in IPython)?

I think I later came up with a crash lookalike repl, with a feature to switch between a normal Python REPL and the crash shell. This would have been nice, but honestly playing a game of catch-up where you reimplement all of the crash commands wasn't something I have time for. My coworkers, if they need crash commands, will use crash. We use drgn for things that are difficult in crash.

I've got your point, drng is great when it comes to more complex scripts. With a reasonable effort, it may also become a drop-in replacement for crash in the future.

from drgn.

On the other hand, what is the correct "command language" for drgn?

That's a crucial question I guess and I don't have a good answer (see my previous comment).

Should it emulate crash? There's already @pfactum's crush which attempts something close to that. I've only dabbled with it but it seems good, and it has a lot of groundwork already laid; I wouldn't want to start parallel efforts. Then there's sdb, which is another CLI. As a crash user, I didn't want to learn another CLI paradigm for debugging (on top of crash and drgn APIs), so I haven't really used it, but it seems to have a reasonably extensive library of functionality too.

Oh, thank you for the pointers to the interesting projects. I would separate the problem into 2 parts:

1. command foundation - a toolkit of commands that provide what crash does (ps, dev, ...); basics are provided in contrib folder and the linked projects definitely go even further; it would be nice to unify the effort and have a unified commands
2. CLI / command language - that's something that would leverage the existing commands and would provide a glue in between these commands and yes I see there are multiple options that can be chosen

So from a random observer perspective, I see drgn-based tools as very promising candidates for crash replacement in the future.

from drgn.

I like the IPython interface and the SList type, it's pretty neat. Please see my WIP patch in #283. There's a small demo:

drgn --core vmcore ~/Programming/drgn/ipython_init.py
In [2]: %cmd
Available commands:
  cgroup: List the paths of all descendants of a cgroup v2
  fs_inodes: List the paths of all inodes cached in a given filesystem
  kcore_list: Dump the list of memory regions exposed by /proc/kcore.
  kernel_sys: Display system information and configuration data.
  lsmod: An implementation of lsmod(8) using drgn
  mount: A simplified implementation of mount(1) using drgn
  ps: A simplified implementation of ps(1) using drgn
  tcp_sock: List all TCP sockets and their cgroup v2 paths
  vmmap: Print memory map of a given task.
  vmstat: Dump /proc/vmstat statistics.
In [4]: ps = %cmds ps

In [5]: [(x[-1], prog.stack_trace(int(x[0]))) for x in ps.grep('sh').fields()]
Out[5]: 
[('[zswap-shrink]',
  #0  context_switch (../kernel/sched/core.c:4851)
#1  __schedule (../kernel/sched/core.c:6167)
#2  schedule (../kernel/sched/core.c:6243)
#3  rescuer_thread (../kernel/workqueue.c:2626)
#4  kthread (../kernel/kthread.c:327)
#5  ret_from_fork+0x22/0x2d (../arch/x86/entry/entry_64.S:298)),
 ('sshd',
  #0  context_switch (../kernel/sched/core.c:4851)
#1  __schedule (../kernel/sched/core.c:6167)
#2  schedule (../kernel/sched/core.c:6243)
#3  schedule_hrtimeout_range_clock (../kernel/time/hrtimer.c:2296)
#4  poll_schedule_timeout (../fs/select.c:244)
#5  do_select (../fs/select.c:607)
#6  core_sys_select (../fs/select.c:681)
#7  kern_select (../fs/select.c:722)
#8  __do_sys_select (../fs/select.c:729)
#9  __se_sys_select (../fs/select.c:726)
#10 __x64_sys_select (../fs/select.c:726)
#11 do_syscall_x64 (../arch/x86/entry/common.c:50)
#12 do_syscall_64 (../arch/x86/entry/common.c:80)
#13 entry_SYSCALL_64+0x99/0x194 (../arch/x86/entry/entry_64.S:118)),
 ('sshd',
  #0  context_switch (../kernel/sched/core.c:4851)
#1  __schedule (../kernel/sched/core.c:6167)
#2  schedule (../kernel/sched/core.c:6243)
#3  schedule_hrtimeout_range_clock (../kernel/time/hrtimer.c:2296)
#4  poll_schedule_timeout (../fs/select.c:244)
#5  do_select (../fs/select.c:607)
#6  core_sys_select (../fs/select.c:681)
#7  kern_select (../fs/select.c:722)
#8  __do_sys_select (../fs/select.c:729)
#9  __se_sys_select (../fs/select.c:726)
#10 __x64_sys_select (../fs/select.c:726)
#11 do_syscall_x64 (../arch/x86/entry/common.c:50)
#12 do_syscall_64 (../arch/x86/entry/common.c:80)
#13 entry_SYSCALL_64+0x99/0x194 (../arch/x86/entry/entry_64.S:118)),
 ('bash',
  #0  crash_setup_regs (../arch/x86/include/asm/kexec.h:95)
#1  __crash_kexec (../kernel/kexec_core.c:954)
#2  panic (../kernel/panic.c:251)
#3  sysrq_handle_crash (../drivers/tty/sysrq.c:155)
#4  __handle_sysrq (../drivers/tty/sysrq.c:604)
#5  write_sysrq_trigger (../drivers/tty/sysrq.c:1163)
#6  pde_write (../fs/proc/inode.c:335)
#7  proc_reg_write (../fs/proc/inode.c:347)
#8  vfs_write (../fs/read_write.c:603)
#9  ksys_write (../fs/read_write.c:658)
#10 do_syscall_x64 (../arch/x86/entry/common.c:50)
#11 do_syscall_64 (../arch/x86/entry/common.c:80)
#12 entry_SYSCALL_64+0x99/0x194 (../arch/x86/entry/entry_64.S:118))]

In [10]: mods = %cmds lsmod

In [15]: mods.grep('ip').grep(lambda x: int(x.split()[2]) > 5).fields(0)
Out[15]: ['nf_tables', 'x_tables', 'scsi_mod']

from drgn.

It is good have certain well supported drgn scripts/cmd and infrastructure to run them in drgn interactive cli environment to extract basic/advance information from dump.

It would be great if drgn has dedicated directory to keep well supported drgn scripts and import them in drgn interactive cli by default.

In addition to generic infrastructure to run drgn script #283, we should also explore the format/standard for drgn scripts. Let me know if can help here.

• Sourabh

from drgn.