memo/2021/01/02/linux-kernel-build-debug about josehu07.github.io HOT 12 OPEN

Thanks for the blogpost 👍

Minor note about the GDB script: It's at <kernelsourcedir>/scripts/gdb/vmlinux-gdb.py

from josehu07.github.io.

Hi @codingjin,

I want to ask, did you ever try to make the apt update/install available in your vm? I want to run some apps. Or do you have some suggestions on it?

I don't think anything as complex as a package manager is supported in the minimal root fs by buildroot. If your app has external dependency on dynamic libraries, it won't run properly in the minimal root fs anyway -- it doesn't even have a libc.

There're basically 2 options:

1. Install the customly built kernel into a complete ubuntu distribution, e.g., to your host machine (note the risk of kernel crashing) or to an already complete ubuntu vm (so that way you can probably still do debugging), and boot with that kernel. Follow the installation and booting sections for instructions.
2. If you want to stick to the minimal root fs, and your host environment is the same architecture as the qemu vm, compile and statically link the executable you are tring to run, so that it is self-containing and has no external dynamic library dependencies. That way, you can scp the executable in and run it directly. The last section has instructions for this.

from josehu07.github.io.

Hi @MooMooHorse,

vmlinux is the uncompressed ELF-format object file which contains debugging symbols, while bzImage is a compressed binary file that QEMU understands and knows how to load. The start_kernel symbol is the entry point to vmlinux; so if you try to break at early bootloading points before that symbol, you won't be able to do that with the method described in this post, because there are no corresponding debugging symbols for things before that point.

There may be two options if you really need to break before start_kernel:

1. You could try setting GDB hardware breakpoints with hbreak -- this might give you breakable points at some instructions before the start_kernel symbol is hit. (I used hbreak in this post as well.)
2. It is said that with QEMU >= 4.0, it is able to load the uncompressed ELF vmlinux given that the PVH headers are included during kernel build. Ref: https://stefano-garzarella.github.io/posts/2019-08-23-qemu-linux-kernel-pvh/. I haven't tried this so am not sure if that post describes what I'm thinking. You could give it try.

from josehu07.github.io.

Hi, Jose. Really thank you for your sharing. I want to ask, did you ever try to make the apt update/install available in your vm? I want to run some apps. Or do you have some suggestions on it? Thank you

from josehu07.github.io.

Got it. Thank you, Jose.

from josehu07.github.io.

Hi, I don't know why, but I got the issue like https://stackoverflow.com/questions/36621505/debugging-linux-kernel-using-gdb-in-qemu-unable-to-hit-function-or-given-address. I think it has a point in that post. Basically, you link bzImage to QEMU but use vmlinux to debug it. The address just doesn't match. Am I understanding it incorrectly?

from josehu07.github.io.

Thanks for your reply. But my problem is I set the breakpoints even at console_init and I also used hb start_kernel, it just doesn't stop at those places. One observations is, bt stack-print prints out address different from addresses set by breakpoints. My qemu flags are qemu-system-x86_64 -kernel arch/x86/boot/bzImage -append "console=ttyS0 nokaslr" -initrd ramdisk.img -m 8G -append nokaslr -append "memmap=2G!6G" -smp 3 -s -S and I checked and confirmed it's not multi-core issue by checking https://qemu-project.gitlab.io/qemu/system/gdb.html. And I checked that those functions I set breakpoints with can be entered by the printk messages on the screen. Do you have any ideas about what's going on? To summarize, the breakpoint address (symbol file items) have a certain deviation to the actual addresses of the instructions.

from josehu07.github.io.

I run command:
sudo qemu-system-x86_64 -kernel /root/kernel_tmp/linux-4.19.1/arch/x86_64/boot/bzImage -nographic -drive format=raw,file=/root/kernel_tmp/buildroot/output/images/rootfs.ext4,if=virtio -append "root=/dev/vda console=ttyS0 nokaslr other-paras-here-if-needed" -m 4G -smp $(nproc) -net nic,model=virtio -net user,hostfwd=tcp::10022-:22 -s -S

later I run :
1、gdb vmlinux
2、b start_kernel
3、c

the qemu window is no responding after this log:
[ 1.154845] md: If you don't use raid, use raid=noautodetect
[ 1.156406] md: Autodetecting RAID arrays.
[ 1.156531] md: autorun ...
[ 1.156624] md: ... autorun DONE.
[ 1.163645] EXT4-fs (vda): INFO: recovery required on readonly filesystem
[ 1.164161] EXT4-fs (vda): write access will be enabled during recovery
[ 1.188326] EXT4-fs (vda): recovery complete
[ 1.190738] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
[ 1.191120] VFS: Mounted root (ext4 filesystem) readonly on device 253:0.
[ 1.192548] devtmpfs: mounted
[ 1.233169] Freeing unused kernel image memory: 2588K
[ 1.238499] Write protecting the kernel read-only data: 22528k
[ 1.243825] Freeing unused kernel image memory: 2008K
[ 1.244942] Freeing unused kernel image memory: 1996K
[ 1.276616] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[ 1.277239] Run /sbin/init as init process
[ 1.392350] EXT4-fs (vda): re-mounted. Opts: (null)
Seeding 2048 bits without crediting
[ 1.595103] random: seedrng: uninitialized urandom read (256 bytes read)
Saving 2048 bits of non-creditable seed for next boot
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Starting network: OK

from josehu07.github.io.

I run command:
sudo qemu-system-x86_64 -kernel /root/kernel_tmp/linux-4.19.1/arch/x86_64/boot/bzImage -nographic -drive format=raw,file=/root/kernel_tmp/buildroot/output/images/rootfs.ext4,if=virtio -append "root=/dev/vda console=ttyS0 nokaslr other-paras-here-if-needed" -m 4G -smp $(nproc) -net nic,model=virtio -net user,hostfwd=tcp::10022-:22 -s -S

It seems that you included the argument string "other-paras-here-if-needed" within the -append argument value. That's just a placeholder for other kernel boot parameters (I should have made this clearer in the post =)

from josehu07.github.io.

from josehu07.github.io.

Hey Guanzhou,

Thank you for the tutorial. One issue when I followed your process during booting though(log pasted below). Have you seen something similar like this?

I made sure the kernel config is correctly updated and I have manually mounted the rootfs to ensure it is valid.

[    0.851130] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    0.853332] CPU: 7 PID: 1 Comm: swapper/0 Not tainted 6.5.3 #4 6aa75bb7bc7fc595e0925e6788dabd26a94322f4
[    0.856206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Arch Linux 1.16.2-2-2 04/01/2014
[    0.858120] Call Trace:
[    0.858630]  <TASK>
[    0.859080]  dump_stack_lvl+0x47/0x60
[    0.859799]  panic+0x180/0x330
[    0.860379]  mount_root_generic+0x1ac/0x340
[    0.861166]  prepare_namespace+0x63/0x250
[    0.861920]  kernel_init_freeable+0x2c6/0x320
[    0.862780]  ? __pfx_kernel_init+0x10/0x10
[    0.863606]  kernel_init+0x1a/0x1c0
[    0.864320]  ret_from_fork+0x31/0x50
[    0.865072]  ? __pfx_kernel_init+0x10/0x10
[    0.865834]  ret_from_fork_asm+0x1b/0x30
[    0.866567]  </TASK>
[    0.867797] Kernel Offset: disabled
[    0.868538] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---

from josehu07.github.io.

from josehu07.github.io.