git clone https://github.com/************/linux-kernel-module-cheat
cd linux-kernel-module-cheat
./configure && ./build && ./run

insmod /hello.ko
insmod /hello2.ko
rmmod hello
rmmod hello2

hello init
hello2 init
hello cleanup
hello2 cleanup

Many of the modules have userland test scripts / executables with the same name as the module, e.g. form inside the guest:

The sources of those tests will further clarify what the corresponding kernel modules does. To find them on the host, do a quick:

After making changes to a package, you must explicitly request it to be rebuilt.

For example, you you modify the kernel modules, you must rebuild with:

which is just an alias for:

where kernel_module is the name of out Buildroot package that contains the kernel modules.

Other important targets are:

which rebuild the Linux kernel, and QEMU and gem5 respectively. They are essentially aliases for:

However, some of our aliases such as -l also have some magic convenience properties. So generally just use the aliases instead.

We don’t rebuild by default because, even with make incremental rebuilds, the timestamp check takes a few annoying seconds.

Not all packages have an alias, when they don’t, just use the form:

It gets annoying to retype -a aarch64 for every single command, or to remember ./build -B setups.

So simplify that, do:

and then edit the cli.gitignore file to your needs.

That file is used to pass extra command line arguments to most of our utilities.

Of course, you could get by with the shell history, or your own aliases, but we’ve felt that it was worth introducing a common mechanism for that.

You did something crazy, and nothing seems to work anymore?

All builds are stored under buildroot/,

The most coarse thing you can do is:

To only nuke one architecture, do:

Only nuke one one package:

This is sometimes necessary when changing the version of the submodules, and then builds fail. We should try to understand why and report bugs.

The root filesystem is persistent across:

then:

The command:

also saves the disk.

This is particularly useful to re-run shell commands from the history of a previous session with Ctrl + R.

However, when you do:

the disk image gets overwritten by a fresh filesystem and you lose all changes.

Remember that if you forcibly turn QEMU off without sync or poweroff from inside the VM, e.g. by closing the QEMU window, disk changes may not be saved.

When booting from initrd however without a disk, persistency is lost.

We use printk a lot, and it shows on the QEMU terminal by default. If that annoys you (e.g. you want to see stdout separately), do:

See also: https://superuser.com/questions/351387/how-to-stop-kernel-messages-from-flooding-my-console

When in graphical mode, you can scroll up a bit on the default TTY with:

but I never managed to increase that buffer:

• https://askubuntu.com/questions/709697/how-to-increase-scrollback-lines-in-ubuntu14-04-2-server-edition

• https://unix.stackexchange.com/questions/346018/how-to-increase-the-scrollback-buffer-size-for-tty

The superior alternative is to use text mode or a telnet connection.

By default, we show the serial console directly on the current terminal, without opening a QEMU window.

To enable graphic mode, use:

Text mode is the default due to the following considerable advantages:

• copy and paste commands and stdout output to / from host

• get full panic traces when you start making the kernel crash :-) See also: https://unix.stackexchange.com/questions/208260/how-to-scroll-up-after-a-kernel-panic

• have a large scroll buffer, and be able to search it, e.g. by using tmux on host

• one less window floating around to think about in addition to your shell :-)

• graphics mode has only been properly tested on x86_64.

Text mode has the following limitations over graphics mode:

• you can’t see graphics such as those produced by X11

• very early kernel messages such as early console in extract_kernel only show on the GUI, since at such early stages, not even the serial has been setup.

Both good and bad:

• Ctrl + C kills the host emulator instead of sending SIGINT to the guest process.

  On one hand, this provides an easy way to quit QEMU.

  On the other, we are unable to easily kill the foreground process, which is specially problematic when it is something like an infinite loop. and not sent to guest processes.

  TODO: understand why and how to change that. See:

  • https://unix.stackexchange.com/questions/167165/how-to-pass-ctrl-c-in-qemu

  • cloudius-systems/osv#49

  It is also hard to enter the monitor for the same reason:

  • http://stackoverflow.com/questions/14165158/how-to-switch-to-qemu-monitor-console-when-running-with-curses

  • https://superuser.com/questions/488263/how-to-switch-to-the-qemu-control-panel-with-nographics

  Our workaround is:

  ./qemumonitor

  I think the problem was reversed in older QEMU versions: https://superuser.com/questions/1087859/how-to-quit-the-qemu-monitor-when-not-using-a-gui/1211516#1211516

  sendkey sendkey ctrl-c does not work on the text terminal either.

  This is however fortunate when running QEMU with GDB, as the Ctrl + C reaches GDB and breaks.

• Not everything generates interrupts, only the final enter.

  This makes things a bit more reproducible, since the microsecond in which you pressed a key does not matter.

  But on the other hand maybe you are interested in observing the interrupts generated by key presses.

When debugging a module, it becomes tedious to wait for build and re-type:

every time.

To automate that, use the methods described at: init

Bootloaders can pass a string as input to the Linux kernel when it is booting to control its behaviour, much like the execve system call does to userland processes.

This allows us to control the behaviour of the kernel without rebuilding anything.

With QEMU, QEMU itself acts as the bootloader, and provides the -append option and we expose it through ./run -e, e.g.:

Then inside the host, you can check which options were given with:

They are also printed at the beginning of the boot message:

See also:

• https://unix.stackexchange.com/questions/48601/how-to-display-the-linux-kernel-command-line-parameters-given-for-the-current-bo

• https://askubuntu.com/questions/32654/how-do-i-find-the-boot-parameters-used-by-the-running-kernel

The arguments are documented in the kernel documentation: https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html

When dealing with real boards, extra command line options are provided on some magic bootloader configuration file, e.g.:

• GRUB configuration files: https://askubuntu.com/questions/19486/how-do-i-add-a-kernel-boot-parameter

• Raspberry pi /boot/cmdline.txt on a magic partition: https://raspberrypi.stackexchange.com/questions/14839/how-to-change-the-kernel-commandline-for-archlinuxarm-on-raspberry-pi-effectly

Double quotes can be used to escape spaces as in opt="a b", but double quotes themselves cannot be escaped, e.g. opt"a\"b"

This even lead us to use base64 encoding with -E!

When asking for help on upstream repositories outside of this repository, you will need to provide the commands that you are running in detail without referencing our scripts.

For example, QEMU developers will only want to see the final QEMU command that you are running.

We make that easy by building commands as strings, and then echoing them before evaling.

So for example when you run:

Stdout shows a line with the full command of type:

This line is also saved to a file for convenience:

If you are feeling fancy, you can also insert modules with:

This method also deals with module dependencies, which we almost don’t use to make examples simpler:

• https://askubuntu.com/questions/20070/whats-the-difference-between-insmod-and-modprobe

• https://stackoverflow.com/questions/22891705/whats-the-difference-between-insmod-and-modprobe

Removal also removes required modules that have zero usage count:

but it can’t know if you actually insmodded them separately or not:

so it is a bit risky.

modprobe searches for modules under:

Kernel modules built from the Linux mainline tree with CONFIG_SOME_MOD=m, are automatically available with modprobe, e.g.:

https://stackoverflow.com/questions/5947286/how-to-load-linux-kernel-modules-from-c-code

If you are feeling raw, you can insert and remove modules with our own minimal module inserter and remover!

which teaches you how it is done from C code.

-d makes QEMU wait for a GDB connection, otherwise we could accidentally go past the point we want to break at:

Say you want to break at start_kernel. So on another shell:

or at a given line:

Now QEMU will stop there, and you can use the normal GDB commands:

See also:

• http://stackoverflow.com/questions/11408041/how-to-debug-the-linux-kernel-with-gdb-and-qemu/33203642#33203642

• http://stackoverflow.com/questions/4943857/linux-kernel-live-debugging-how-its-done-and-what-tools-are-used/42316607#42316607

O=0 is an impossible dream, O=2 being the default: https://stackoverflow.com/questions/29151235/how-to-de-optimize-the-linux-kernel-to-and-compile-it-with-o0 So get ready for some weird jumps, and <value optimized out> fun. Why, Linux, why.

Let’s observe the kernel as it reacts to some userland actions.

Start QEMU with just:

and after boot inside a shell run:

which counts to infinity to stdout. Then in another shell, run:

and then hit:

And you now control the counting on the first shell from GDB!

When you hit Ctrl + C, if we happen to be inside kernel code at that point, which is very likely if there are no heavy background tasks waiting, and we are just waiting on a sleep type system call of the command prompt, we can already see the source for the random place inside the kernel where we stopped.

tmux just makes things even more fun by allowing us to see both terminals at once without dragging windows around! https://unix.stackexchange.com/questions/152738/how-to-split-a-new-window-and-run-a-command-in-this-new-window-using-tmux/432111#432111

Loadable kernel modules are a bit trickier since the kernel can place them at different memory locations depending on load order.

So we cannot set the breakpoints before insmod.

However, the Linux kernel GDB scripts offer the lx-symbols command, which takes care of that beautifully for us.

Shell 1:

Wait for the boot to end and run:

This prints a message to dmesg every second.

Shell 2:

In GDB, hit Ctrl + C, and note how it says:

That’s lx-symbols working! Now simply:

and we now control the callback from GDB!

Just don’t forget to remove your breakpoints after rmmod, or they will point to stale memory locations.

TODO: why does break work_func for insmod kthread.ko not break the first time I insmod, but breaks the second time?

See also: http://stackoverflow.com/questions/28607538/how-to-debug-linux-kernel-modules-with-qemu/44095831#44095831

TODO: why can’t we break at early startup stuff such as:

Maybe it is because they are being copied around at specific locations instead of being run directly from inside the main image, which is where the debug information points to?

See also: https://stackoverflow.com/questions/2589845/what-are-the-first-operations-that-the-linux-kernel-executes-on-boot

QEMU’s -gdb GDB breakpoints are set on virtual addresses, so you can in theory debug userland processes as well.

• https://stackoverflow.com/questions/26271901/is-it-possible-to-use-gdb-and-qemu-to-debug-linux-user-space-programs-and-kernel

• https://stackoverflow.com/questions/16273614/debug-init-on-qemu-using-gdb

You will generally want to use gdbserver for this as it is more reliable, but this method can overcome the following limitations of gdbserver:

• the emulator does not support host to guest networking. This seems to be the case for gem5: gem5 host to guest networking

• cannot see the start of the init process easily

• gdbserver alters the working of the kernel, and makes your run less representative

Known limitations of direct userland debugging:

• the kernel might switch context to another process or to the kernel itself e.g. on a system call, and then TODO confirm the PIC would go to weird places and source code would be missing.

• TODO step into shared libraries. If I attempt to load them explicitly:

  (gdb) sharedlibrary ../../staging/lib/libc.so.0
  No loaded shared libraries match the pattern `../../staging/lib/libc.so.0'.

  since GDB does not know that libc is loaded.

GDB can call functions as explained at: https://stackoverflow.com/questions/1354731/how-to-evaluate-functions-in-gdb

However this is failing for us:

• some symbols are not visible to call even though b sees them

• for those that are, call fails with an E14 error

E.g.: if we break on sys_write on /count.sh:

even though fdget_pos is the first thing sys_write does:

See also: ************/linux-kernel-module-cheat#19

./run -k
./rungdb -k

c

/count.sh &
/kgdb.sh

b sys_write
c
c
c
c

In QEMU:

In GDB:

and you now control the count.

TODO: if I -ex lx-symbols to the gdb command, just like done for QEMU -gdb, the kernel oops. How to automate this step?

If you modify runqemu to use:

instead of kgdboc=ttyS0,115200, you enter a different debugging mode called KDB.

Usage: in QEMU:

Boot finishes, then:

And you are back in KDB. Now you can:

And you will break whenever sys_write is hit.

The other KDB commands allow you to instruction steps, view memory, registers and some higher level kernel runtime data.

But TODO I don’t think you can see where you are in the kernel source code and line step as from GDB, since the kernel source is not available on guest (ah, if only debugging information supported full source).

/gdbserver.sh /myinsmod.out /hello.ko

./rungdbserver kernel_module-1.0/user/myinsmod.out

find out/x86_64/buildroot/build -name myinsmod.out

As usual, different archs work with:

BusyBox executables are all symlinks, so if you do on guest:

on host you need:

Our setup gives you the rare opportunity to step debug libc and other system libraries e.g. with:

Or simply by stepping into calls:

This is made possible by the GDB command:

which automatically finds unstripped shared libraries on the host for us.

See also: https://stackoverflow.com/questions/8611194/debugging-shared-libraries-with-gdbserver/45252113#45252113

./build -a arm
./run -a arm

./run -a arm -d
# On another terminal.
./rungdb -a arm

TODOs:

• only managed to run in the terminal interface (but weirdly a blank QEMU window is still opened)

• GDB not connecting to KGDB. Possibly linked to -serial stdio. See also: https://stackoverflow.com/questions/14155577/how-to-use-kgdb-on-arm

As usual, we use Buildroot’s recommended QEMU setup QEMU aarch64 setup:

• https://github.com/buildroot/buildroot/blob/2017.08/board/qemu/aarch64-virt/readme.txt

• https://github.com/buildroot/buildroot/blob/2017.08/configs/qemu_aarch64_virt_defconfig

This makes aarch64 a bit different from arm:

• uses -M virt. https://wiki.qemu.org/Documentation/Platforms/ARM explains:

  Most of the machines QEMU supports have annoying limitations (small amount of RAM, no PCI or other hard disk, etc) which are there because that’s what the real hardware is like. If you don’t care about reproducing the idiosyncrasies of a particular bit of hardware, the best choice today is the "virt" machine.

  -M virt has some limitations, e.g. I could not pass -drive if=scsi as for arm, and so Snapshot fails.

Keep in mind that MIPS has the worst support compared to our other architectures due to the smaller community. Patches welcome as usual.

TODOs:

• networking is not working. See also:

  • https://stackoverflow.com/questions/21496449/networking-is-not-working-on-qemu-guest-malta-mips

  • https://unix.stackexchange.com/questions/208266/setting-up-qemu-and-mipsel-networking-trouble

  • https://unix.stackexchange.com/questions/354127/qemu-mips-and-debian

• GDB step debug does not work properly, does not find start_kernel

To have more control over the system, you can replace BusyBox’s init with your own.

The following method replaces init and evals a command from the Kernel command line parameters:

It is basically a shortcut for:

although -E is smarter:

• allows quoting and newlines by using base64 encoding, see: Kernel command line parameters escaping

• automatically chooses between init= and rcinit= for you, see: Path to init

so you should almost always use it, unless you are really counting each cycle ;-)

If the script is large, you can add it to a gitignored file and pass that to -E as in:

or add it to a file to the root filesystem guest and rebuild:

Remember that if your init returns, the kernel will panic, there are just two non-panic possibilities:

• run forever in a loop or long sleep

• poweroff the machine

If you rely on something that BusyBox' init set up for you like networking, you could do:

The lkmc_eval option gets evaled by our default S98 startup script if present.

Alternatively, add them to a new init.d entry to run at the end o the BusyBox init:

and they will be run automatically before the login prompt.

Scripts under /etc/init.d are run by /etc/init.d/rcS, which gets called by the line ::sysinit:/etc/init.d/rcS in /etc/inittab.

The init is selected at:

• initrd or initramfs system: /init, a custom one can be set with the rdinit= kernel command line parameter

• otherwise: default is /sbin/init, followed by some other paths, a custom one can be set with init=

More details: https://unix.stackexchange.com/questions/30414/what-can-make-passing-init-path-to-program-to-the-kernel-not-start-program-as-i/430614#430614

Documented at https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html:

The kernel parses parameters from the kernel command line up to "-"; if it doesn’t recognize a parameter and it doesn’t contain a '.', the parameter gets passed to init: parameters with '=' go into init’s environment, others are passed as command line arguments to init. Everything after "-" is passed as an argument to init.

And you can try it out with:

Also note how the annoying dash - also gets passed as a parameter to init, which makes it impossible to use this method for most executables.

Finally, the docs are lying, arguments with dots that come after - are still treated specially (of the form subsystem.somevalue) and disappear:

We disable networking by default because it starts an userland process, and we want to keep the number of userland processes to a minimum to make the system more understandable.

Enable:

Disable:

Test:

BusyBox' ping does not work with hostnames even when networking is working fine:

TODO why: https://unix.stackexchange.com/questions/124283/busybox-ping-ip-works-but-hostname-nslookup-fails-with-bad-address

To enable networking by default, use the methods documented at Automatic startup commands

./run -K

./build -b br2_x11
./run -x

startx

xcalc
xeyes

TODO 9076c1d9bcc13b6efdb8ef502274f846d8d4e6a1 I’m 100% sure that it was working before, but I didn’t run it forever, and it stopped working at some point. Needs bisection, on whatever commit last touched x11 stuff.

• https://askubuntu.com/questions/730891/how-can-i-get-a-mouse-cursor-in-qemu

• https://stackoverflow.com/questions/19665412/mouse-and-keyboard-not-working-in-qemu-emulator

-show-cursor did not help, I just get to see the host cursor, but the guest cursor still does not move.

Doing:

shows that interrupts do happen when mouse and keyboard presses are done, so I expect that it is some wrong either with:

• QEMU. Same behaviour if I try the host’s QEMU 2.10.1 however.

• X11 configuration. We do have BR2_PACKAGE_XDRIVER_XF86_INPUT_MOUSE=y.

/var/log/Xorg.0.log contains the following interesting lines:

The file /dev/inputs/mice does not exist.

On ARM, startx hangs at a message:

and nothing shows on the screen, and:

says:

A friend told me this but I haven’t tried it yet:

• xf86-video-modesetting is likely the missing ingredient, but it does not seem possible to activate it from Buildroot currently without patching things.

• xf86-video-fbdev should work as well, but we need to make sure fbdev is enabled, and maybe add some line to the Xorg.conf

Also if I do:

the screen fills up with random colors, so I think it can truly work.

Haven’t tried it, doubt it will work out of the box! :-)

Maybe: https://stackoverflow.com/questions/47857023/booting-a-graphical-mips-qemu-machine

./build -i
./run -i

cat ./run.log

date >f
poweroff
cat f
# can't open 'f': No such file or directory

end Kernel panic - not syncing: Out of memory and no killable processes...

./run -im 256M

Most modern desktop distributions have an initrd in their root disk to do early setup.

The rationale for this is described at: https://en.wikipedia.org/wiki/Initial_ramdisk

One obvious use case is having an encrypted root filesystem: you keep the initrd in an unencrypted partition, and then setup decryption from there.

I think GRUB then knows read common disk formats, and then loads that initrd to memory with a /boot/grub/grub.cfg directive of type:

Related: https://stackoverflow.com/questions/6405083/initrd-and-booting-the-linux-kernel

initramfs is just like initrd, but you also glue the image directly to the kernel image itself.

So the only argument that QEMU needs is the -kernel, no -drive not even -initrd! Pretty cool.

Try it out with:

The -l (ell) should only be used the first time you move to / from a different root filesystem method (ext2 or cpio) to initramfs to overcome: https://stackoverflow.com/questions/49260466/why-when-i-change-br2-linux-kernel-custom-config-file-and-run-make-linux-reconfi

It is interesting to see how this increases the size of the kernel image if you do a:

before and after using initramfs, since the .cpio is now glued to the kernel image.

In the background, it uses BR2_TARGET_ROOTFS_INITRAMFS, and this makes the kernel config option CONFIG_INITRAMFS_SOURCE point to the CPIO that will be embedded in the kernel image.

http://nairobi-embedded.org/initramfs_tutorial.html shows a full manual setup.

TODO we were not able to get it working yet: https://stackoverflow.com/questions/49261801/how-to-boot-the-linux-kernel-with-initrd-or-initramfs-with-gem5

We try to use the latest possible kernel major release version.

In QEMU:

or in the source:

During update all you kernel modules may break since the kernel API is not stable.

They are usually trivial breaks of things moving around headers or to sub-structs.

The userland, however, should simply not break, as Linus enforces strict backwards compatibility of userland interfaces.

This backwards compatibility is just awesome, it makes getting and running the latest master painless.

This also makes this repo the perfect setup to develop the Linux kernel.

The kernel is not forward compatible, however, so downgrading the Linux kernel requires downgrading the userland too to the latest Buildroot branch that supports it.

The default Linux kernel version is bumped in Buildroot with commit messages of type:

So you can try:

Those commits change BR2_LINUX_KERNEL_LATEST_VERSION in /linux/Config.in.

You should then look up if there is a branch that supports that kernel. Staying on branches is a good idea as they will get backports, in particular ones that fix the build as newer host versions come out.

You can also try those on the Ctrl + Alt + F3 of your Ubuntu host, but it is much more fun inside a VM!

Must be run in graphical mode.

Stop the cursor from blinking:

Rotate the console 90 degrees!

Requires CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y.

Documented under: fb/.

TODO: font and keymap. Mentioned at: https://cmcenroe.me/2017/05/05/linux-console.html and I think can be done with Busybox loadkmap and loadfont, we just have to understand their formats, related:

• https://unix.stackexchange.com/questions/177024/remap-keyboard-on-the-linux-console

• https://superuser.com/questions/194202/remapping-keys-system-wide-in-linux-not-just-in-x

Trace a single function:

Sample output:

Trace all possible functions, and draw a call graph:

Sample output:

TODO: what do + and ! mean?

Each enable under the events/ tree enables a certain set of functions, the higher the enable more functions are enabled.

• https://www.quora.com/How-many-instructions-does-a-typical-Linux-kernel-boot-take

• https://github.com/************/chat/issues/31

• https://rwmj.wordpress.com/2016/03/17/tracing-qemu-guest-execution/

• qemu/docs/tracing.txt and qemu/docs/replay.txt

• https://stackoverflow.com/questions/39149446/how-to-use-qemus-simple-trace-backend/46497873#46497873

Results (boot not excluded):

QEMU:

sample output:

gem5:

Notes:

• 0x1000000 is the address where QEMU puts the Linux kernel at with -kernel in x86.

  It can be found from:

  readelf -e out/x86_64/buildroot/build/linux-*/vmlinux | grep Entry

  TODO confirm further. If I try to break there with:

  ./rungdb *0x1000000

  but I have no corresponding source line. Also note that this line is not actually the first line, since the kernel messages such as early console in extract_kernel have already shown on screen at that point. This does not break at all:

  ./rungdb extract_kernel

  It only appears once on every log I’ve seen so far, checked with grep 0x1000000 trace.txt

  Then when we count the instructions that run before the kernel entry point, there is only about 100k instructions, which is insignificant compared to the kernel boot itself.

  TODO -a arm and -a aarch64 does not count firmware instructions properly because the entry point address of the ELF file does not show up on the trace at all.

• We can also discount the instructions after init runs by using readelf to get the initial address of init. One easy way to do that now is to just run:

  ./rungdb-user kernel_module-1.0/user/poweroff.out main

  And get that from the traces, e.g. if the address is 4003a0, then we search:

  grep -n 4003a0 trace.txt

  I have observed a single match for that instruction, so it must be the init, and there were only 20k instructions after it, so the impact is negligible.

• on arm, you need to hit Ctrl + C once after seeing the message reboot: System halted due to arm shutdown

• to disable networking. Is replacing init enough?

  • https://superuser.com/questions/181254/how-do-you-boot-linux-with-networking-disabled

  • https://superuser.com/questions/684005/how-does-one-permanently-disable-gnu-linux-networking/1255015#1255015

  CONFIG_NET=n did not significantly reduce instruction counts, so maybe replacing init is enough.

• gem5 simulates memory latencies. So I think that the CPU loops idle while waiting for memory, and counts will be higher.

I once got UML running on a minimal Buildroot setup at: https://unix.stackexchange.com/questions/73203/how-to-create-rootfs-for-user-mode-linux-on-fedora-18/372207#372207

But in part because it is dying, I didn’t spend much effort to integrate it into this repo, although it would be a good fit in principle, since it is essentially a virtualization method.

Maybe some brave should will send a pull request one day.

Let’s have some fun.

Those only work in graphical mode.

I think most are implemented under:

TODO find all.

Scroll up / down the terminal:

Or inside ./qemumonitor:

https://en.wikipedia.org/wiki/Magic_SysRq_key Those can be tested through the monitor with:

or you can try the much more boring method of:

Implemented in

Switch between TTYs with:

TODO: only works after I do a chvt 1, but then how to put a terminal on alt-f2? I just get a blank screen. One day, one day:

• https://stackoverflow.com/questions/16706423/two-instances-of-busybox-on-separate-serial-lines-ttysn

• https://serverfault.com/questions/119736/how-to-enable-multiple-virtual-consoles-on-linux

• https://superuser.com/questions/33065/console-commands-to-change-virtual-ttys-in-linux-and-openbsd

Also tried to add some extra lines to /etc/inittab of type:

but nothing changed.

Note that on Ubuntu 17.10, to get to the text terminal from the GUI we first need Ctrl + Alt + Fx, and once in the text terminals, Alt + Fn works without Ctrl.

https://stackoverflow.com/questions/40227651/does-qemu-emulator-have-checkpoint-function/48724371#48724371

QEMU allows us to take snapshots at any time through the monitor.

You can then restore CPU, memory and disk state back at any time.

qcow2 filesystems must be used for that to work.

To test it out, login into the VM with and run:

On another shell, take a snapshot:

The counting continues.

Restore the snapshot:

and the counting goes back to where we saved. This shows that CPU and memory states were reverted.

We can also verify that the disk state is also reversed. Guest:

Monitor:

Guest:

Monitor:

Guest:

And the output is 0.

Our setup does not allow for snapshotting while using initrd.

We have added and interacted with a few educational hardware models in QEMU.

This is useful to learn:

• how to create new hardware models for QEMU. Overview: https://stackoverflow.com/questions/28315265/how-to-add-a-new-device-in-qemu-source-code

• how the Linux kernel interacts with hardware

To get started, have a look at the "Hardware device drivers" section under kernel_module/README.adoc, and try to run those modules, and then grep the QEMU source code.

This protocol allows sharing a mountable filesystem between guest and host.

With networking, it’s boring, we can just use any of the old tools like sshfs and NFS.

https://superuser.com/questions/628169/how-to-share-a-directory-with-the-host-without-networking-in-qemu

One advantage of this method over NFS is that can run without sudo on host, or having to pass host credentials on guest for sshfs.

TODO performance compared to NFS.

As usual, we have already set everything up for you. On host:

Guest:

Host:

The main ingredients for this are:

• 9P settings in our kernel_config_fragment

• 9p entry on our rootfs_overlay/etc/fstab

  Alternatively, you could also mount your own with:

  mkdir /mnt/my9p
  mount -t 9p -o trans=virtio,version=9p2000.L host0 /mnt/my9p

• Launch QEMU with -virtfs as in your run script

  When we tried:

  security_model=mapped

  writes from guest failed due to user mismatch problems: https://serverfault.com/questions/342801/read-write-access-for-passthrough-9p-filesystems-with-libvirt-qemu

The feature is documented at: https://wiki.qemu.org/Documentation/9psetup

First ensure that networking is enabled before trying out anything in this section: Networking

This has nothing to do with the Linux kernel, but it is cool:

This uses QEMU’s user-mode emulation mode that allows us to run cross-compiled userland programs directly on the host.

The reason this is cool, is that ls is not statically compiled, but since we have the Buildroot image, we are still able to find the shared linker and the shared library at the given path.

In other words, much cooler than:

It is also possible to compile QEMU user mode from source with BR2_PACKAGE_HOST_QEMU_LINUX_USER_MODE=y, but then your compilation will likely fail with:

since we are using a bleeding edge kernel, which is a sanity check in the Buildroot QEMU package.

Anyways, this warns us that the userland emulation will likely not be reliable, which is good to know. TODO: where is it documented the host kernel must be as new as the target one?

GDB step debugging is also possible with:

TODO: find source. Lazy now.

When you start interacting with QEMU hardware, it is useful to see what is going on inside of QEMU itself.

This is of course trivial since QEMU is just an userland program on the host, but we make it a bit easier with:

Then you could:

And in QEMU:

Just make sure that you never click inside the QEMU window when doing that, otherwise you mouse gets captured forever, and the only solution I can find is to go to a TTY with Ctrl + Alt + F1 and kill QEMU.

You can still send key presses to QEMU however even without the mouse capture, just either click on the title bar, or alt tab to give it focus.

QEMU has a mechanism to log all instructions executed to a file.

To do it for the Linux kernel boot we have a helper:

You can then inspect the instructions with:

This functionality relies on the following setup:

• ./configure --enable-trace-backends=simple. This logs in a binary format to the trace file.

  It makes 3x execution faster than the default trace backend which logs human readable data to stdout.

  Logging with the default backend log greatly slows down the CPU, and in particular leads to this boot message:

  All QSes seen, last rcu_sched kthread activity 5252 (4294901421-4294896169), jiffies_till_next_fqs=1, root ->qsmask 0x0
  swapper/0       R  running task        0     1      0 0x00000008
   ffff880007c03ef8 ffffffff8107aa5d ffff880007c16b40 ffffffff81a3b100
   ffff880007c03f60 ffffffff810a41d1 0000000000000000 0000000007c03f20
   fffffffffffffedc 0000000000000004 fffffffffffffedc ffffffff00000000
  Call Trace:
   <IRQ>  [<ffffffff8107aa5d>] sched_show_task+0xcd/0x130
   [<ffffffff810a41d1>] rcu_check_callbacks+0x871/0x880
   [<ffffffff810a799f>] update_process_times+0x2f/0x60

  in which the boot appears to hang for a considerable time.

• patch QEMU source to remove the disable from exec_tb in the trace-events file. See also: https://rwmj.wordpress.com/2016/03/17/tracing-qemu-guest-execution/

Sometimes in Ubuntu 14.04, after the QEMU SDL GUI starts, it does not get updated after keyboard strokes, and there are artifacts like disappearing text.

We have not managed to track this problem down yet, but the following workaround always works:

This started happening when we switched to building QEMU through Buildroot, and has not been observed on later Ubuntu.

Using text mode is another workaround if you don’t need GUI features.

gem5 is a system simulator, much like QEMU: http://gem5.org/

For the most part, just add the -g option to the QEMU commands and everything should magically work:

On another shell:

A full rebuild is currently needed even if you already have QEMU working unfortunately, see: gem5 and QEMU with the same kernel configuration

Tested architectures:

• arm

• aarch64

• x86_64

Like QEMU, gem5 also has a syscall emulation mode (SE), but in this tutorial we focus on the full system emulation mode (FS). For a comparison see: https://stackoverflow.com/questions/48986597/when-should-you-use-full-system-fs-vs-syscall-emulation-se-with-userland-program

• advantages of gem5:

  • simulates a generic more realistic pipelined and optionally out of order CPU cycle by cycle, including a realistic DRAM memory access model with latencies, caches and page table manipulations. This allows us to:

    • do much more realistic performance benchmarking with it, which makes absolutely no sense in QEMU, which is purely functional

    • make certain functional cache observations that are not possible in QEMU, e.g.:

      • use Linux kernel APIs that flush memory like DMA, which are crucial for driver development. In QEMU, the driver would still work even if we forget to flush caches.

      • TODO spectre / meltdown

        It is not of course truly cycle accurate, as that

  • would require exposing proprietary information of the CPU designs: https://stackoverflow.com/questions/17454955/can-you-check-performance-of-a-program-running-with-qemu-simulator/33580850#33580850

  • would make the simulation even slower TODO confirm, by how much

    but the approximation is reasonable.

    It is used mostly for microarchitecture research purposes: when you are making a new chip technology, you don’t really need to specialize enormously to an existing microarchitecture, but rather develop something that will work with a wide range of future architectures.

  • runs are deterministic by default, unlike QEMU which has a special [record-and-replay] mode, that requires first playing the content once and then replaying

• disadvantage of gem5: slower than QEMU, see: gem5 vs QEMU performance

  This implies that the user base is much smaller, since no Android devs.

  Instead, we have only chip makers, who keep everything that really works closed, and researchers, who can’t version track or document code properly >:-) And this implies that:

  • the documentation is more scarce

  • it takes longer to support new hardware features

  Well, not that AOSP is that much better anyways.

• not sure: gem5 has BSD license while QEMU has GPL

  This suits chip makers that want to distribute forks with secret IP to their customers.

  On the other hand, the chip makers tend to upstream less, and the project becomes more crappy in average :-)

OK, this is why we used gem5 in the first place, performance measurements!

Let’s benchmark Dhrystone which Buildroot provides.

The most flexible way is to do:

These commands output the approximate number of CPU cycles it took Dhrystone to run.

A more naive and simpler to understand approach would be a direct:

but the problem is that this method does not allow to easily run a different script without running the boot again, see: gem5 checkpoint restore and run a different script

A few imperfections of our benchmarking method are:

• when we do m5 resetstats and m5 exit, there is some time passed before the exec system call returns and the actual benchmark starts and ends

• the benchmark outputs to stdout, which means so extra cycles in addition to the actual computation. But TODO: how to get the output to check that it is correct without such IO cycles?

Solutions to these problems include:

• modify benchmark code with instrumentation directly, as PARSEC and ARM employees have been doing: https://github.com/arm-university/arm-gem5-rsk/blob/aa3b51b175a0f3b6e75c9c856092ae0c8f2a7cdc/parsec_patches/xcompile-patch.diff#L230

• monitor known addresses

Discussion at: https://stackoverflow.com/questions/48944587/how-to-count-the-number-of-cpu-clock-cycles-between-the-start-and-end-of-a-bench/48944588#48944588

Those problems should be insignificant if the benchmark runs for long enough however.

Now you can play a fun little game with your friends:

• pick a computational problem

• make a program that solves the computation problem, and outputs output to stdout

• write the code that runs the correct computation in the smallest number of cycles possible

To find out why your program is slow, a good first step is to have a look at the statistics for the run:

Whenever we run m5 dumpstats or m5 exit, a section with the following format is added to that file:

Analogous to QEMU:

Internals: when we give --command-line= to gem5, it overrides default command lines, including some mandatory ones which are required to boot properly.

Our run script hardcodes the require options in the default --command-line and appends extra options given by -e.

To find the default options in the first place, we removed --command-line and ran:

and then looked at the line of the Linux kernel that starts with:

Analogous to QEMU’s Snapshot, but better since it can be started from inside the guest, so we can easily checkpoint after a specific guest event, e.g. just before init is done.

Documentation: http://gem5.org/Checkpoints

In the guest, wait for the boot to end and run:

To restore the checkpoint, kill the VM and run:

Let’s create a second checkpoint to see how it works, in guest:

Kill the VM, and try it out:

and now in the guest:

contains the date. The file f wouldn’t exist had we used the first checkpoint with -r 1.

If you automate things with Kernel command line parameters as in:

Then there is no need to pass the kernel command line again to gem5 for replay:

since boot has already happened, and the parameters are already in the RAM of the snapshot.

Our scripts "namespace" with the checkpoint by architecture with --checkpoint-dir, so if you make two checkpoints:

• one in x86

• the other in arm

Then you would still restore both of them with -- -r 1.

This makes it easier to remember which checkpoint is which, especially since there appears to be no runtime way to set the checkpoint names.

Internals:

• the checkpoints are stored under out/$arch/gem5/m5out/cpt.$todo_whatisthis

• m5 is a guest utility present inside the gem5 tree which we cross-compiled and installed into the guest

Pass options to the fs.py script:

• get help:

  ./run -g -- -h

• boot with the more detailed and slow HPI CPU model:

  ./run -a arm -g -- --caches --cpu-type=HPI

Pass options to the gem5 executable itself:

• get help:

  ./run -G '-h' -g

gem5 just assigns new ports if some ports are occupied, so we can do:

And a second instance:

TODO Now we just need to network them up to have some more fun! See dist-gem5: http://publish.illinois.edu/icsl-pdgem5/

We would like to be able to run both gem5 and QEMU with the same minimal kernel build to:

• do a single Buildroot build for both. Otherwise, we have to create two full out/.*/buildroot/ directories, which takes up a lot of time.

  Alternatively, we could try to be brave and switch between two kernel builds inside out/.*/buildroot/, but that would be too hackish.

• be able to compare behaviour between QEMU and gem5 when one is doing something weird.

  Note however that there are also variations which need to be controlled, e.g. kernel command line, DTB and QEMU’s non-determinism.

Unfortunately, we have only managed to find a working config for aarch64, which just works transparently.

The others use the Buildroot config for QEMU, and magic huge post-olddefconfig config files floating around the web for GEM5.

Subsections of this section document our failed attempts so far.

This is the strategy that we used to make it work for aarch64:

• make savedefconfig on the working gem5 kernel tree

• paste the result on kernel_config_fragment

• bisect it up

but this strategy failed for the other archs for some reason.

m5 is a guest command line utility that is installed and run on the guest.

It generates magic instructions, which lead gem5 to do magic things, like dumpstats or exit.

It is however under-documented, so let’s document some of its capabilities here.

• networking not working. We currently just disable it from inittab by default to prevent waiting at startup

This method runs the kernel modules directly on your host computer without a VM, and saves you the compilation time and disk usage of the virtual machine method.

It has however severe limitations, and you will soon see that the compilation time and disk usage are well worth it:

• can’t control which kernel version and build options to use. So some of the modules will likely not compile because of kernel API changes, since the Linux kernel does not have a stable kernel module API.

• bugs can easily break you system. E.g.:

  • segfaults can trivially lead to a kernel crash, and require a reboot

  • your disk could get erased. Yes, this can also happen with sudo from userland. But you should not use sudo when developing newbie programs. And for the kernel you don’t have the choice not to use sudo

  • even more subtle system corruption such as not being able to rmmod

• can’t control which hardware is used, notably the CPU architecture

• can’t step debug it with GDB easily

Still interested?

If the compilation of any of the C files fails because of kernel or toolchain differences that we don’t control on the host, just rename it to remove the .c extension and try again:

Once you manage to compile, and have come to terms with the fact that this may blow up your host, try it out with:

Once you are done with this method, you must clean up the in-tree build objects before you decide to do the right thing and move on to the superior ./build Buildroot method:

otherwise they will cause problems.

Minimal host build system sanity check example.

We provide the following mechanisms:

• ./build -b mybr2.gitignore: append the file mybr2.gitignore to a single build. Must be passed every time you run ./build. A good template is provided by:

• ./build -B 'BR2_SOM_OPTION="myval"': append a single option to a single build.

make menuconfig is a convenient way to find Buildroot configurations:

Hit / and search for the settings.

Save and quit.

Then copy and paste the diff additions to br2 to make them permanent.

At startup, we login automatically as the root user as explained at: https://unix.stackexchange.com/questions/299408/how-to-login-automatically-without-typing-root-in-buildroot-x86-64-qemu/300152#300152

If you want to switch to another user to test some permissions, we have already created an user0 user through the user_table file, and you can just login as that user with:

and password:

Then test that the user changed with:

which gives:

We have enabled ccached builds by default.

BR2_CCACHE_USE_BASEDIR=n is used, which means that:

• absolute paths are used and GDB can find source files

• but builds are not reused across separated LKMC directories

ccache can considerably speed up builds when you:

• are switching between multiple configurations for a given package to bisect something out, as mentioned at: Use your own kernel config

• clean the build because things stopped working. We store the cache outside of this repository, so you can nuke away without fear

The default ccache environment variables are honored if you have them set, which we recommend you do. E.g., in your .bashrc:

The choice basically comes down to:

• should I store my cache on my HD or SSD?

• how big is my build, and how many build configurations do I need to keep around at a time?

If you don’t set it, the default is to use ~/.buildroot-ccache with 5G, which is a bit small for us.

I find it very relaxing to watch ccache at work with:

or if you have it installed on host and the environment variables exported simply with:

while a build is going on in another terminal and my cooler is humming. Especially when the hit count goes up ;-) The joys of system programming.

When adding new large package to the Buildroot root filesystem, it may fail with the message:

The solution is to simply add:

where 512Mb is "large enough".

Note that dots cannot be used as in 1.5G, so just use Megs as in 1500M instead.

Unfortunately, TODO we don’t have a perfect way to find the right value for BR2_TARGET_ROOTFS_EXT2_SIZE. One good heuristic is:

https://stackoverflow.com/questions/49211241/is-there-a-way-to-automatically-detect-the-minimum-required-br2-target-rootfs-ex

One way to overcome this problem is to mount benchmarks from host instead of adding them to the root filesystem, e.g. with: 9P

Buildroot is not designed for large root filesystem images, and the rebuild becomes very slow when we add a large package to it.

This is due mainly to the pkg-generic GLOBAL_INSTRUMENTATION_HOOKS sanitation which go over the entire tree doing complex operations…​ I no like, in particular check_bin_arch and check_host_rpath, which get stuck for a long time on the message:

The pause is followed by:

so which shows that the whole delay is inside our install itself.

I put an echo f in check_bin_arch, and it just loops forever, does not stop on a particular package.

Our philosophy is:

• if something adds little to the build time, build it in by default

• otherwise, make it optional

• try to keep the toolchain (GCC, Binutils) unchanged, otherwise a full rebuild is required.

  So we generally just enable all toolchain options by defaut, even though this adds a bit of time to the build.

  The biggest build time hog is always GCC, and it does not look like we can use a precompiled one: https://stackoverflow.com/questions/10833672/buildroot-environment-with-host-toolchain

• if something is very vaulable, we just add it by default even if it increases the Build time, notably GDB and QEMU

• runtime is sacred.

  We do our best to reduce the instruction and feature count to the bare minimum needed, to make the system:

  • easier to understand

  • run faster, specially for gem5

  One possibility we could play with is to build loadable modules instead of built-in modules to reduce runtime, but make it easier to get started with the modules.

The build times are calculated after doing make source, which downloads the sources, and basically benchmarks the Internet.

https://stackoverflow.com/questions/47997565/gem5-system-requirements-for-decent-performance/48941793#48941793

https://git.kernel.org/pub/scm/utils/kernel/kmod/kmod.git

Multi-call executable that implements: lsmod, insmod, rmmod, and other tools on desktop distros such as Ubuntu 16.04, where e.g.:

gives:

and:

contains:

BusyBox also implements its own version of those executables. There are some differences.

Buildroot also has a kmod package, but we are not using it since BusyBox' version is good enough so far.

This page will only describe features that differ from kmod to the BusyBox implementation.

platform_device.c together with its kernel and QEMU forks contains a minimal runnable example.

Good format descriptions:

• https://www.raspberrypi.org/documentation/configuration/device-tree.md

Minimal example

Check correctness with:

Separate nodes are simply merged by node path, e.g.:

then dtc a.dts gives:

buildroot_patches/README.adoc

global_patch_dir/README.adoc

kernel_module/README.adoc

kernel_module/user/README.adoc

rootfs_overlay/README.adoc

build-usage.adoc

run-usage.adoc

CONTRIBUTING.adoc

This project is for people who want to learn and modify low level system components:

• Linux kernel and Linux kernel modules

• full systems emulators like QEMU and gem5

• C standard libraries. This could also be put on a submodule if people show interest.

• Buildroot. We use and therefore document, a large part of its feature set.

Philosophy:

• automate as much as possible to make things more reproducible

• do everything from source to make things understandable and hackable

This project should be called "Linux kernel playground", like: https://github.com/Fuzion24/AndroidKernelExploitationPlayground maybe I’ll rename it some day. Would semi conflict with: http://copr-fe.cloud.fedoraproject.org/coprs/jwboyer/kernel-playground/ though.

Runnable stuff:

• https://lwn.net/Kernel/LDD3/ the best book, but outdated. Updated source: https://github.com/martinezjavier/ldd3 But examples non-minimal and take too much brain power to understand.

• https://github.com/satoru-takeuchi/elkdat manual build process without Buildroot, very few and simple kernel modules

• https://github.com/tinyclub/linux-lab Buildroot based, no kernel modules?

• https://github.com/agelastic/eudyptula

• https://github.com/linux-kernel-labs Yocto based, source inside a kernel fork subdir: https://github.com/linux-kernel-labs/linux/tree/f08b9e4238dfc612a9d019e3705bd906930057fc/tools/labs which the author would like to upstream https://www.reddit.com/r/programming/comments/79w2q9/linux_device_driver_labs_the_linux_kernel/dp6of43/

• Android AOSP: https://stackoverflow.com/questions/1809774/how-to-compile-the-android-aosp-kernel-and-test-it-with-the-android-emulator/48310014#48310014 AOSP is basically a uber bloated Buildroot (2 hours build vs 30 minutes), Android is Linux based, and QEMU is the emulator backend. These instructions might work for debugging the kernel: https://github.com/Fuzion24/AndroidKernelExploitationPlayground

Theory:

• http://nairobi-embedded.org you will fall here a lot when the hard Google queries start popping. They have covered everything we do here basically, but with a more manual approach, while this repo automates everything.

• https://balau82.wordpress.com awesome low level resource

• https://rwmj.wordpress.com/ awesome red hatter

• https://lwn.net

• http://www.makelinux.net

Awesome lists:

• https://github.com/gurugio/lowlevelprogramming-university

• https://github.com/uhub/awesome-c