We may want to make sure to run some Linux kernel CI using this repository when a pull request is submitted and/or a change is pushed to master. I'd guess (but have not checked) that we could probably reuse large swaths of libbpf CI hooks to go about that.

Please add initial documentation for this repo.

Logs are showing:

  #250     verif_stats:OK
  #251     verif_twfw:OK
  ./verify_sig_setup.sh: line 41: openssl: command not found
  ./verify_sig_setup.sh: line 58: keyctl: command not found
  ./verify_sig_setup.sh: line 58: keyctl: command not found
  test_verify_pkcs7_sig:PASS:mkdtemp 0 nsec
  test_verify_pkcs7_sig:FAIL:_run_setup_process unexpected error: 127 (errno 126)

TL;DR: I'm trying to fix feature detection for LLVM in the kernel's tools/build system. If I do, this breaks the CI because bpftool misses the LLVM libraries as runtime dependencies.

Context

In the kernel repository, under tools/build/feature, we can test for a number of "features" that may or may not be supported by the host for building tools. These are used by perf or bpftool, for example. In particular, there is a probe to detect the availability of LLVM libraries. But the detection is broken for LLVM v16+, and new versions of LLVM are always reported as missing. I'm trying to fix that.

The fix is simple. So I've got a patch in preparation for it. But this patch also breaks the BPF CI.

Issue

Here's what happens:

• With the patch applied and LLVM feature detection fixed, LLVM libraries are correctly reported as available in the CI setup when building bpftool.

• Bpftool uses the LLVM disassembler for JIT-ed programs by default; given that the libraries are available, it links against them.

• But the libraries also need to be available at runtime for bpftool to work properly. So when running in the VM image, bpftool invocations fail. In test_progs and test_progs_no_alu32, core_reloc attempts to run bpftool gen min_core_btf and fails with the following message:

  ./bpftool: error while loading shared libraries: libLLVM-17.so.1: cannot open shared object file: No such file or directory
  

The reason we never noticed the issue before is likely that the runtime dependencies are met for the alternative libbfd-based disassembler (binutils is installed in the rootfs), and that the CI has likely been using LLVM 16 or higher since before the switch to the LLVM disassembler by default was merged.

Possible fix

The "easy" fix would be to add the LLVM libraries to the rootfs image (using the same version as used to compile bpftool, not just Debian's llvm-dev). But I'm not sure this is desirable, as it would increase the size and build time for the image. So I'm opening this Issue to get some feedback and see whether this is acceptable, or if someone has a better alternative to suggest.

Note: We don't need the JIT-disassembler at all for the selftests at the moment, so if we could build bpftool without LLVM support for selftests, this should solve the issue. But we don't currently have a clean way to disable LLVM support in bpftool (if the libraries are here, we use them, period). We could possibly hack something by crafting and passing a specific value for $(FEATURE_TESTS), or abusing $(LLVM_CONFIG) to make the LLVM feature detection fail, but both hacks would require a way to pass down arguments to the call to $(MAKE) for building bpftool from selftests' Makefile, which we currently don't have. And it doesn't look super clean. Similarly, building bpftool statically would require passing down the right flags, as well as getting compatible LLVM libraries (the default ones can only be used for linking dynamically - we'd have to build them or download some, like in bpftool's CI).

Hi,

I was encountering some relocation issue when test PR on BPF-CI. After some digging the error says:

libbpf: prog 'test_lwt_redirect_in': bad map relo against '.rodata.str1.1' in section '.rodata.str1.1'

It turns out the latest root image libbpf-vmtest-rootfs-2022.10.23-bullseye-amd64.tar.zst has iproute2 with libbpf-0.3, and support for .rodata. was introduced in commit 5bf62459b185 ("libbpf: Support multiple .rodata.* and .data.* BPF maps") from v0.6.0. Unpacking and running the root image locally with same BPF object confirms that upgrading libbpf0 would fix the issue.

Will it be feasible if admins can update the root image to pull libbpf0 from bullseye-backport? It does have slight newer version https://packages.debian.org/bullseye-backports/libbpf0, or better if we can have newer iproute2 and libbpf1 in the root image?

thanks

Github actions run are throwing a bunch of deprecation warning:

The checks seem valid yes, when the vm starts we get this error in the logs (see https://github.com/kernel-patches/bpf/actions/runs/3303109637/jobs/5450787223#step:6:425)

+ [ -x /etc/rcS.d/S10-mount ]
  + /etc/rcS.d/S10-mount
  + /bin/mount proc /proc -t proc
  + /bin/mount -t devtmpfs
  + [[ -z devtmpfs on /dev type devtmpfs (rw,relatime,size=3029136k,nr_inodes=757284,mode=755) ]]
  [[: on: unknown operand
  + /bin/mount sysfs /sys -t sysfs
  + /bin/mount bpffs /sys/fs/bpf -t bpf
  + /bin/mount debugfs /sys/kernel/debug -t debugfs
  + [[ -a /dev/fd ]]
  [[: /dev/fd: unknown operand
  + ln -s /proc/self/fd /dev/fd
  + echo Listing currently mounted file systems
  Listing currently mounted file systems
  + /bin/mount
  /dev/vda on / type ext4 (rw,relatime)

Building x-arch rootfs started failing. This is because a change may have happen in bullseye recently that calls a binary as part of debootstrap:

The exact error is:

I: Extracting zlib1g...
W: Failure trying to run: chroot "/home/chantra/devel/libbpf/ci/rootfs/tmp.jwLB9EVVaG" /bin/true
W: See /home/chantra/devel/libbpf/ci/rootfs/tmp.jwLB9EVVaG/debootstrap/debootstrap.log for details

and the actual log line:

chroot: failed to run command '/bin/true': Exec format error

The solution to this is to use the --foreign option of debootstrap and using qemu.

Example: https://askubuntu.com/questions/287789/what-is-debootstrap-second-stage-for

If the container does not have any /boot/vmlinuz* file, preepare-rootfs will fail with:

Disk image: /tmp/root.img
vmlinux_setup - Preparing Linux image
  
  chmod: cannot access '/boot/vmlinuz*': No such file or directory
  libguestfs: error: /usr/bin/supermin exited with error status 1.
  To see full error messages you may need to enable debugging.
  Do:
    export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
  and run the command again.  For further information, read:
    http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
  You can also run 'libguestfs-test-tool' and post the *complete* output
  into a bug report or message to the libguestfs mailing list.
  Error: Process completed with exit code 2.

This is due to

Installing linux-image-generic in the container works.

apt install linux-image-generic

but we should re-assess if this is really needed still.

Currently, we run tests in a Debian Bullseye rootfs. This works with our current CI runners (Ubuntu focal) but this is very brittle.

1. in the past, we had some dependency issues.
2. people may run vmtests from different hosts then Ubuntu focal.

The way we do rootfs selection at the moment is pretty simple: https://github.com/libbpf/ci/blob/1b794627fd7431428a7e5a077010285733eaedf2/prepare-rootfs/run.sh#L223C26-L231

The rootfs have the distro release in their name, so we could possibly filter on this.
The script to generate rootfs uses debootstrap and should work with pretty much any Debian derivative.

We could generate a rootfs for common rootfs we care about, put them all in INDEX and when selecting the rootfs, try to match the hosts' distro, and fallback on a default one otherwise.

This should help with #83, and avoid issues like we had in the past (see comment in that PR).

Because currently libbpf/ci and vmtest in kernel tree differ, we would also need to update
https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/tree/tools/testing/selftests/bpf/vmtest.sh?id=c1a3daf7363b48c6c4b86aee2efa2287f695f649#n106 with the change.

Given the current behaviour of that function, we may be able to get away with updating the index file and the test will just pick up whatever is latest alphabetically.