The intent is to have the same user experience for managing multiple programs regardless of the program type. In particular, programs should be prioritized in the same way and we should support the atomic addition and deletion of programs.

The goal for this issue is to prove out the following design.

Instead of adding tc programs directly to the tc subsystem, the intent is to use a dispatcher in the same way as is currently done for xdp programs.

Atomic replacement will be accomplished as follows:

1. If none of the added tc programs returns a terminal value, the dispatcher will return a TC_ACT_OK so that no lower priority tc programs get executed.
2. When a program is added or deleted, we will create a new dispatcher program with the new list of programs as eBPF extension programs.
3. The new dispatcher program will first be added at a higher priority than the current dispatcher program first, and then the old dispatcher will be removed.
4. A set of tc priority slots will be reserved for bpfd programs. We will need to handle the wrap case in which the current dispatcher is at the highest priority of the reserved priorities. We intend to handle this case by moving the current dispatcher to the lowest priority before the new dispatcher is added.

It might be desirable to expose the API over DBus. We should investigate if this is possible with https://crates.io/crates/zbus and what features we might not be able to support (not sure how authn would work).

See "New style daemons" in man daemon(7):
https://man7.org/linux/man-pages/man7/daemon.7.html

Hopefully we don't need to be a SysV daemon...

• #175
• #91
• #92
• Provide a systemd service file
• Rely on systemd to manage privileges (we do, but we also drop some ourselves on unpriv threads for security)
• #463
• #464
• #465

This depends on #8.
Initial use case would be to provide the path to the certificate to avoid having to pass it via flag to every command.

Follow up to #102

Simply add some unit tests for the new bpfd bytecode pulling functionality in pull_bytecode.go

Test...

• A bad image url
• A malformed image (empty or otherwise not following the spec)
• A non-public image
• etc ...

Assuming the BTF types are compatible I'm hopeful that this is posisble

Initially this should allow setting the XDP mode for an interface

Current layout in /etc:

/etc/bpfctl/
├── bpfctl.toml
└── certs
    ├── bpfctl
    │   ├── bpfctl.key
    │   └── bpfctl.pem
    └── gocounter
        ├── gocounter.key
        └── gocounter.pem

Notes:

1. bpfctl should support local configuration with a higher precedence e.g in ~/.config/bpfctl or somesuch.

2. We use the SubjectCN of certificates as the username as far as bpfd is concerned. This will be used to correctly set ownership of files in /var/run/bpfd/bpffs with ${username}:bpfd and to set the ACL to be something like 0664 to allow r/w to owner and bpfd group and r/o to everyone else (see #18) as well as to determine API access. We have 2 choices:
   a. Implement (1) and deprecate using /etc for bpfctl
   b. Decouple the transport authentication (TLS certs) from the user authentication somehow. i.e have our own username/apitoken database that we use for authentication. Then you can bootstrap a /etc/bpfd/certs/client/client.pem certificate that any user in the right group can use... but you must also provide an API token.

Since #8 the API is restricted to authenticated users.
The process of authentication is to provide $user with a certificate.

Currently the API consists of:

• Load
• Unload
• GetMap

With #36 operations will only be permitted against resources that are owned by the user.
And GetMap is likely to be deprecated as pinned map paths can be sent in the LoadResponse.

That leaves only 2 APIs that would require authorization and it doesn't seem necessary at this point.
However if we decide to add #9 perhaps there is an argument for restricting the scope of calls made to only permit the loading of certain program types.

Since the map fd over UDS API is here to stay, we want to make sure we get a new fd to send to userspace.
That new fd should only allow map operations that we want... and kernel support for that in unvpriv bpf is landing soon.

Would require some work in Aya to:

1. Call bpf(BPF_OBJ_GET_INFO_BY_FD) to populate the map info the Map struct
2. Add an API to call Map::new_fd(flags), which would do bpf(BPF_MAP_GET_FD_BY_ID) where flags should be an enum that supports ReadOnly and ReadWrite where the former propagates BPF_F_RDONLY and the latter, none.

CONTRIBUTING.MD

• Explain the Github Workflow
• Explain patch formatting with cargo +nightly fmt
• Explain patch liniting with cargo +nightly clippy

docs/design.md

• Explains a little bit of the motivation behind bpfd and how it is designed

docs/quickstart.md

• A walkthrough of how to start and configure bpfd
• An example of how to deploy the sample programs in-tree
• An example of how to manage (list/unload) applications

docs/go-client.md

• A walkthrough of the example code example/gocounter that explains how to use the go bindings example/gobpfd

1. I think I removed the bpf code that this used 😅 It might make more sense to use whatever cilium/ebpf recommends or just write the prog in bpf asm directly as it's only small. Was inspired by this program
2. It now needs a certificate generated - ./scripts/certificates.sh should be updated to allow easy generation of a client cert.

With the addition of adding the ability to load programs at daemon startup, bpfd won't run if "/etc/bpfd/programs.d" doesn't exist:

$ sudo ./target/debug/bpfd
Error: No such file or directory (os error 2)

Some interfaces have an altname.

$ ip a
:
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:42:48:36 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 192.168.122.118/24 brd 192.168.122.255 scope global dynamic noprefixroute ens3
       valid_lft 2686sec preferred_lft 2686sec
    inet6 fe80::5054:ff:fe42:4836/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

If a program is inserted on ens3 and then insert one on enp0s3, then the bpfd panics. Since the index to the map is the interface name, they are treated as different interfaces, so on the second call, there is a panic when the interface is attempted to be taken. Switching the index to map to be the ifindex would ensure unique interfaces and therefore less error prone.

Programs created by $user should only be allowed to be removed by $user or superuser (i.e bpfctl)

We should consider (and implement) the best way of deploying bpfd in a k8s environment.

1. A user would install bpfd-operator which would add a CRD that lets K8s users submit BPF programs to be loaded
2. The operator would install a new component bpfd-agent via a DaemonSet. bpfd-agent would be a K8s controller that watches for changes in the CRD, and (if applicable) uses bpfd's GRPC API to perform the operations against bpfd on the host.

Open questions:

1. Is bpfd installed on the host (i.e via rpm) or is it a running in a DaemonSet - or can we support both models?
2. Can we re-use the TOML config file format described in #30 for this?
3. Does bpfd somehow need to tag resources that are managed by k8s - i.e in case of controller restart/upgrade so it can re-compute it's state.

Most of the upstream work that was blocking this has now been implemented in Aya

• Ability to manipulate .rodata maps
• Support for loading BTF into the kernel
• Support for BPF_PROG_TYPE_EXT
• Ability to load program from a pinned file

We may also need

• Ability to load a map from a pinned file. See: aya-rs/aya#75

Generate a client using the protobuf file and load a program form Go.

According to this article stdout/stderr is automatically captured.
We should use the correct format for logs to get correctly picked at the right levels in the journal...
So either syslog, if /dev/log is supported/available or perhaps using a log file.

UPDATE: looks like https://docs.rs/systemd-journal-logger/0.5.0/systemd_journal_logger/ could solve this for us.
And using a snippet like this we can fall back to using env_logger if not connected to the journal

Aya is making this change also

We should use cargo-fuzz to fuzz the API to ensure that we don't panic on invalid input.
Fuzzing should be guided with valid input data for each API.

From a bpfd standpoint we should:

1. Build config from BTF (skip that for now)
2. Apply overrides from image labels (if container)
3. Apply overrides from the API request

That probably means that a lot of what's in the GRPC api now can actually become optional... and the CLI design would follow.
So --path and --image are the only required items (mutually exclusive)
Everything else would be Option<...>

This issue will track the work to explore how GRPC handles optional fields, and how we can optimize the API and bpfd to reduce "data over the wire" as much as possible.

This is blocked on having stable BTF from rustc. See aya-rs/bpf-linker#1

1. Remove bpfd/build.rs
2. Follow https://github.com/hyperium/tonic/tree/master/tonic-build#google-apis-example to add a command to the xtask crate to generate the code, setting out_dir to bpfd-api/src/api_v0.rs and adding a lib.rs appropriately.
3. Update dev docs on new process for updating protobuf files.

Bonus

1. Migrate the bash script for updating Go bindings to the same xtask.

Motivation

We can remove the build script setting from rust-analyzer. Builds should be faster.

Originates from #102 (comment)

In #102 we introduced the ability to extract bytecode from container images.

These images are pulled from remote repositories and currently only public repositories are supported.

We need to add a config option to bpfd.toml to authenticate against private image repositories

something like

[ registry ]

[ registry.docker-io ]
username = foo
password = bar

and then nfer the registry from the image path and login if required.

and/or provide package names for Debian/Ubuntu and other distros

The current issue is "func#0 @0\nSubprog prog0 doesn't exist which is likely related to the func_info and line_info that we get in .BTF.ext. We could be not correctly handling this in bpf-linker or, more likely, in aya.

Targeting Go might be a good first step.
A go client should be able to load a program using this API client.

Currently, the lifecycle of all bpf loaded by bpfd is equivalent to the lifetime of the process.
Once bpfd dies, all bpf programs are detached and unloaded from the kernel.

For systemctl restart bpfd, this behaviour isn't desirable.
We can either:

Pin all programs, links etc...

Pro:

• eBPF code continues to run unhindered while the daemon restarts

Con:

• We need to manage additional code to pin all objects... and to remove the reference from bpffs when we drop them to avoid leaking resources. Not only that, we also need to rebuild our runtime state using the BPF query APIs - some work may be necessary upstream in Aya to make this possible.

Persist state to /var/run/bpfd/$iface/$program.toml

Pro:

• Less lifecycle code to maintain as all runtime state can successfully be rebuilt from /var/run

Cons:

• eBPF programs will be unloaded/reloaded

I'm not really sure which option is best as of now.

I think the best way is probably to have an API like GetMap(iface, id, map_name, path_to_uds).
Since gRPC doesn't have a native API for passing FDs, we could probably clone the map fd and send a copy over a UDS.
Userspace programs would then need an API for creating a map from an FD, but hopefully some existing BPF libs already have this feature

https://docs.fedoraproject.org/en-US/packaging-guidelines/Rust/

We should build RPMs in CI that we can use for testing...

If you pass a bad section name using:

sudo ./target/debug/bpfctl load -p xdp -i wlp2s0 -s "invalid" --priority 50 ~/dev/xdp-tutorial/basic01-xdp-pass/xdp_pass_kern.o

We return an error, but we never clean up the program metadata.
As a result, a subsequent call with a valid section name will fail, since it's considered a "new" extension, and the old failed one will still fail to load.

We should Bpf::load_file(...) and bpf.programs_get() BEFORE we add this to our list extension programs.
We don't need to actually load the program (although we could, and just not attach it)

A simple script to standup a VM, install Fedora 36, install the RPMs from #5 and provide SSH access for testing.
For bonus points we could do with a kernel that's built on net-next with unprivileged BPF enabled.

Running in a F36 VM, if two programs are added to an interface and then 'unloaded`, I lose access to the VM and need to reboot it.

$ sudo ./target/debug/bpfctl load -p xdp -i ens3 --priority 55 -s "xdp" /home/bmcfall/src/xdp-tutorial/basic01-xdp-pass/xdp_pass_kern.o
f7fe8625-5453-4eca-bb0b-c9130f15d6ea
$ sudo ./target/debug/bpfctl load -p xdp -i ens3 --priority 60 -s "xdp" /home/bmcfall/src/xdp-tutorial/basic01-xdp-pass/xdp_pass_kern.o
909edd07-f023-4dd1-9fee-16b0975bffd6

$ sudo ./target/debug/bpfctl list -i ens3
ens3
xdp_mode: skb

0: f7fe8625-5453-4eca-bb0b-c9130f15d6ea
	name: "xdp"
	priority: 55
	path: /home/bmcfall/src/xdp-tutorial/basic01-xdp-pass/xdp_pass_kern.o
1: 909edd07-f023-4dd1-9fee-16b0975bffd6
	name: "xdp"
	priority: 60
	path: /home/bmcfall/src/xdp-tutorial/basic01-xdp-pass/xdp_pass_kern.o

$ sudo ./target/debug/bpfctl unload -i ens3 f7fe8625-5453-4eca-bb0b-c9130f15d6ea
$ sudo ./target/debug/bpfctl unload -i ens3 909edd07-f023-4dd1-9fee-16b0975bffd6
packet_write_wait: Connection to 192.168.122.118 port 22: Broken pipe

After Loading and unloading an xdp program which defines a map, such as the one in #53 , the dispatcher program is left attached to the specified interface and the program's map is orphaned.

To recreate the stale map scenario simply load and unload a BPF program
which defines a new map.

An easy way to recreate is to checkout #53,
run the go example, exit it while leaving bpfd running, you'll see that the
stats program is gone but it's map xdp_stats_map is left orphaned even after closing
the FD.

[astoycos@localhost bpfd]$ sudo bpftool map list name xdp_stats_map
733: percpu_array  name xdp_stats_map  flags 0x0
        key 4B  value 16B  max_entries 5  memlock 4096B
        pids bpfd(644765)

Clearly bpfd is still owning/ maintaing a reference to this even though the program that
defined it (the xdp counter prog) no longer exists.

#55 Is a first attempt at fixing this however I don't know if it's the right method.

Currently depends on aya-rs/aya#282
And the MapFd trait I added...
And various other things.

The dispatcher code is already written but hasn't been tested. We can already attach multiple programs to a TC hook but having deterministic ordering using a dispatcher seems a better option - I think. Either way we should investigate whether a dispatcher is necessary, or whether the existing mechanism in kernel can be tamed.

While the API is built specifically to handle network (TC/XDP) program types, there is an argument to be made that supporting other programs could be useful.

In which case we'd need to:

1. Rename our current Load API to LoadXDP and maybe LoadTC depending on #9
2. Add new APIs for the other supported program types

Expanding the API to support more program types would likely require #37 to reduce the scope of authorized APIs for each user.

See: https://github.com/xdp-project/xdp-tools/blob/master/lib/libxdp/protocol.org#populating-the-dispatcher-configuration-map

From the command line, we should have something like --proceed-on pass --proceed-on redirect.. where that argument can be repeated multiple times...
The default would be to assume --proceed-on pass. Should be a Vec<XdpAction>, where XdpAction is an enum that has a TryFrom<String> implementation - see ProgramType for reference.

The bpfd.proto file will need a new argument added to the Load RPC. I would suggest a repeated arg of type String.

On the server side, we will need some code to convert the Vec<String> into a bitmap (see: libxdp protocol linked above).
The bitmap would be cached in ExtensionProgram for re-use...
And it needs to be populated into the XdpDispatcherConfig

We should support loading from /etc/bpfd/programs.d/foo.toml.
On start, the daemon would parse files from this directory and load them to the specified interface.
The configuration would look something like this:

[program]
interface ="eth0"
path = "/opt/bin/myapp/lib/myebpf.o"
section_name = "firewall"
program_type ="xdp"
priority = 50

It should be relatively easy to do using oci-rs.

1. Define the rules that images should conform to. At the minimum we'll need an annotation/label to identify this as eBPF.
   We may want to add optional labels/annotations to include API defaults - like program_section etc.. - but we would also need to document how these can be overridden.
2. Create an image repository: /var/lib/bpfd/images
3. Add the ability for bpfd to pull an image from a registry and extract the bytecode from the rootfs.
4. As a bonus we might want to consider integration with cosign to prevent the load of unsigned images.

What options do we have to restrict who can access the API - we don't want just anyone to access this

.. and probably CAP_NET_ADMIN and maybe some caps to access sysfs.

We don't need full superuser privilieges

Relates to #37

Not quite sure how we'd model policy for this, but it's certainly an interesting use case.