Modify the build pipeline both upstream & downstream to build Falco with modern bpf enabled.

Part of #1014

Hello Team, received the following error while deploying the collector in openshift 4.9. Initially thought this is a permission issue and added the required SCC to collector's service account, but still the issue persists.

terminate called after throwing an instance of 'scap_open_exception'
  what():  can't create map: Permission denied
collector[0x448f7d]
/lib64/libc.so.6(+0x4eb80)[0x7f726981fb80]
/lib64/libc.so.6(gsignal+0x10f)[0x7f726981faff]
/lib64/libc.so.6(abort+0x127)[0x7f72697f2ea5]
/lib64/libstdc++.so.6(+0x9009b)[0x7f726a1c109b]
/lib64/libstdc++.so.6(+0x9653c)[0x7f726a1c753c]
/lib64/libstdc++.so.6(+0x96597)[0x7f726a1c7597]
/lib64/libstdc++.so.6(+0x967f8)[0x7f726a1c77f8]
/usr/local/lib/libsinsp-wrapper.so(+0x240ef5)[0x7f726c82cef5]
/usr/local/lib/libsinsp-wrapper.so(_ZN5sinsp4openERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x36)[0x7f726c866c16]
collector[0x4d2b34]
collector[0x46631c]
collector[0x442bec]
/lib64/libc.so.6(__libc_start_main+0xe5)[0x7f726980bd85]
collector[0x448e2e]
Caught signal 6 (SIGABRT): Aborted
/bootstrap.sh: line 94:    10 Aborted                 eval exec "$@"

Overview

With the eBPF enablement for s390x in Falco libs (https://github.com/hbrueckner/falcosecurity-libs/tree/mauro-pull-libs-20221115-s390x-bpf), I looked further into the custom eBPF probe, introduced with #670. There I experienced few issues that I'd like to discuss here.

My approach to get familiar and make the custom eBPF by replacing the Falco libs probe directly with the custom probe and using sinsp-example to install and capture events.

Oberservations

BPF_SUPPORTS_RAW_TRACEPOINTS

The BPF_SUPPORTS_RAW_TRACEPOINTS macro is explicitly undefined in line 12. While this has effects on the raw syscalls enter/exit (catch-all), this also disables the functionality of the sched_process_fork trace point. This tracepoint is actually required for s390x and aarch64 to capture newly created child process information as both architecture do not generate in the fork/vfork/cloneX system calls. See here for more details. Finally, my question here is whether collector and the upper layer require to obtain child process events?

Additional background: The raw tracepoints support is required to get access to the TP_PROTO arguments, in particular, to the struct task_struct of the parent and child processes. Those are required by the respective Falco libs filler (child and parent).

COLLECTOR_PROBEs

Looking into the those collector probes and reviewing related PR #670, I have few questions on that.

Instead of using the raw system call enter / exit events, those COLLECTOR_PROBE install BPF programs directly on the specified system call, e.g., chdir. This approach is very nice to focus on specific system calls rather hooking into every system call.

Trying those on s390x (including fixing few BPF verifier issues), resulted in this error message:

# ./libsinsp/examples/sinsp-example -b  driver/bpf/probe.o -j
-- Try to open: 'bpf' engine.
driver/bpf/probe.o
terminate called after throwing an instance of 'scap_open_exception'
  what():  PERF_EVENT_IOC_SET_BPF: Permission denied
Aborted (core dumped)

Looking into this issue, attaching the program to the syscall fails and further looking into the respective kernel sources, this issue seems to relate that the BPF program tries to data beyond the defined tracepoint structure (format).

For the chdir, the tracepoint structure format for its enter and exit is defined as follows:

# cat /sys/kernel/tracing/events/syscalls/sys_enter_chdir/format 
name: sys_enter_chdir
ID: 604
format:
        field:unsigned short common_type;       offset:0;       size:2; signed:0;
        field:unsigned char common_flags;       offset:2;       size:1; signed:0;
        field:unsigned char common_preempt_count;       offset:3;       size:1; signed:0;
        field:int common_pid;   offset:4;       size:4; signed:1;

        field:int __syscall_nr; offset:8;       size:4; signed:1;
        field:const char * filename;    offset:16;      size:8; signed:0;

print fmt: "filename: 0x%08lx", ((unsigned long)(REC->filename))

# cat /sys/kernel/tracing/events/syscalls/sys_exit_chdir/format 
name: sys_exit_chdir
ID: 603
format:
        field:unsigned short common_type;       offset:0;       size:2; signed:0;
        field:unsigned char common_flags;       offset:2;       size:1; signed:0;
        field:unsigned char common_preempt_count;       offset:3;       size:1; signed:0;
        field:int common_pid;   offset:4;       size:4; signed:1;

        field:int __syscall_nr; offset:8;       size:4; signed:1;
        field:long ret; offset:16;      size:8; signed:1;

print fmt: "0x%lx", REC->ret

This basically translates to those structure definition (there are similar the raw system call enter exit structures):

struct sys_enter_chdir_args
{
        __u64 pad;
        int __syscall_nr;
        const char *filename;

struct sys_exit_chdir_args
{
        __u64 pad;
        int __syscall_nr;
        long ret;
};

These structures are passed as context (ctx) to the enter_probe and exit_probe functions when the tracepoint is triggered. However, those tracepoint structure are syscall specific. The Falco libs implementation however deals with struct sys_enter_args and struct sys_exit_args. Looking specifically to the struct sys_exit_args:

struct sys_exit_args {
	__u64 pad;
	long id;
	long ret;
};

and comparing with the struct sys_exit_chdir_args above, accessingret (return value) would read beyond the struct sys_exit_chdir_args structure because the Falco probe and filler implementation works on struct sys_exit_args. This seems to explain the PERF_EVENT_IOC_SET_BPF: Permission denied from above.

With above background on syscall-specific structures vs. Falco implementation on struct sys_{enter,exit}_args, I tried to map the syscall-specific structure into the syscall enter structures, e.g. for chdir.

In probe_enter, perform the conversion before storing/stashing the arguments:

  //memcpy(stack_ctx.args, ctx->args, sizeof(ctx->args));
  switch (id)   /* maybe use sc_evt here..... */
  {
  case __NR_chdir: {
        struct sys_enter_chdir_args *chdir_args = (struct sys_enter_chdir_args *)ctx;
        stack_ctx.args[0] = (unsigned long)chdir_args->filename;
        break;
  }
  }

With that change, the original ctx is preserved and the stack_ctx is a struct sys_enter_args structure containing the arguments from the syscall specific tracepoint format. Also note that the ctx needs to be passed down the stack, especially to call_filler and fork kind of filler as they requires the context to perform the bpf_tail_call to subsequent filler implementation.

Using this approach (and limiting here for the enter event only), results in:

./libsinsp/examples/sinsp-example -b  driver/bpf/probe.o -j :
{"evt.args":"","evt.cpu":1,"evt.dir":">","evt.num":5,"evt.time":1669651102819446242,"evt.type":"chdir","proc.name":"bash","thread.tid":1056}

For the probe_exit function, this becomes more difficult because the system call specific tracepoint format does not match the generic system call exit args structure (int vs. long). Also Falco fillers use bpf_syscall_get_retval that operates on ctx and casts it to struct sys_exit_args.

Summary

Based on the above observations, and trying different variations with the custom probe, here is my summary for s390x:

• No raw tracepoint support (BPF_SUPPORTS_RAW_TRACEPOINTS) + COLLECTOR_PROBE has a system call specific context structure different from Falco assumption. (See above).

• No raw tracepoint support (BPF_SUPPORTS_RAW_TRACEPOINTS) + raw_syscalls/sys_{enter,exit} aka. COLLECTOR_LEGACY_PROBE: Works with some BPF verifier fixes but sched_process_fork filler does not work due to filler requirements to access the arguments of the tracepoint definition (= TP_PROTO). Without sched_process_fork, there will be no child process events on s390x (and aarch64). Also it has some performance effects because of generic system call probes (catch-all).

• Supporting raw tracepoints (BPF_SUPPORTS_RAW_TRACEPOINTS) + COLLECTOR_LEGACY_PROBE. Raw tracepoint support also enables sched_process_fork tracepoint. This is more aligned how falco libs works but, of course, has some performance effects because of generic system call probes (catch-all) -- potentially a bit less due to raw tracepont support.

I know this a quite large documentation on my observations and hope that the details help to understand it.

cc: @Molter73 @kcrane @robbycochran

The idea is to make CI pipelines support the "modern" probes. To achieve that:

• Build Falco with modern bpf enabled #1068
• Create a new testing pipeline for "modern" probes #1067
• (Optional for MVP) Modify the build pipeline to create and embed the new probe into an image #1019
• (Optional for MVP) To make solution more robust, include new probe into the regular probe delivery channels as well (so that Collector can try to fetch those probes via some alternative way)
• (Optional for MVP) Investigate possibility of using btfhub

See ROX-15070 for the higher-level details.

Figure out if an issue could be relevant for other platforms as well.

Hi,

Disclaimer: I have opened the same issue at stackrox/stackrox#3195 because I am not sure on which repository this should be tracked as here we have a area/collector label. Please close the one which is at the wrong location.

we are experiencing crashes in collector containers across all nodes in one of our OpenShift clusters.

Debug Log:

Collector Version: 3.9.0
OS: Red Hat Enterprise Linux CoreOS 49.84.202205050701-0 (Ootpa)
Kernel Version: 4.18.0-305.45.1.el8_4.x86_64
Starting StackRox Collector...
[I 20220926 112218 HostInfo.cpp:126] Hostname: '<redacted>'
[I 20220926 112218 CollectorConfig.cpp:119] User configured logLevel=debug
[I 20220926 112218 CollectorConfig.cpp:149] User configured collection-method=kernel_module
[I 20220926 112218 CollectorConfig.cpp:206] Afterglow is enabled
[D 20220926 112218 HostInfo.cpp:200] EFI directory exist, UEFI boot mode
[D 20220926 112218 HostInfo.h:100] identified kernel release: '4.18.0-305.45.1.el8_4.x86_64'
[D 20220926 112218 HostInfo.h:101] identified kernel version: '#1 SMP Wed Apr 6 13:48:37 EDT 2022'
[D 20220926 112218 HostInfo.cpp:297] SecureBoot status is 2
[D 20220926 112218 collector.cpp:254] Core dump not enabled
[I 20220926 112218 collector.cpp:302] Module version: 2.0.1
[I 20220926 112218 collector.cpp:329] Attempting to download kernel module - Candidate kernel versions:
[I 20220926 112218 collector.cpp:331] 4.18.0-305.45.1.el8_4.x86_64
[D 20220926 112218 GetKernelObject.cpp:148] Checking for existence of /kernel-modules/collector-4.18.0-305.45.1.el8_4.x86_64.ko.gz and /kernel-modules/collector-4.18.0-305.45.1.el8_4.x86_64.ko
[D 20220926 112218 GetKernelObject.cpp:151] Found existing compressed kernel object.
[I 20220926 112218 collector.cpp:262]
[I 20220926 112218 collector.cpp:263] This product uses kernel module and ebpf subcomponents licensed under the GNU
[I 20220926 112218 collector.cpp:264] GENERAL PURPOSE LICENSE Version 2 outlined in the /kernel-modules/LICENSE file.
[I 20220926 112218 collector.cpp:265] Source code for the kernel module and ebpf subcomponents is available upon
[I 20220926 112218 collector.cpp:266] request by contacting [email protected].
[I 20220926 112218 collector.cpp:267]
[I 20220926 112218 collector.cpp:162] Inserting kernel module /module/collector.ko with indefinite removal and retry if required.
[D 20220926 112218 collector.cpp:109] Kernel module arguments: s_syscallIds=26,27,56,57,246,247,248,249,94,95,14,15,156,157,216,217,222,223,4,5,22,23,12,13,154,155,172,173,214,215,230,231,282,283,288,289,292,293,96,97,182,183,218,219,224,225,16,186,234,194,195,192,193,200,201,198,199,36,37,18,19,184,185,220,221,226,227,-1 verbose=0 exclude_selfns=1 exclude_initns=1
[I 20220926 112218 collector.cpp:183] Done inserting kernel module /module/collector.ko.
[I 20220926 112218 collector.cpp:215] gRPC server=sensor.mcs-security.svc:443
[I 20220926 112218 CollectorService.cpp:50] Config: collection_method:kernel_module, useChiselCache:1, snapLen:0, scrape_interval:30, turn_off_scrape:0, hostname:<redacted>, logLevel:DEBUG
[I 20220926 112218 CollectorService.cpp:79] Network scrape interval set to 30 seconds
[I 20220926 112218 CollectorService.cpp:82] Waiting for GRPC server to become ready ...
[I 20220926 112218 CollectorService.cpp:87] GRPC server connectivity is successful
[D 20220926 112218 ConnTracker.cpp:314] ignored l4 protocol and port pairs
[D 20220926 112218 ConnTracker.cpp:316] udp/9
[I 20220926 112218 NetworkStatusNotifier.cpp:187] Started network status notifier.
[I 20220926 112218 NetworkStatusNotifier.cpp:203] Established network connection info stream.
[D 20220926 112218 SysdigService.cpp:262] Updating chisel and flushing chisel cache
[D 20220926 112218 SysdigService.cpp:263] New chisel:
args = {}
function on_event()
    return true
end
function on_init()
    filter = "not container.id = 'host'\n"
    chisel.set_filter(filter)
    return true
end

[I 20220926 112218 SignalServiceClient.cpp:43] Trying to establish GRPC stream for signals ...
[I 20220926 112218 SignalServiceClient.cpp:61] Successfully established GRPC stream for signals.
[D 20220926 112219 ConnScraper.cpp:406] Could not open process directory 1626873: No such file or directory
[D 20220926 112219 ConnScraper.cpp:406] Could not open process directory 1626877: No such file or directory
[W 20220926 112219 ProtoAllocator.h:41] Allocating a memory block on the heap for the arena, this is inefficient and usually avoidable
collector[0x44746d]
/lib64/libc.so.6(+0x4eb20)[0x7f8425ceeb20]
Caught signal 11 (SIGSEGV): Segmentation fault
/bootstrap.sh: line 94:    11 Segmentation fault      (core dumped) eval exec "$@"
Collector kernel module has already been loaded.
Removing so that collector can insert it at startup.

I am not sure how to debug this as all daemonSet containers experience this problem.

We are using StackRox 3.71.0. I have tried with collector images 3.9.0 and 3.11.0. Please reach out for any missing information.

The idea is to cover necessary bits to support BTF in the rest of Stackrox components. To achieve that:

• Add new collection method "modern_bpf" option to Helm charts #1020
• Update documentation stackrox/stackrox#6243
• (Optional for MVP) Add new mount for vmlinux BTF into the Collector DaemonSet

See ROX-15072 for the higher-level details.

Hi.
My AKS uses the kernel 5.4.0-1089-azure, so it needed the kernel module collector-5.4.0-1089-azure.ko.gz. But I haven't been seeing it here:

https://collector-modules.stackrox.io/612dd2ee06b660e728292de9393e18c81a88f347ec52a39207c5166b5302b656/b6745d795b8497aaf387843dc8aa07463c944d3ad67288389b754daaebea4b62/collector-5.4.0-1089-azure.ko.gz

However, other modules are there, e.g. collector-5.4.0-1086-azure.ko.gz

I think that collector-modules.starox.io is the right domain to look up new modules. My stackrox version is 3.0.62.0

I want to notify this behavior and request to upload the module mentioned. I appreciate any help you can provide.

The idea is to make "modern" Falco probe engine available for Collector. To achieve that:

• Syncing up our Falco fork with the latest changes #1016
• Update Collector probe loader to recognize and load new type of probes #1065
• Modern Falco probes supports only relatively new kernel versions (>= 5.8), the lower boundary is enforced via version checking. Figure out if it's possible to bump it down for lower versions where necessary features were backported #1037
• Customize modern Falco probes to load only a certain subset of tail-called programs to avoid unsupported syscalls problems #1044
• Make sure sinsp-example could be used for testing, and specify few tests for the new probe in isolation (optionally) #1043
• Implement standalone Collector for testing, and specify few tests for the new probe with Collector only (optionally)

See ROX-15068 for the higher-level details.

Following removal of kernel modules, collector will be running with all capabilities (as a privileged container.) To harden the collector process, we should drop all capabilities that we don't require.

• Identify required capabilities for running the various collector components
  • Kernel driver (eBPF/CORE bpf)
  • conn scraper
  • self-check process
• Implement a collection-method agnostic way of dropping capabilities once startup has concluded.
• Identity if any mounts are no longer needed

Temporary do not fallback to eBPF if modern probes are failed to load. Turns out that e.g. verifier error is not preventing Collector from continuing. Note, that in general it's good, just not for MVP.

Part of #1011

Prevent the stackrox tests (e2e as well as unit tests) from configuring a collector with kernel-module collection, to avoid test breakages while removing/disabling kernel module collection.

Note: it is possible they are using eBPF only, due to the default, but this should be verified.

Modern probe require /sys/kernel/btf/vmlinux present and available from the Collector container. Currently it is actually the case at least for OCP 4.12 -- a privileged container is able too read vmlinux file without any extra mounts. At the same time Collector is mounting host sysfs under /host/ path, including required vmlinux file. What needs to be done:

• Figure out, if we can rely on vmlinux present by K8S/OCP in a privileged container? Is it completely the same as what is mounted from the host via /host path?
• If those are not the same, or vmlinux file is not provided on all supported platforms, make sure it's included into the current sysfs mounts into the Collector container. If not, add it into the DaemonSet definition.
• If vmlinux will end up by the path different from /sys/kernel/btf/vmlinux or one of other standard locations (/boot/vmlinux..., /lib/modules/..., /usr/lib/modules/..., see libbpf for more details), Falco modern bpf engine has to be extended to pass load options for the probe. This would be probably a separate task from this one.

Part of #1015

our aggregated logging was exploding due to stackrox collector logging.. there where millions of lines like this the last days from 3/10 collector nodes..

actually to mitigate as fast as possible we had to delete stackrox

this is the error message which where present millions of times, this message appaered on 3 from 10 collector nodes about 6-10 times every millisecond..
[E 20221209 165715 ConnScraper.cpp:415] Could not determine network namespace: No such file or directory

it was deployed like this:
/bin/bash <(curl -fsSL https://raw.githubusercontent.com/stackrox/stackrox/master/scripts/quick-helm-install.sh)

on gke v1.24

stackrox/stackrox#5728

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/stackrox_stackrox/5728/pull-ci-stackrox-stackrox-master-gke-postgres-qa-e2e-tests/1650416371739136000

For MPV we would go with a probe embedded into the binary, a usual approach with libbpf skeletons. But it's important to understand if it could be changed in the future. Figure out how much efforts would that be to load the actual bpf probe from a not embedded file.

Part of #1014

stackrox/stackrox#5728

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/stackrox_stackrox/5728/pull-ci-stackrox-stackrox-master-gke-qa-e2e-tests/1650416371785273344

Hi!

I'm trying to run StackRox on 22.04.1 LTS (Jammy Jellyfish) with Kernel Linux 5.15.0-52-generic x86_64. This Kernel is supported according to KERNEL_VERSIONS.

Looks like Collector starts normally (I use quay.io/stackrox-io/collector-slim:3.72.1):

Collector Version: 3.11.1
OS: Ubuntu 22.04.1 LTS
Kernel Version: 5.15.0-52-generic
Starting StackRox Collector...
[I 20221118 071354 HostInfo.cpp:126] Hostname: 'dsworker2'
[I 20221118 071354 CollectorConfig.cpp:149] User configured collection-method=ebpf
[I 20221118 071354 CollectorConfig.cpp:206] Afterglow is enabled
[I 20221118 071354 collector.cpp:302] Module version: 2.1.0
[I 20221118 071354 collector.cpp:329] Attempting to download eBPF probe - Candidate kernel versions:
[I 20221118 071354 collector.cpp:331] 5.15.0-52-generic
[I 20221118 071354 GetKernelObject.cpp:180] Local storage does not contain collector-ebpf-5.15.0-52-generic.o
[I 20221118 071354 GetKernelObject.cpp:194] Successfully downloaded and decompressed /module/collector-ebpf.o
[I 20221118 071354 collector.cpp:262]
[I 20221118 071354 collector.cpp:263] This product uses kernel module and ebpf subcomponents licensed under the GNU
[I 20221118 071354 collector.cpp:264] GENERAL PURPOSE LICENSE Version 2 outlined in the /kernel-modules/LICENSE file.
[I 20221118 071354 collector.cpp:265] Source code for the kernel module and ebpf subcomponents is available upon
[I 20221118 071354 collector.cpp:266] request by contacting [email protected].
[I 20221118 071354 collector.cpp:267]
[I 20221118 071354 collector.cpp:215] gRPC server=sensor.stackrox.svc:443
[I 20221118 071354 CollectorService.cpp:50] Config: collection_method:ebpf, useChiselCache:1, snapLen:0, scrape_interval:30, turn_off_scrape:0, hostname:dsworker2, logLevel:INFO
[I 20221118 071354 CollectorService.cpp:79] Network scrape interval set to 30 seconds
[I 20221118 071354 CollectorService.cpp:82] Waiting for GRPC server to become ready ...
[I 20221118 071354 CollectorService.cpp:87] GRPC server connectivity is successful
[I 20221118 071354 NetworkStatusNotifier.cpp:168] Started network status notifier.
[I 20221118 071354 NetworkStatusNotifier.cpp:182] Established network connection info stream.
[I 20221118 071354 SignalServiceClient.cpp:43] Trying to establish GRPC stream for signals ...
[I 20221118 071354 SignalServiceClient.cpp:61] Successfully established GRPC stream for signals.

And that's it. In StackRox console I can see only this message: "No processes discovered. The selected deployment may not have running pods, or Collector may not be running in your cluster. It is recommended to check the logs for more information". Nothing happens.

And if I will check "System Health" status - everything is "green". Maybe you have some ideas what am I doing wrong? Thanks in advance!!!

stackrox/stackrox#5728

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/stackrox_stackrox/5728/pull-ci-stackrox-stackrox-master-ocp-4-12-qa-e2e-tests/1650416371944656896

The kernel module patches are no longer needed and largely unsuitable for upstreaming. They should be removed from the falco fork to ease maintenance and reduce friction during fork updates.

See https://docs.engineering.redhat.com/display/StackRox/Stackrox+changes+to+Falco+libs for details

Should be completed after #1103

Add new heuristic to Collector to figure out if BTF probe could be used. Two
main things to verify are:

• BTF support in the kernel. Could be done similar to bpftool via checking
  kernel configuration.

• BTF vmlinux is available. Similar to libbpf, usually /sys/kernel/btf/vmlinux

• BPF ringbuf maps are available. There are helpers in libbpf that could be helpful.

Part of #1011

Ansible playbooks should now only run eBPF & core_bpf and any special cases for kernel module collection should be removed from the tests themselves.

To monitor performance of the new modern probe, we should include its performance results in the baseline and associated PR comments.

Part of #1321

Falco modern bpf engine is not yet fully tested in the field. To make troubleshooting of potential issues easier improve Collector stacktrace to provide more information.

Part of #1011

Modify the build pipeline to embed a probe into the image. Here we don't have to bother yet what probe is that, for testing as close as possible to the reality we could e.g. build a vanilla falco modern probe with libsinsp example and use it. This will involve similar steps, e.g. actual building of something, fetching a new dependency (libbpf) etc.

Part of #1014

Removal from the main repository requires adjustment of:

• CollectionMethod storage definition (https://github.com/stackrox/stackrox/blob/master/proto/storage/cluster.proto#L55)
• Operator generated files
  • https://github.com/stackrox/stackrox/blob/master/operator/apis/platform/v1alpha1/securedcluster_types.go#L193
• Helm configuration
• UI
  • https://github.com/stackrox/stackrox/blob/master/ui/apps/platform/src/types/cluster.proto.ts#L52
• roxctl
  • https://github.com/stackrox/stackrox/blob/master/roxctl/sensor/generate/collection_method.go#L14

Must include documentation changes, especially in generated configurations/files.

Hello! The collector pods are bouncing between Running and CrashLoopBackOff due to the below error:

[2022-11-08T13:30:43.219349718+03:00 I 20221108 103043 collector.cpp:329] Attempting to download eBPF probe - Candidate kernel versions: 

[I 20221108 103043 collector.cpp:331] 5.15.35-5.el7.3.x86_64
[I 20221108 103043 GetKernelObject.cpp:180] Local storage does not contain collector-ebpf-5.15.35-5.el7.3.x86_64.o
[2022-11-08T13:30:43.440220207+03:00 I2022-11-08T13:30:43.440283536+03:00  20221108 103043 FileDownloader.cpp:316] Fail to download /module/collector-ebpf.o.gz - Failed writing body (0 != 10)
[I 20221108 103043 FileDownloader.cpp:318] HTTP Request failed with error code '404' - HTTP Body Response: not found
[I 20221108 103044 FileDownloader.cpp:316] Fail to download /module/collector-ebpf.o.gz - Failed writing body (0 != 10)
[I 2022112022-11-08T13:30:44.445460645+03:00 08 103044 FileDownloader.cpp:318] HTTP Request failed with error code '404' - HTTP Body Response: not found
......
[W 20221108 103112 FileDownloader.cpp:332] Failed to download /module/collector-ebpf.o.gz
[W 20221108 103112 GetKernelObject.cpp:183] Unable to download kernel object collector-ebpf-5.15.35-5.el7.3.x86_64.o to /module/collector-ebpf.o.gz
[W 2022112022-11-08T13:31:12.584992655+03:00 08 103112 collector.cpp:343] Error getting kernel object: collector-ebpf-5.15.35-5.el7.3.x86_64.o
[I 20221108 103112 collector.cpp:215] gRPC server=sensor.stackrox.svc:443
[2022-11-08T13:31:12.585667313+03:00 I2022-11-08T13:31:12.585678270+03:00  2022-11-08T13:31:12.585689123+03:00 20222022-11-08T13:31:12.585698916+03:00 112022-11-08T13:31:12.585708326+03:00 082022-11-08T13:31:12.585717352+03:00  2022-11-08T13:31:12.585726504+03:00 102022-11-08T13:31:12.585735847+03:00 312022-11-08T13:31:12.585745043+03:00 122022-11-08T13:31:12.585754222+03:00  2022-11-08T13:31:12.585763262+03:00 collector.cpp2022-11-08T13:31:12.585772387+03:00 :2022-11-08T13:31:12.585781692+03:00 3572022-11-08T13:31:12.585790958+03:00 ] 2022-11-08T13:31:12.585800229+03:00 Attempting to connect to GRPC server2022-11-08T13:31:12.585809299+03:00 

[2022-11-08T13:31:12.585830551+03:00 E2022-11-08T13:31:12.585839485+03:00  2022-11-08T13:31:12.585849167+03:00 20222022-11-08T13:31:12.585858503+03:00 112022-11-08T13:31:12.585868634+03:00 082022-11-08T13:31:12.585877519+03:00  2022-11-08T13:31:12.585886703+03:00 102022-11-08T13:31:12.585895845+03:00 312022-11-08T13:31:12.585904982+03:00 122022-11-08T13:31:12.585913900+03:00  2022-11-08T13:31:12.585922860+03:00 collector.cpp2022-11-08T13:31:12.585931770+03:00 :2022-11-08T13:31:12.585940638+03:00 3592022-11-08T13:31:12.585949507+03:00 ] 2022-11-08T13:31:12.585958593+03:00 Unable to connect to the GRPC server.2022-11-08T13:31:12.585967404+03:00 

[2022-11-08T13:31:12.586043197+03:00 F2022-11-08T13:31:12.586052670+03:00  2022-11-08T13:31:12.586062981+03:00 20222022-11-08T13:31:12.586072428+03:00 112022-11-08T13:31:12.586081771+03:00 082022-11-08T13:31:12.586090799+03:00  2022-11-08T13:31:12.586100030+03:00 102022-11-08T13:31:12.586125944+03:00 312022-11-08T13:31:12.586135654+03:00 122022-11-08T13:31:12.586145126+03:00  2022-11-08T13:31:12.586154678+03:00 collector.cpp2022-11-08T13:31:12.586178476+03:00 :368] No suitable kernel object downloaded

How can I troubleshoot? How cain i build own kernel-module?

This should cover the actual deletion of code related to kernel module collection from within collector.

• Kernel module loading
• Kernel module driver download logic
• HostHeuristics that force kernel module collection
• ForceKernelModules configuration option

In order to achieve #1017, the way collector handles driver candidates needs to be refactored in order to make it more flexible.

For the new module version (2.5.0 or 3.0) kernel module driver builds should be disabled/blocked via the block list.

stackrox/stackrox#5728

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/stackrox_stackrox/5728/pull-ci-stackrox-stackrox-master-gke-postgres-upgrade-tests/1650416371755913216

Teach Collector to load probes located inside the image. Here we don't have to
bother what type of probe is that yet, e.g. for testing it could be a regular
eBPF probe. The path for searching could be fixed, and later on aligned with CI
changes.

Part of #1011

Incorporate modern bpf engine into CO-RE kernel candidate, including open_modern_bpf and any other needed bits.

Part of #1008

Investigate integration test results for BTF aware probes.

Part of #1014

COS kernels have task_struct that differs from the vanilla one in regards how audit information is stored, loginuid is present in an intermediate structure audit_task_info. This mean regular CO-RE read will not pass verifier on COS kernels. To address it, check the field presence first, before extracting the value. Note, it's important that the condition is formulated this way -- the first branch should be an existing one, otherwise it will be removed as a dead code and no comparison will be made.

Part of #1008

After Collector has loaded the modern probe, it has to perform few simple checks that it works:

• Verify the Collector process is scraped and reported.
• Spin up a canary process and verify it is reported.
• Verify the Collector open connections are scraped and reported.
• Open a connection and verify it is reported.

The list above is a suggestion, any items that make sense could be added or removed (if too complicated). Similar idea could be e.g. that bpf maps contain expected records (configuration maps, tail called maps, rigbuf for events), but we go for it only if it could be implemented with relatively less efforts.

Part of #1011

Right now collector requires a gRPC server for it to run, when we need to debug an issue we have to either:

• Deploy it alongside a full StackRox deployment.
• Deploy it with a mock gRPC server that mocks sensor’s behaviour

It’d be great if we could directly run the collector binary without any external requirements, since it would make debugging quick changes a lot faster (compile the binary and run vs. compile, create the image, setup the environment, deploy the image with the change, etc.).

In order to get this done, the first requirement would be to make the grpc connection optional:

We also want to keep the behavior of running with the GRPC server to still be the default, so any changes need to be placed behind a feature flag, an environment variable like COLLECTOR_STANDALONE set to a none empty string would be nice. Here is an example on how we handle these flags:

The final step would be to adjust the signal handlers we currently have to either be ignored or print to stdout. I think the easiest way would be to create a new signal handler and set it here:

For a first implementation, a signal handler printing process information to stdout would be nice and the network signal can be directly turned off when running in standalone, leaving its implementation for a follow up.

We have instructions on running the collector-builder image as a development environment in our how to start guide, running the collector binary directly in that container without any companion containers is the end goal.

As with #1101, disable kernel module builds for the new module version in downstream builds.

stackrox/stackrox#5728

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/stackrox_stackrox/5728/pull-ci-stackrox-stackrox-master-ocp-4-10-qa-e2e-tests/1650416371885936640

Add "modern_bpf" as a collection method option for Collector configuration in Central helm charts.

Part of #1015

The kernel version on SLES-12 is deemed too old for eBPF so the heuristics fail over to kernel module collection.

https://github.com/stackrox/collector/actions/runs/4805861889/jobs/8557085684

Currently modern Falco probe have hard coded boundary for oldest supported
kernels, although it can as well work with any kernel that have back patched
needed things (BTF, ring buffer maps, trace programs).

Make modern bpf engine more flexible, verify required features instead of
simply checking the kernel version.

Part of #1008

This describes overall efforts about tightening up logging for core_bpf.

• We need to go through various scenarious and verify if the output information is correct and sufficient. E.g. now as far as I see even with core_bpf, Collector reports that something like collector-ebpf-<kernel-version>.o is used, which is not quite true. Another annoyance is that in the debug output where Collector print out it's config, collection method mentioned as a number from enum, the actual name would be better. I'm pretty sure there are plenty of such small rough edges, it would be valuable to once run most important cases (success, failure, heuristic failure, self-checks failure) and check the output.
• Add more information when possible. E.g. I've noticed that heuristics do not show an actual errno after probing. This leads to a system perfectly fine from support perspective report that BTF is not supported, when it was e.g. lacking sys_admin privileges or such. If not possible, mention that in the comments for the future us.
• Verify that core_bpf details are represented in the termination log when it makes sense.
• In the heuristics there are already hints a-la "it's not supported, try something else". We could add similar stuff if ebpf has failed, say "Hint: missing or not supported kernels issues could be avoided using core_bpf collection method".

Part of #1011

Update Falco fork to the latest changes and run the regular tests to verify regular probes are working.

Part of #1008

The idea is to teach Collector to use "modern" Falco probe engine in a backward compatible way. To achieve that:

• Implement new way of loading probes located inside the image. #1017
• Implement heuristics for Collector to find out if BTF is supported. In the simplest scenario it could be a kernel version check, but ideally more elaborated feature verification is needed (in case of backported functionality). #1018
• Implement fallback mechanism, to let Collector switch back to regular probes if it fails to load the new one. The definition of "failed to load" means simply an error from kernel, but in theory could be extended to more runtime validation that everything is in place. #1066
• Display failed BTF load via Collector status, similar to cases when no probes could be loaded. #1077
• (Optional for MVP) Implement new way of getting probes via searching in the container itself, since BTF aware probes are going to be embedded into the container. This has to be implemented flexible though, as there still could be cases when it's beneficial to load even BTF aware probe, and the original probes have to be loaded as usual anyway.
• (Optional for MVP) Try to refactor the loading logic to make it more flexible, having in mind potentially delegating probe loading into the kernel to other components.

See ROX-15069 for the higher-level details.

For flexibility and operational purposes it's necessary to be able to run Falco modern probes with only a subset of tail-called programs loaded. This could be relatively easy achieved via customizing the build process to exclude certain BPF programs, and reduce the error to a warning, when they're not found by a name later on.

Part of #1008

Falco library features two sets of tests for probes:

• e2e tests based on sinsp-example. They allow to simulate a variety of low-level events relatively easy, don't support modern probes directly.
• drivers test suite. More basic unit-tests following the strategy "fire a syscall and verify what the driver has captured", supports modern probes as well.

Figure out of those two cover the functionality of modern probes good enough, and if not, extend to improve that and include into the testing pipeline.

Part of #1008

There was a significant amount of friction and lost time when updating the Falco fork. To combat this, it should be reorganized in such a way as to make updates easier and make it easy to remove patches that are unneeded or have been upstreamed.

The proposed procedure is:

• make the main branch track upstream/master
• extract stackrox patches onto module version branches (e.g. release-2.4 or similar) on top of the main branch.
• Current master, and previously tagged module versions will remain until they are out of support (up to module version 2.4.0, collector 3.14.1, ACS 4.0)

This can be simulated on separate branches before committing to the approach, but should be done before existing patches are removed or modified.