Overview

Hi all!

As the development team behind host-scanner, we greatly value the feedback and ideas of our users. Your contributions have been instrumental in shaping the tool and making it more efficient and user-friendly. As we gear up for the next major update, we would like to extend an invitation to all of you to share your thoughts on possible improvements and features you would like to see in the upcoming release.

Your involvement is crucial to making host-scanner an even more powerful and efficient tool. Whether you are a seasoned developer or a new user, your perspective matters, and we welcome everyone to participate. So, feel free to share your thoughts here!

Which kind of new functionality you would like to see in the upcoming release?
Is the current documentation self-explanatory?

Thank you for being part of our community!

Best regards,
Kubescape Development Team

Hi Team,

I am trying to scan Azure AKS Node by deploying a host scanner. But host scanner pods are continuously restarting with probe-related errors.

Azure AKS Version - v1.25.6

I've used below command to deploy the host scanner in azure aks:

kubectl apply -f https://raw.githubusercontent.com/kubescape/kubescape/master/core/pkg/hostsensorutils/hostsensor.yaml

Error:

shaik [ ~ ]$ kubectl get events -n kubescape | grep -i probe 15s Warning Unhealthy pod/host-scanner-5rmdl Startup probe failed: Get "http://10.224.0.196:7888/readyz": dial tcp 10.224.0.196:7888: connect: connection refused 2m28s Warning Unhealthy pod/host-scanner-8chcc Startup probe failed: Get "http://10.224.0.144:7888/readyz": dial tcp 10.224.0.144:7888: connect: connection refused 57s Warning Unhealthy pod/host-scanner-bnmjz Startup probe failed: Get "http://10.224.0.171:7888/readyz": dial tcp 10.224.0.171:7888: connect: connection refused 3m13s Warning Unhealthy pod/host-scanner-c7lqd Startup probe failed: Get "http://10.224.0.151:7888/readyz": dial tcp 10.224.0.151:7888: connect: connection refused 2m29s Warning Unhealthy pod/host-scanner-fg7d8 Startup probe failed: Get "http://10.224.1.108:7888/readyz": dial tcp 10.224.1.108:7888: connect: connection refused 90s Warning Unhealthy pod/host-scanner-fh94r Startup probe failed: Get "http://10.224.1.31:7888/readyz": dial tcp 10.224.1.31:7888: connect: connection refused 115s Warning Unhealthy pod/host-scanner-gfsvm Startup probe failed: Get "http://10.224.1.43:7888/readyz": dial tcp 10.224.1.43:7888: connect: connection refused 16s Warning Unhealthy pod/host-scanner-jft8x Startup probe failed: Get "http://10.224.1.139:7888/readyz": dial tcp 10.224.1.139:7888: connect: connection refused 57s Warning Unhealthy pod/host-scanner-jjxm6 Startup probe failed: Get "http://10.224.1.74:7888/readyz": dial tcp 10.224.1.74:7888: connect: connection refused 2m18s Warning Unhealthy pod/host-scanner-mpdmz Startup probe failed: Get "http://10.224.1.11:7888/readyz": dial tcp 10.224.1.11:7888: connect: connection refused 2m51s Warning Unhealthy pod/host-scanner-nbpzf Startup probe failed: Get "http://10.224.0.118:7888/readyz": dial tcp 10.224.0.118:7888: connect: connection refused 2m28s Warning Unhealthy pod/host-scanner-nn6w8 Startup probe failed: Get "http://10.224.1.3:7888/readyz": dial tcp 10.224.1.3:7888: connect: connection refused 2m51s Warning Unhealthy pod/host-scanner-qp5f5 Startup probe failed: Get "http://10.224.0.251:7888/readyz": dial tcp 10.224.0.251:7888: connect: connection refused 15s Warning Unhealthy pod/host-scanner-wmxhn Startup probe failed: Get "http://10.224.0.226:7888/readyz": dial tcp 10.224.0.226:7888: connect: connection refused 58s Warning Unhealthy pod/host-scanner-wv4tg Startup probe failed: Get "http://10.224.1.157:7888/readyz": dial tcp 10.224.1.157:7888: connect: connection refused 7m20s Warning Unhealthy pod/host-scanner-z49cl Startup probe failed: Get "http://10.224.0.252:7888/readyz": dial tcp 10.224.0.252:7888: connect: connection refused

Description

Execution of host-scanner is stuck when scanning Talos based clusters.

Environment

OS: Talos Linux
Version: v1.4.0
Kubernetes version: v1.26.3

Steps To Reproduce

Steps to reproduce the behavior:

1. Run the following command kubescape scan framework cis-v1.23-t1.0.1 --enable-host-scan against a talos based kubernetes cluster. At this point we should be stuck from kubescape output with this log:

[info] Kubescape scanner starting
[debug] Kubescape Cloud URLs. api: api.armosec.io; auth: auth.armosec.io; report: report.armo.cloud; UI: cloud.armosec.io
[info] Installing host scanner
[debug] The host scanner is a DaemonSet that runs on each node in the cluster. The DaemonSet will be running in it's own namespace and will be deleted once the scan is completed. If you do not wish to install the host scanner, please run the scan without the --enable-host-scan flag.
[info] Downloading/Loading policy definitions
Downloading framework. framework: cis-v1.23-t1.0.1
[success] Downloaded/Loaded policy
[info] Accessing Kubernetes objects
[success] Accessed to Kubernetes objects
[info] Requesting Host scanner data
[debug] Collecting host scanner resources
[debug] Accessing host scanner
[info] Host scanner version : v1.0.54

2. Run the following one-liner for i in controlplaneinfo cniinfo kernelversion kubeletinfo kubeproxyinfo cloudproviderinfo osrelease openedports linuxsecurityhardening version; do echo $i && wget -qO- http://localhost:7888/$i; done.
3. Check for logs:

{"level":"info","ts":"2023-04-26T14:19:55Z","msg":"Listening...","port":7888}
{"level":"warn","ts":"2023-04-26T14:50:46Z","msg":"failed to MakeHostFileInfo","path":"/etc/kubernetes/manifests/kube-apiserver.yaml","error":"stat /host_fs/etc/kubernetes/manifests/kube-apiserver.yaml: no such file or directory","in":"makeProcessInfoVerbose","path":"/etc/kubernetes/manifests/kube-apiserver.yaml"}
{"level":"warn","ts":"2023-04-26T14:50:47Z","msg":"failed to MakeHostFileInfo","path":"/etc/kubernetes/manifests/kube-controller-manager.yaml","error":"stat /host_fs/etc/kubernetes/manifests/kube-controller-manager.yaml: no such file or directory","in":"makeProcessInfoVerbose","path":"/etc/kubernetes/manifests/kube-controller-manager.yaml"}
{"level":"warn","ts":"2023-04-26T14:50:47Z","msg":"failed to MakeHostFileInfo","path":"/etc/kubernetes/controller-manager.conf","error":"stat /host_fs/etc/kubernetes/controller-manager.conf: no such file or directory","in":"makeProcessInfoVerbose","path":"/etc/kubernetes/controller-manager.conf"}
{"level":"warn","ts":"2023-04-26T14:50:48Z","msg":"failed to MakeHostFileInfo","path":"/etc/kubernetes/manifests/kube-scheduler.yaml","error":"stat /host_fs/etc/kubernetes/manifests/kube-scheduler.yaml: no such file or directory","in":"makeProcessInfoVerbose","path":"/etc/kubernetes/manifests/kube-scheduler.yaml"}
{"level":"warn","ts":"2023-04-26T14:50:48Z","msg":"failed to MakeHostFileInfo","path":"/etc/kubernetes/scheduler.conf","error":"stat /host_fs/etc/kubernetes/scheduler.conf: no such file or directory","in":"makeProcessInfoVerbose","path":"/etc/kubernetes/scheduler.conf"}
{"level":"warn","ts":"2023-04-26T14:50:48Z","msg":"failed to MakeHostFileInfo","path":"/etc/kubernetes/manifests/etcd.yaml","error":"stat /host_fs/etc/kubernetes/manifests/etcd.yaml: no such file or directory","in":"SenseControlPlaneInfo","component":"EtcdConfigFile"}
{"level":"warn","ts":"2023-04-26T14:50:48Z","msg":"failed to MakeHostFileInfo","path":"/etc/kubernetes/admin.conf","error":"stat /host_fs/etc/kubernetes/admin.conf: no such file or directory","in":"SenseControlPlaneInfo","component":"AdminConfigFile"}
{"level":"warn","ts":"2023-04-26T14:50:48Z","msg":"MakeHostFileInfo","error":"open /host_fs/etc/passwd: no such file or directory"}
{"level":"warn","ts":"2023-04-26T14:50:48Z","msg":"MakeHostFileInfo","error":"open /host_fs/etc/group: no such file or directory"}
{"level":"warn","ts":"2023-04-26T14:50:48Z","msg":"MakeHostFileInfo","error":"open /host_fs/etc/passwd: no such file or directory"}
{"level":"warn","ts":"2023-04-26T14:50:48Z","msg":"MakeHostFileInfo","error":"open /host_fs/etc/group: no such file or directory"}
{"level":"warn","ts":"2023-04-26T14:50:48Z","msg":"MakeHostFileInfo","error":"open /host_fs/etc/passwd: no such file or directory"}
{"level":"warn","ts":"2023-04-26T14:50:48Z","msg":"MakeHostFileInfo","error":"open /host_fs/etc/group: no such file or directory"}
{"level":"warn","ts":"2023-04-26T14:50:51Z","msg":"getCNIConfigDirFromConfig- Failed to Call ReadDir","configDirPath":"/host_fs/etc/containerd/containerd.conf.d","error":"open /host_fs/etc/containerd/containerd.conf.d: no such file or directory"}
{"level":"warn","ts":"2023-04-26T14:50:51Z","msg":"MakeHostFileInfo","error":"open /host_fs/etc/passwd: no such file or directory"}
{"level":"warn","ts":"2023-04-26T14:50:51Z","msg":"MakeHostFileInfo","error":"open /host_fs/etc/group: no such file or directory"}
{"level":"warn","ts":"2023-04-26T14:50:51Z","msg":"MakeHostFileInfo","error":"open /host_fs/etc/passwd: no such file or directory"}
{"level":"warn","ts":"2023-04-26T14:50:51Z","msg":"MakeHostFileInfo","error":"open /host_fs/etc/group: no such file or directory"}
{"level":"warn","ts":"2023-04-26T14:50:51Z","msg":"MakeHostFileInfo","error":"open /host_fs/etc/passwd: no such file or directory"}
{"level":"warn","ts":"2023-04-26T14:50:51Z","msg":"MakeHostFileInfo","error":"open /host_fs/etc/group: no such file or directory"}
{"level":"warn","ts":"2023-04-26T14:50:52Z","msg":"getCNIName- Failed to locate process for cni","cni name":"aws","error":"no process with given suffix found"}
{"level":"warn","ts":"2023-04-26T14:50:54Z","msg":"getCNIName- Failed to locate process for cni","cni name":"Flannel","error":"no process with given suffix found"}
{"level":"warn","ts":"2023-04-26T14:50:55Z","msg":"getCNIName- Failed to locate process for cni","cni name":"Cilium","error":"no process with given suffix found"}
{"level":"warn","ts":"2023-04-26T14:50:57Z","msg":"getCNIName- Failed to locate process for cni","cni name":"WeaveNet","error":"no process with given suffix found"}
{"level":"warn","ts":"2023-04-26T14:50:58Z","msg":"getCNIName- Failed to locate process for cni","cni name":"Kindnet","error":"no process with given suffix found"}
{"level":"warn","ts":"2023-04-26T14:50:59Z","msg":"getCNIName- Failed to locate process for cni","cni name":"Multus","error":"no process with given suffix found"}

Expected behavior

host-scanner should be able to read information from the OS.

Actual Behavior

host-scanner is unable to retrieve data from /kubeletinfo endpoint.

Additional context

Thanks to @bnason for reporting the bug. We had a conversation on slack here: https://cloud-native.slack.com/archives/C04EY3ZF9GE/p1682517113961639

Overview

Why "hostipc:true" is needed?

Problem

I didn't see host-scanner collecting IPC resource information from the host. So, why is 'HostIPC: true' needed?

Solution

Alternatives

Additional context

Describe the bug

The DaeonSet is successfully deployed, but every time, the hostscanner stays in CrashLoopBackOff at start.

The Pod log contains only:

exec ./kube-host-sensor: exec format error

I downloaded the image quay.io/kubescape/host-scanner:v1.0.35 to check, and it's the same.

Environment

OS: Linux 5.15.0-1021-oracle #27-Ubuntu SMP Fri Oct 14 20:04:20 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
Linux: Ubuntu 22.04.1 LTS
Version: v2.0.172

Steps To Reproduce

Steps to reproduce the behavior:

1. Run a hostscanner with a kube containing an ARM64 host: kubescape scan --enable-host-scan

Expected behavior

All work right without error

Actual Behavior

The Pod stays in CrashLoopBackOff

Additional context

I made these command to investigate:

# Retrieve the image on the host
$ docker pull quay.io/kubescape/host-scanner:v1.0.35

# The image was downloaded without any problem
$ docker image save quay.io/kubescape/host-scanner:v1.0.35 | tar xv

2dc561b874360f41fbb24505e750635afbfc3da75191cb226690667094ab0f61.json
54b1afaa52eb119c0d8eb6c8298ba785cb06134770f40a5a1656fc77ebf32cc7/
54b1afaa52eb119c0d8eb6c8298ba785cb06134770f40a5a1656fc77ebf32cc7/VERSION
54b1afaa52eb119c0d8eb6c8298ba785cb06134770f40a5a1656fc77ebf32cc7/json
54b1afaa52eb119c0d8eb6c8298ba785cb06134770f40a5a1656fc77ebf32cc7/layer.tar
72528bc5c69b9f4f8dd8d685d1ffa658a69a9eb815aeea6f9f54a3a070b7b556/
72528bc5c69b9f4f8dd8d685d1ffa658a69a9eb815aeea6f9f54a3a070b7b556/VERSION
72528bc5c69b9f4f8dd8d685d1ffa658a69a9eb815aeea6f9f54a3a070b7b556/json
72528bc5c69b9f4f8dd8d685d1ffa658a69a9eb815aeea6f9f54a3a070b7b556/layer.tar
manifest.json
repositories

# Extracted the first layer
$ cd 72528bc5c69b9f4f8dd8d685d1ffa658a69a9eb815aeea6f9f54a3a070b7b556
$ tar xf layer.tar

# Checked the architecture of the binary
$ file bin/busybox

bin/busybox: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, stripped

$cd ..
# Extracted the second layer
$ cd 54b1afaa52eb119c0d8eb6c8298ba785cb06134770f40a5a1656fc77ebf32cc7
$ tar xvf layer.tar

kube-host-sensor

# Checked the architecture of the binary
$ file kube-host-sensor

kube-host-sensor: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=_MDqQEPq5W5_wU8AWG1H/JWTZrNxTFKoqV5w8mZW0/Rd383K46e_qWKJyGf2Pu/NjUPnzF3NTah5VQVrhmj, stripped

And on kube-host-sensor, there is a problem, the binary is for x86-64…

After some search, I found the source of the problem, I think, directly inside the Dockerfile:

You specify to build for GOARCH=amd64, even for building ARM64 image

I'm running Kubescape with --enable-host-scan against the Azure AKS cluster default node image and getting error:

{"level":"error","ts":"2022-06-30T13:10:37.3024482Z","msg":"Request failed","method":"GET","requestURI":"/kubeletConfigurations","remoteAddr":"10.1.0.28:33890","requestStartTime":"2022-06-30T13:10:37.046321419Z","Request body":"","HTTP status":500,"Response size":104,"Response body":"failed to sense kubelet conf: in SenseKubeletConfigurations failed to find kubelet config File location\n"}
[negroni] 2022-06-30T13:10:37Z | 500 |   256.705187ms | ...westeurope.azmk8s.io:443 | GET /kubeletConfigurations

I understand that this is because the --config parameter was not used when running the kubelet process. But as this parameter is optional and all the other parameters can override configuration from this file, other parameters should be also considered in this endpoint and merged with the configuration file (if present).

Overview

At this moment, it is difficult to figure out which version of this repo corresponds to the tagged image available on quay.io.

Problem

I've noticed this when working on the client-side calling the host-scanner API in a package in kubescape (kubescape/core/pkg/hostsensorutils): routes are hard-coded there and we don't really know which is which.

Solution

1. Tag the host-scanner repo
2. Build images according to the tag

Alternatives

Additional context