Error when run ksniff in privileged mode about ksniff HOT 15 CLOSED

@bostrt I've put up a PR here - #86

from ksniff.

Hey @ffais, I made some changes related to privileged pod usage in ksniff v1.5.0. Sorry to hear something isn't working right for you.

Can you make sure that docker is up-and-running on the node aks-d8sv3-38575711-vmss000001 and you can execute docker commands normally (just docker ps would be fine).

Are you seeing this issue with ksniff every time you try to run it?

Lastly, does the ksniff pod hang around after the failure so you can try getting a shell into it for some testing?

# kubectl exec -it ksniff-5pbv6 -n <NAMESPACE> /bin/sh
$ docker --host unix:///host/var/run/docker.sock ps 
$ ls -la /host/var/run/docker.sock
$ ps aux | grep dockerd

from ksniff.

Hi @bostrt,

docker is up-and-running on every node, this is the output of docker ps command:

aks-d8sv3-38575711-vmss000001:~$ docker ps
56db633f5f8e        mcr.microsoft.com/oss/kubernetes/pause:1.3.1                "/pause"                 2 hours ago         Up 2 hours                              k8s_POD_acid-minimal-cluster-0_global_12453477-d3f4-4d99-9eee-a63529c39142_0
07d7db3873f8        nicolaka/netshoot                                           "/bin/bash"              3 hours ago         Up 3 hours                              k8s_tmp-shell_tmp-shell_global_52ce5d23-de5f-4fa2-ab55-d06b4a0e4a72_0

And this is the output of ksniff pods:

/ # kubectl -n global exec -it ksniff-4m957 -c ksniff-privileged /bin/sh
/ # docker --host unix:///host/var/run/docker.sock ps
Cannot connect to the Docker daemon at unix:///host/var/run/docker.sock. Is the docker daemon running?
/ # ls -la /host/var/run/docker.sock
ls: /host/var/run/docker.sock: No such file or directory
/ # ps aux | grep dockerd
 3327 root      7h53 /usr/bin/dockerd -H fd:// --storage-driver=overlay2 --bip=172.17.0.1/16
19490 root      0:00 grep dockerd
/ # 

Sometimes i see this error:

INFO[0000] waiting for pod successful startup           
INFO[0012] pod: 'ksniff-bc2kz' created successfully on node: 'aks-d8sv3-38575711-vmss000002' 
INFO[0012] spawning wireshark!                          
INFO[0012] starting remote sniffing using privileged pod 
INFO[0012] executing command: '[docker --host unix:///host/var/run/docker.sock run --rm --name=ksniff-container-nPNThiKe --net=container:ab652c801a2616a5f47603054cdfc579eb4a403f4597c4b08f146f58d0c71ee9 maintained/tcpdump -i any -U -w - ]' on container: 'istio-proxy', pod: 'ksniff-bc2kz', namespace: 'global' 
INFO[0013] command: '[docker --host unix:///host/var/run/docker.sock run --rm --name=ksniff-container-nPNThiKe --net=container:ab652c801a2616a5f47603054cdfc579eb4a403f4597c4b08f146f58d0c71ee9 maintained/tcpdump -i any -U -w - ]' executing successfully exitCode: '126', stdErr :'' 
INFO[0013] remote sniffing using privileged pod completed 

EDIT: I found the problem, on AKS clusters docker socket file is located at /run/docker.sock.
Changing this line:

from ksniff.

EDIT: I found the problem, on AKS clusters docker socket file is located at /run/docker.sock.

Good find! In most Linux systems /var/run is a symlink to /run (per FHS) so interesting that AKS Nodes don't have that!

AFAIK, It should be safe to use /run/docker.sock (referencing the same FHS link above). I'll do some searching around to make sure it won't introduce more problems. Or, if we should be checking for docker.sock in both locations.

from ksniff.

Nice find and research! maybe we can dynamically locate the docker sock path? might also be future proof and a more robust solution.

from ksniff.

About the only way I can think of dynamically locating the docker socket path would be using lsof, netstat, or ss, then filtering one of those commands' output for dockerd.
Another (less future proof) option might be to switch the docker command in ksniff to use /run/docker.sock since that should be globally default and then add yet another ksniff flag to specify a custom docker socket path to support odd configurations.

from ksniff.

would a try/catch pattern work well here? check if /var/run/docker.sock exists, otherwise check if /run/docker.sock exists, use the first match

from ksniff.

@jarojasm95 imo that's the best way. Using a dynamic method like I mentioned in #82 (comment) would add more dependencies to the images used by ksniff and benefit might not be as great.

from ksniff.

when the issue expected to be resolved ?

from ksniff.

@halradaideh I'll be reviewing the PR soon.

In the meantime, you may try rolling back to ksniff v1.4.2 to see if that helps.

from ksniff.

Hi all, #90 has been merged which includes support for --socket path which will address this problem. I'm closing this but please reopen if there's anything that was missed.

Note that the fix hasn't been released yet but will be upcoming after other things get merged and some more testing happens.

from ksniff.

Same issue at GKE.

@bostrt Is there a workaroud?

Update:
compiled kubectl-sniff on masterand used this on GKE

kubectl sniff ... -p --socket /var/run/docker.sock

from ksniff.

Hi @FaKod, Unforutnately, I don't have a reference GKE cluster. Can you make sure that /var/run/docker.sock is indeed where the socket lives? If it does live there, let me know and I'll open up a new issue (or feel free to go ahead and open a new one) to focus on GKE.

from ksniff.

@bostrt Works with current version (master) and the option --socket /var/run/docker.sock

from ksniff.

Thanks @FaKod!

from ksniff.