Comments (15)
Thanks for the detailed issue,
Let's try the following:
-
Verify that the static tcpdump is running correctly on the remote pod: when getting the error, exec into the remote pod and verify that it's running using "ps aux". If not, exec into the container and manually try to run /tmp/static-tcpdump - do you get an error? which one?
-
If the static tcpdump run correctly, try to run ksniff with output file flag (using "-o") this way we should have a pcap file instead of "streaming" the results to wireshark. sniff for couple seconds and try to open the file with wireshark - do you see the capture? same error?
Eldad
from ksniff.
Hello Eldad, thanks for quick reply,
for first question, I don't find tcpdump executable in /tmp
🚀 kc -n epc1 exec -ti mme-0 -- bash
root@mme-0:/opt/c3po/mme/bin# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 4504 704 ? Ss+ 19:28 0:00 /bin/sh -c /opt
root 7 0.0 0.0 18036 3000 ? S+ 19:28 0:00 /bin/bash /opt/
root 35 0.1 0.0 3699596 45064 ? Sl+ 19:28 0:11 ./vb_acc
root 82 0.0 0.0 18244 3296 ? Ss+ 19:45 0:00 bash
root 210 0.0 0.0 18244 3268 ? Ss 22:05 0:00 bash
root 220 0.0 0.0 34424 2836 ? R+ 22:05 0:00 ps aux
root@mme-0:/opt/c3po/mme/bin# cd /tmp/
root@mme-0:/tmp# ls
root@mme-0:/tmp# ls -al
total 8
drwxrwxrwt 2 root root 4096 Sep 26 16:23 .
drwxr-xr-x 52 root root 4096 Nov 28 19:28 ..
And I have print output log to see, it shows:
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "exec: \"/tmp/static-tcpdump\": stat /tmp/static-tcpdump: no such file or directory"
Seems sniff doesn't send tcpdump executable into the pod.
from ksniff.
If for some reason ksniff failed to upload tcpdump to the remote container the capture will fail.
Are you sure you executed into the container "mme" of the pod "mme-0" (I can't see that from kubectl output)?
Is uploading files using kubectl works fine with that pod? (ksniff upload files in a very similar way)
Edit: now I see you only have one container in that pod, so the answer to my first question is yes :)
from ksniff.
Yes, I can ensure I am in the mme container.
And I tried to upload tcpdump manually, it works, please help me with checking following log :)
🚀 kubectl -n epc1 cp ~/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump mme-0:/tmp/
🚀 kubectl -n epc1 exec -ti mme-0 -- bash
root@mme-0:/opt/c3po/mme/bin# cd /tmp
root@mme-0:/tmp# ls
static-tcpdump
root@mme-0:/tmp# ./static-tcpdump
static-tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
23:10:30.256254 IP 119.0.0.10.36412 > 10.244.2.13.36412: sctp (1) [HB REQ]
23:10:30.256415 IP 10.244.2.13.36412 > 119.0.0.10.36412: sctp (1) [HB ACK]
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
root@mme-0:/tmp#
I think the problem is why sniff can't upload to the pod now ;)
from ksniff.
Please try ksniff after manually uploading tcpdump - is it working now? can you see packets in wireshark?
I think it's pretty clear now that ksniff fail to upload tcpdump (and doesn't detect that) to the pod.
I'm not on Mac right now, my guess is that it something related to ksniff being used on Mac, I'll dig into it and update.
In the meanwhile, can you please try to manually specify the local path for the static tcpdump file using the "-l" flag?
please copy it from "~/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump" to other path on you machine and than use ksniff with "-l STATIC_TCPDUMP_PATH"
( make sure you delete tcpdump from the remote pod before trying)
from ksniff.
Yes, after I uploaded tcpdump manually, wireshark can work perfectly.
and I have change the tcpdump location do the executable path assignment:
🚀 kc -n epc1 exec -ti mme-0 -- bash
root@mme-0:/opt/c3po/mme/bin# rm /tmp/static-tcpdump
root@mme-0:/opt/c3po/mme/bin# exit
🚀 kc sniff mme-0 -n epc1 -l assets/static-tcpdump
INFO[0000] using tcpdump path at: 'assets/static-tcpdump'
INFO[0000] no container specified, taking first container we found in pod.
INFO[0000] selected container: 'mme'
INFO[0000] sniffing on pod: 'mme-0' [namespace: 'epc1', container: 'mme', filter: '']
INFO[0000] checking for static tcpdump binary on: '/tmp/static-tcpdump'
INFO[0000] couldn't find static tcpdump binary on: '/tmp/static-tcpdump', starting to upload
INFO[0000] tcpdump uploaded successfully
INFO[0000] spawning wireshark!
but the error message same as first time. I think it's some bug make macOS can't upload executable into pod.
from ksniff.
OK, thanks for the detailed info.
I'll try again on macOS soon and update with my findings.
For now, please use the workaround (manually uploading tcpdump using kubectl).
from ksniff.
Sure, thanks so much for your help 👍
from ksniff.
I've tried similar conditions on macOS, couldn't reproduce this issue.
To better investigate it I created a new version of ksniff (1.2.0) which include a verbose flag and better verification of the upload process.
Please manually upgrade your version (i'll update krew index later) by downloading the zip (https://github.com/eldadru/ksniff/releases/download/v1.2.0/ksniff.zip) and unpacking the darwin version instead of the existing version on krew folder ( ~/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/)
Execute ksniff in the same way you did and add "-v":
kc sniff mme-0 -n epc1 -v
Please post the output here.
Don't forget to make sure there is no leftover tcpdump binary on the remote container.
from ksniff.
Hello Eldad,
Here is the log, still can't work, please have a check.
INFO[0000] running in verbose mode
DEBU[0000] searching for tcpdump binary using lookup list: '[ /Users/aweimeow/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump /usr/local/bin/static-tcpdump /Users/aweimeow/.kube/plugin/sniff/static-tcpdump]'
DEBU[0000] tcpdump binary was not found at: ''
DEBU[0000] tcpdump binary found at: '/Users/aweimeow/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump'
INFO[0000] using tcpdump path at: '/Users/aweimeow/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump'
DEBU[0000] pod 'mme-0' status: 'Running'
INFO[0000] no container specified, taking first container we found in pod.
INFO[0000] selected container: 'mme'
INFO[0000] sniffing on pod: 'mme-0' [namespace: 'epc1', container: 'mme', filter: '']
INFO[0000] checking for static tcpdump binary on: '/tmp/static-tcpdump'
DEBU[0000] checked for tcpdump on remote pod: exit-code: '2', stdout: '', stderr: 'ls: cannot access '/tmp/static-tcpdump': No such file or directory
'
INFO[0000] couldn't find static tcpdump binary on: '/tmp/static-tcpdump', starting to upload
DEBU[0000] uploading file from: '/Users/aweimeow/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump' to '/tmp/static-tcpdump'
DEBU[0000] read '/Users/aweimeow/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump' to memory, file size: '2642872'
DEBU[0000] formatted '/Users/aweimeow/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump' as tar, tar size: '2644480'
DEBU[0000] done uploading file, exitCode: '0', stdOut: '', stdErr: 'tar: Removing leading `/' from member names
'
INFO[0000] verifying tcpdump uploaded successfully
DEBU[0000] checked for tcpdump on remote pod: exit-code: '2', stdout: '', stderr: 'ls: cannot access '/tmp/static-tcpdump': No such file or directory
'
ERRO[0000] failed to upload tcpdump.
Error: couldn't locate tcpdump on pod after upload done
from ksniff.
I am guessing if my kuberctl version and the way I control cluster (remotely) cause this bug, because according your log have verifying tcpdump uploaded successfully
, then it indicated the tcpdump script should exist on /tmp/
. But I checked with it, there is an empty directory.
Might be bug of other software?
from ksniff.
Very weird, according to the verbose output, everything looks fine but the file isn't on the remote path.
I used exactly the same environment as yours and it still works for me:
➜ ~ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:25:46Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Is there anything unusual about your kubectl/cluster/container? is it a public container which I can try to pull by myself and try to reproduce?
Long shot: what is the version of tar on the remote pod? (tar --version)
from ksniff.
I'll try with a public accessible container and do the experiment again :) because this container is private image file.
🚀 kc -n epc1 exec -ti mme-0 -- tar --version
tar (GNU tar) 1.28
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by John Gilmore and Jay Fenlason.
from ksniff.
OK, pretty sure I found the problem.
Ksniff had a bug in the way it use tar, I didn't passed the -C flag and therefore tar always operated on the current directory.
It works most of the times because most of the containers use "/" as their working dir.
Your container use custom working dir (/opt/c3po/mme/bin) so the upload worked but the file extracted to the wrong directory (/opt/c3po/mme/bin/tmp/static-tcpdump)
I fixed the bug by adding -C when needed and setting up the tar header properly.
I merged the fix as v1.2.1 (https://github.com/eldadru/ksniff/releases/download/v1.2.1/ksniff.zip)
Please try again with this version and tell me if that fixed the issue.
from ksniff.
@eldadru Sorry for late reply, I'm too busy in these 2 days.
After tested, ksniff can work now, many thanks to your fix, this plugin worked perfectly and helped me a lot 👍
from ksniff.
Related Issues (20)
- can't execute 'ctr': No such file or directory HOT 3
- can't run in privileged mode
- Openshift 4.10 Mac M1 nsenter: can't execute 'tcpdump': No such file or directory
- sniff on RKE2 fails to create the ksniff pod HOT 2
- Vulnerable 3rd party libs
- Release new version HOT 1
- Add istio inject false label/annotation
- Add ARM linux based static-tcpdump HOT 1
- ksniff failing to create a privileged pod on the node with taints
- Issues running on arm64 HOT 1
- Can ksniff be used for one-time packet captures (~10 min) in production environment for debugging purposes? Apart from clean up of left over pods or containers, what else should I look into after using it for a short pcap?
- cloud storage support for sniff output
- Not able to build static-tcpdump on Ubuntu 22.04 Jammy
- Can we take tcpdump on multiple pods using ksniff ? For an example if a namespace has 4 PODs running can we take TCPDUMP on all the 4 PODs using kubectl sniff -p <pod1> <pod2> <pod3>
- 'kubectl sniff' command returning 139 exit/error code during execution. RCA required for failed attempt at packet capture so that workaround can be identified. HOT 6
- exitCode: '126' when using --tcpdump-image for an airgapped scenario HOT 1
- kubectl sniff fails with certificate error.
- Incorrect Makefile install path for kubectl >= 1.28
- K8S API client 30s timeout: request canceled (Client.Timeout exceeded while awaiting headers)
- ksnif with GKE (cos) capture with privileged (-p) option: Nothing get captured
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ksniff.