Giter VIP home page Giter VIP logo

Comments (15)

eldadru avatar eldadru commented on July 23, 2024

Thanks for the detailed issue,

Let's try the following:

  • Verify that the static tcpdump is running correctly on the remote pod: when getting the error, exec into the remote pod and verify that it's running using "ps aux". If not, exec into the container and manually try to run /tmp/static-tcpdump - do you get an error? which one?

  • If the static tcpdump run correctly, try to run ksniff with output file flag (using "-o") this way we should have a pcap file instead of "streaming" the results to wireshark. sniff for couple seconds and try to open the file with wireshark - do you see the capture? same error?

Eldad

from ksniff.

aweimeow avatar aweimeow commented on July 23, 2024

Hello Eldad, thanks for quick reply,

for first question, I don't find tcpdump executable in /tmp

🚀  kc -n epc1 exec -ti mme-0 -- bash
root@mme-0:/opt/c3po/mme/bin# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   4504   704 ?        Ss+  19:28   0:00 /bin/sh -c /opt
root         7  0.0  0.0  18036  3000 ?        S+   19:28   0:00 /bin/bash /opt/
root        35  0.1  0.0 3699596 45064 ?       Sl+  19:28   0:11 ./vb_acc
root        82  0.0  0.0  18244  3296 ?        Ss+  19:45   0:00 bash
root       210  0.0  0.0  18244  3268 ?        Ss   22:05   0:00 bash
root       220  0.0  0.0  34424  2836 ?        R+   22:05   0:00 ps aux
root@mme-0:/opt/c3po/mme/bin# cd /tmp/
root@mme-0:/tmp# ls
root@mme-0:/tmp# ls -al
total 8
drwxrwxrwt  2 root root 4096 Sep 26 16:23 .
drwxr-xr-x 52 root root 4096 Nov 28 19:28 ..

And I have print output log to see, it shows:

rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "exec: \"/tmp/static-tcpdump\": stat /tmp/static-tcpdump: no such file or directory"

Seems sniff doesn't send tcpdump executable into the pod.

from ksniff.

eldadru avatar eldadru commented on July 23, 2024

If for some reason ksniff failed to upload tcpdump to the remote container the capture will fail.

Are you sure you executed into the container "mme" of the pod "mme-0" (I can't see that from kubectl output)?

Is uploading files using kubectl works fine with that pod? (ksniff upload files in a very similar way)

Edit: now I see you only have one container in that pod, so the answer to my first question is yes :)

from ksniff.

aweimeow avatar aweimeow commented on July 23, 2024

Yes, I can ensure I am in the mme container.
And I tried to upload tcpdump manually, it works, please help me with checking following log :)

🚀  kubectl -n epc1 cp ~/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump mme-0:/tmp/

🚀  kubectl -n epc1 exec -ti mme-0 -- bash
root@mme-0:/opt/c3po/mme/bin# cd /tmp
root@mme-0:/tmp# ls
static-tcpdump
root@mme-0:/tmp# ./static-tcpdump
static-tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
23:10:30.256254 IP 119.0.0.10.36412 > 10.244.2.13.36412: sctp (1) [HB REQ]
23:10:30.256415 IP 10.244.2.13.36412 > 119.0.0.10.36412: sctp (1) [HB ACK]
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
root@mme-0:/tmp#

I think the problem is why sniff can't upload to the pod now ;)

from ksniff.

eldadru avatar eldadru commented on July 23, 2024

Please try ksniff after manually uploading tcpdump - is it working now? can you see packets in wireshark?

I think it's pretty clear now that ksniff fail to upload tcpdump (and doesn't detect that) to the pod.

I'm not on Mac right now, my guess is that it something related to ksniff being used on Mac, I'll dig into it and update.

In the meanwhile, can you please try to manually specify the local path for the static tcpdump file using the "-l" flag?
please copy it from "~/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump" to other path on you machine and than use ksniff with "-l STATIC_TCPDUMP_PATH"
( make sure you delete tcpdump from the remote pod before trying)

from ksniff.

aweimeow avatar aweimeow commented on July 23, 2024

Yes, after I uploaded tcpdump manually, wireshark can work perfectly.

and I have change the tcpdump location do the executable path assignment:

🚀  kc -n epc1 exec -ti mme-0 -- bash
root@mme-0:/opt/c3po/mme/bin# rm /tmp/static-tcpdump
root@mme-0:/opt/c3po/mme/bin# exit

🚀  kc sniff mme-0 -n epc1 -l assets/static-tcpdump
INFO[0000] using tcpdump path at: 'assets/static-tcpdump'
INFO[0000] no container specified, taking first container we found in pod.
INFO[0000] selected container: 'mme'
INFO[0000] sniffing on pod: 'mme-0' [namespace: 'epc1', container: 'mme', filter: '']
INFO[0000] checking for static tcpdump binary on: '/tmp/static-tcpdump'
INFO[0000] couldn't find static tcpdump binary on: '/tmp/static-tcpdump', starting to upload
INFO[0000] tcpdump uploaded successfully
INFO[0000] spawning wireshark!

but the error message same as first time. I think it's some bug make macOS can't upload executable into pod.

from ksniff.

eldadru avatar eldadru commented on July 23, 2024

OK, thanks for the detailed info.
I'll try again on macOS soon and update with my findings.

For now, please use the workaround (manually uploading tcpdump using kubectl).

from ksniff.

aweimeow avatar aweimeow commented on July 23, 2024

Sure, thanks so much for your help 👍

from ksniff.

eldadru avatar eldadru commented on July 23, 2024

I've tried similar conditions on macOS, couldn't reproduce this issue.

To better investigate it I created a new version of ksniff (1.2.0) which include a verbose flag and better verification of the upload process.

Please manually upgrade your version (i'll update krew index later) by downloading the zip (https://github.com/eldadru/ksniff/releases/download/v1.2.0/ksniff.zip) and unpacking the darwin version instead of the existing version on krew folder ( ~/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/)

Execute ksniff in the same way you did and add "-v":
kc sniff mme-0 -n epc1 -v

Please post the output here.

Don't forget to make sure there is no leftover tcpdump binary on the remote container.

from ksniff.

aweimeow avatar aweimeow commented on July 23, 2024

Hello Eldad,

Here is the log, still can't work, please have a check.

INFO[0000] running in verbose mode
DEBU[0000] searching for tcpdump binary using lookup list: '[ /Users/aweimeow/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump /usr/local/bin/static-tcpdump /Users/aweimeow/.kube/plugin/sniff/static-tcpdump]'
DEBU[0000] tcpdump binary was not found at: ''
DEBU[0000] tcpdump binary found at: '/Users/aweimeow/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump'
INFO[0000] using tcpdump path at: '/Users/aweimeow/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump'
DEBU[0000] pod 'mme-0' status: 'Running'
INFO[0000] no container specified, taking first container we found in pod.
INFO[0000] selected container: 'mme'
INFO[0000] sniffing on pod: 'mme-0' [namespace: 'epc1', container: 'mme', filter: '']
INFO[0000] checking for static tcpdump binary on: '/tmp/static-tcpdump'
DEBU[0000] checked for tcpdump on remote pod: exit-code: '2', stdout: '', stderr: 'ls: cannot access '/tmp/static-tcpdump': No such file or directory
'
INFO[0000] couldn't find static tcpdump binary on: '/tmp/static-tcpdump', starting to upload
DEBU[0000] uploading file from: '/Users/aweimeow/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump' to '/tmp/static-tcpdump'
DEBU[0000] read '/Users/aweimeow/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump' to memory, file size: '2642872'
DEBU[0000] formatted '/Users/aweimeow/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump' as tar, tar size: '2644480'
DEBU[0000] done uploading file, exitCode: '0', stdOut: '', stdErr: 'tar: Removing leading `/' from member names
'
INFO[0000] verifying tcpdump uploaded successfully
DEBU[0000] checked for tcpdump on remote pod: exit-code: '2', stdout: '', stderr: 'ls: cannot access '/tmp/static-tcpdump': No such file or directory
'
ERRO[0000] failed to upload tcpdump.
Error: couldn't locate tcpdump on pod after upload done

from ksniff.

aweimeow avatar aweimeow commented on July 23, 2024

I am guessing if my kuberctl version and the way I control cluster (remotely) cause this bug, because according your log have verifying tcpdump uploaded successfully, then it indicated the tcpdump script should exist on /tmp/. But I checked with it, there is an empty directory.

Might be bug of other software?

from ksniff.

eldadru avatar eldadru commented on July 23, 2024

Very weird, according to the verbose output, everything looks fine but the file isn't on the remote path.

I used exactly the same environment as yours and it still works for me:

➜  ~ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:25:46Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Is there anything unusual about your kubectl/cluster/container? is it a public container which I can try to pull by myself and try to reproduce?

Long shot: what is the version of tar on the remote pod? (tar --version)

from ksniff.

aweimeow avatar aweimeow commented on July 23, 2024

I'll try with a public accessible container and do the experiment again :) because this container is private image file.

🚀  kc -n epc1 exec -ti mme-0 -- tar --version
tar (GNU tar) 1.28
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

from ksniff.

eldadru avatar eldadru commented on July 23, 2024

OK, pretty sure I found the problem.

Ksniff had a bug in the way it use tar, I didn't passed the -C flag and therefore tar always operated on the current directory.
It works most of the times because most of the containers use "/" as their working dir.
Your container use custom working dir (/opt/c3po/mme/bin) so the upload worked but the file extracted to the wrong directory (/opt/c3po/mme/bin/tmp/static-tcpdump)

I fixed the bug by adding -C when needed and setting up the tar header properly.

I merged the fix as v1.2.1 (https://github.com/eldadru/ksniff/releases/download/v1.2.1/ksniff.zip)
Please try again with this version and tell me if that fixed the issue.

from ksniff.

aweimeow avatar aweimeow commented on July 23, 2024

@eldadru Sorry for late reply, I'm too busy in these 2 days.
After tested, ksniff can work now, many thanks to your fix, this plugin worked perfectly and helped me a lot 👍

from ksniff.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.