criblio / appscope Goto Github PK
View Code? Open in Web Editor NEWGain observability into any Linux command or application with no code modification
Home Page: https://appscope.dev
License: Apache License 2.0
Gain observability into any Linux command or application with no code modification
Home Page: https://appscope.dev
License: Apache License 2.0
When using TLS connections we are picking up the handshake content as well. While this might be useful in some cases we need to have a switch to not output this content
My preference would be to have this option off by default and allow users to turn it on if desired
To reproduce:
run the following command
SCOPE_PAYLOAD_ENABLE=true ./scope curl -v --http1.1 https://cdn.cribl.io/dl/latest
Look at one of the .in or .out files that are output in tmp
See below for an example of how the request looks like
/tmp$ cat 24778_99.84.198.43:44844:443.out | hexdump -C
00000000 16 03 01 02 00 01 00 01 fc 03 03 65 5a 78 47 f6 |...........eZxG.|
00000010 1b 5c c3 5c 4a f3 ef 5d 65 3d d6 3d 03 42 92 5f |..\J..]e=.=.B._|
00000020 ac bf 4c e8 23 dd 84 74 ad 43 c3 20 bf df 2d aa |..L.#..t.C. ..-.|
00000030 d5 85 b6 21 41 62 86 a9 69 16 8c 12 80 67 78 32 |...!Ab..i....gx2|
00000040 ef ce 1c 9e 4a 7b 20 c0 d3 c9 47 c7 00 3e 13 02 |....J{ ...G..>..|
00000050 13 03 13 01 c0 2c c0 30 00 9f cc a9 cc a8 cc aa |.....,.0........|
00000060 c0 2b c0 2f 00 9e c0 24 c0 28 00 6b c0 23 c0 27 |.+./...$.(.k.#.'|
....
000002b0 47 45 54 20 2f 64 6c 2f 6c 61 74 65 73 74 20 48 |GET /dl/latest H|
000002c0 54 54 50 2f 31 2e 31 0d 0a 48 6f 73 74 3a 20 63 |TTP/1.1..Host: c|
000002d0 64 6e 2e 63 72 69 62 6c 2e 69 6f 0d 0a 55 73 65 |dn.cribl.io..Use|
000002e0 72 2d 41 67 65 6e 74 3a 20 63 75 72 6c 2f 37 2e |r-Agent: curl/7.|
000002f0 35 38 2e 30 0d 0a 41 63 63 65 70 74 3a 20 2a 2f |58.0..Accept: /|
00000300 2a 0d 0a 0d 0a 17 03 03 00 13 44 7d 84 58 58 63 |.........D}.XXc|
00000310 aa 3b d9 93 b8 92 5d 4e 87 aa 4d 8f 6a |.;....]N..M.j|
0000031d
When running a payload capture when using curl we only capture the response of DNS requests, but not the requests.
To reproduce:
run the following command
SCOPE_PAYLOAD_ENABLE=true ./scope curl -v --http1.1 https://cdn.cribl.io/dl/latest
Look at the files that are produced in /tmp for the process
Notice that there is 24778_127.0.0.53:53:0.in but no corresponding .out file
See below for an example of how the request looks like
/tmp$ ls -lha 24778_*
-rw-rw-r-- 1 ledion ledion 146 Jan 7 10:00 24778_127.0.0.53:53:0.in
-rw-rw-r-- 1 ledion ledion 6.4K Jan 7 10:00 24778_99.84.198.43:443:44844.in
-rw-rw-r-- 1 ledion ledion 797 Jan 7 10:00 24778_99.84.198.43:44844:443.out
Currently panics and exits. Should retry with ascii skipped.
Scoping scope fails. This should probably work. The following example generates an error:
/root/.scope/scopec was compiled with a version of go older than go1.8. This is not supported by scope. Continuing without scope.```
Ledion observed splunkweb crashing when scoped. Need more details, repro steps, etc.
Today, when enabled, scope sends out one event for *every *console write, even if it's just one char. This is quite wasteful for both the sender as well as the receiver. We should bundle this data up and send/flush on either new-line, or buffer size or some elapsed time since last send
nc -lk localhost 9109 | jq .body.data
SCOPE_EVENT_CONSOLE=true SCOPE_EVENT_LOGFILE=true ./scope date
"Thu"
" "
"Jan"
" "
" "
"7"
" "
"0"
"9"
":"
"18"
":"
"56"
" "
"PST"
" "
"2021"
"\n"
Currently a counter
Changes I need to be able to build successfully were backed out from yesterday to today. Right now I get this build failure as a result of make all
:
...
make -f cli/Makefile cliall
make[1]: Entering directory '/home/cribl/appscope'
DIR=/home/cribl/appscope/cli/
GO_VER=go1.15.6
GOPATH=/home/cribl/go
go get github.com/go-bindata/go-bindata/...
go get github.com/ahmetb/govvv
cp /home/cribl/appscope/cli/../bin/linux/scopec /home/cribl/appscope/cli/build/
cd /home/cribl/appscope/cli/ && go-bindata -o run/scopec.go -pkg run build/scopec
/bin/sh: 1: go-bindata: not found
cli/Makefile:36: recipe for target 'run/scopec.go' failed
make[1]: *** [run/scopec.go] Error 127
make[1]: Leaving directory '/home/cribl/appscope'
Makefile:29: recipe for target 'all' failed
make: *** [all] Error 2
Since I've started building/testing cli/build/scope it's now common for me to see my VM in a “no space left on device” state. I did not have the problem before building/running the cli.
I have a VM with 10GB of disk. I've done some manual cleanup and got it down a little, but not a lot. I leave this VM open and have it running constantly.
Then I rebooted the VM which reclaimed an additional full GB. This looks like what happens when files are deleted while processes are actively writing to them.
It's all circumstantial at this point, but:
--> I think cli/build/scope is deleting file descriptors that are still open and being written to. <--
error reading events: bufio.Scanner: token too long
Repro with apt-get
Some command help is capitalized and not. Look and make it consistent.
If a command does not exist or there's simply a typo, we will fail to create metrics.json and events.json which will cause errors in the CLI. When detecting a missing metrics.json or events.json, there are a couple of things we might do:
Seeing 3 or 4 line breaks per line some times.
Description
Today, if the queue is full &/or we are rate limited we drop data. For log data, files and console, as well as events, fs, net, dns, we need to not drop the data.
Possibly spool to the local fs.
Note: spool to the local fs presents issues. Will look at a few things.
I've noticed that the dash cli command will crash pretty consistently when running alongside a scoped command that finishes. I'm running it on ubuntu 18.0.4 (on an aws ec2 instance). Additionally, when the dashboard crashes, it leaves the tty in a somewhat unusable state (I'm assuming it's an ncurses thing), though a blind typed reset command seems to fix that.
To Reproduce:
scope find / -name test; date
scope dash; date
The dash will work find, until after the find ends (the reason for the tacked on date commands). In my testing, anywhere from 10-15 seconds after the find finishes, scope crashes with a "Killed" message.
Cant repro, try a few more things.
Today we can signal via our file based method to send environment variables to a running scope instance. However, in the CLI (and likely running with scope connected to LogStream) we don't articulate configs in environment variable syntax. Therefore, sending SCOPE_EVENT_METRICS=true might override watch configurations we've already established.
We need to allow a signal to be sent to scope to have it load its configuration from disk, or in the case of LogStream via a TCP connection, sending a new running configuration to be transitioned to in full YAML or JSON.
When enabled and set to go to a directory, payloads should not also go out the events interface.
Seems to fail silently.
Investigate whether screen clears the environment or why in general that doesn't work.
#24 mentions reverting some of my changes.
This issue was created to hopefully resolve issues which were seen on that branch (bugs/bugathon-1).
Provide example event here.
A few things are needed in order for integ tests to run reliably.
Multiple scoped processes writing to the same file can cause unexpected results. Need to configure scope.yml
from scope
to write to a file per PID, and then modify scope
to read from multiple files.
With commit 673c5b4a4aeec071ee9ad2b14e7712e83a376f86
, we've introduced new null characters into the event output which are causing problems with JSON parsing.
First noticed the error: error reading events: invalid character '\x00' looking for beginning of value
from the CLI. Chased it back to the above mentioned commit where I first observed this. When looking at the events.json file in hexdump:
00000000 00 7b 22 74 79 70 65 22 3a 22 65 76 74 22 2c 22 |.{"type":"evt","|
00000010 62 6f 64 79 22 3a 7b 22 73 6f 75 72 63 65 74 79 |body":{"sourcety|
00000020 70 65 22 3a 22 6e 65 74 22 2c 22 69 64 22 3a 22 |pe":"net","id":"|
00000030 65 36 64 34 37 37 64 61 62 36 66 31 2d 63 75 72 |e6d477dab6f1-cur|
00000040 6c 2d 20 2d 2d 68 74 74 70 31 2e 31 20 68 74 74 |l- --http1.1 htt|
00000050 70 73 3a 2f 2f 77 74 74 72 2e 69 6e 2f 39 34 36 |ps://wttr.in/946|
00000060 31 31 22 2c 22 5f 74 69 6d 65 22 3a 31 36 31 32 |11","_time":1612|
00000070 32 34 35 30 36 37 2e 37 31 39 2c 22 73 6f 75 72 |245067.719,"sour|
00000080 63 65 22 3a 22 6e 65 74 2e 63 6f 6e 6e 2e 63 6c |ce":"net.conn.cl|
00000090 6f 73 65 22 2c 22 68 6f 73 74 22 3a 22 65 36 64 |ose","host":"e6d|
000000a0 34 37 37 64 61 62 36 66 31 22 2c 22 70 72 6f 63 |477dab6f1","proc|
000000b0 22 3a 22 63 75 72 6c 22 2c 22 63 6d 64 22 3a 22 |":"curl","cmd":"|
000000c0 2f 75 73 72 2f 62 69 6e 2f 63 75 72 6c 20 2d 73 |/usr/bin/curl -s|
000000d0 20 2d 2d 68 74 74 70 31 2e 31 20 68 74 74 70 73 | --http1.1 https|
000000e0 3a 2f 2f 77 74 74 72 2e 69 6e 2f 39 34 36 31 31 |://wttr.in/94611|
000000f0 22 2c 22 70 69 64 22 3a 33 36 38 39 2c 22 5f 63 |","pid":3689,"_c|
00000100 68 61 6e 6e 65 6c 22 3a 22 32 31 34 30 33 38 34 |hannel":"2140384|
00000110 35 36 31 36 34 35 39 38 22 2c 22 64 61 74 61 22 |56164598","data"|
00000120 3a 7b 22 6e 65 74 2e 74 72 61 6e 73 70 6f 72 74 |:{"net.transport|
00000130 22 3a 22 49 50 2e 54 43 50 22 2c 22 6e 65 74 2e |":"IP.TCP","net.|
00000140 70 65 65 72 2e 69 70 22 3a 22 35 2e 39 2e 32 34 |peer.ip":"5.9.24|
00000150 33 2e 31 38 37 22 2c 22 6e 65 74 2e 70 65 65 72 |3.187","net.peer|
00000160 2e 70 6f 72 74 22 3a 22 34 34 33 22 2c 22 6e 65 |.port":"443","ne|
00000170 74 2e 68 6f 73 74 2e 69 70 22 3a 22 31 37 32 2e |t.host.ip":"172.|
00000180 31 37 2e 30 2e 34 22 2c 22 6e 65 74 2e 68 6f 73 |17.0.4","net.hos|
00000190 74 2e 70 6f 72 74 22 3a 22 35 31 39 35 32 22 2c |t.port":"51952",|
000001a0 22 6e 65 74 2e 70 72 6f 74 6f 63 6f 6c 22 3a 22 |"net.protocol":"|
000001b0 68 74 74 70 22 2c 22 64 75 72 61 74 69 6f 6e 22 |http","duration"|
000001c0 3a 36 38 37 2c 22 6e 65 74 2e 62 79 74 65 73 5f |:687,"net.bytes_|
000001d0 73 65 6e 74 22 3a 37 34 36 2c 22 6e 65 74 2e 62 |sent":746,"net.b|
000001e0 79 74 65 73 5f 72 65 63 76 22 3a 31 32 31 32 38 |ytes_recv":12128|
000001f0 2c 22 6e 65 74 2e 63 6c 6f 73 65 2e 72 65 61 73 |,"net.close.reas|
00000200 6f 6e 22 3a 22 6c 6f 63 61 6c 22 7d 7d 7d 0a |on":"local"}}}.|
0000020f
Notice the null character before the line that's output.
Repro steps:
scope
rc2: curl -Lo scope https://cdn.cribl.io/dl/scope/$(curl -L https://cdn.cribl.io/dl/scope/latest)/linux/scope && chmod 755 ./scope
scope curl --http 1.1 https://cribl.io/404
scope events
cat $(scope hist -d)/events.json
When running scope events -t http -n 20
we will first find the last 20 events and then filter to only HTTP events. We need to change the order of these operations.
#!/usr/bin/python
import sys
for i in range(1000000): sys.stdout.write(str(i)+"\n") sys.stdout.flush()
We attempt to write 1 million lines to standard out. On my machine, running this way:
SCOPE_EVENT_DEST=file://./milliontest.log SCOPE_EVENT_CONSOLE=true ~/.scope/cscope python ./test.py >/dev/null
We only capture about 20% of the lines:
root /workspaces/scope/cli (feature/scopecli) $ wc -l milliontest.log
261923 milliontest.log
Kafka version 2.15 is no longer supported. Latest is 2.17. This causes integration test build of Kafka to fail.
Make Q exit scope dash
scope prune
when erroring should print help afterwards. Look at other commands to determine whether certain errors should print usage.
/dl/scope/<version>/scope
/dl/scope/stable
-> 0.5.0
/dl/scope/latest
-> /dl/scope/latest
We can fit better on smaller screens without the bigger single value widgets. Collapse them into normal text.
Donn said something about root cause, but didn't write it down.
#!/bin/sh
idx=1 while true; do echo "foo $idx" idx=$((idx+1)) sleep 1 done
The script will echo incrementing counters after foo. However, Scope’s output captures the current line and some random amount from previous:
root /workspaces/scope/cli (feature/scopecli) $ SCOPE_EVENT_CONSOLE=true SCOPE_EVENT_DEST=file://stderr ~/.scope/cscope ./test2.sh 2>&1 | grep evt
{"type":"evt","body":{"sourcetype":"console","id":"d55805e5c25e-sh-/workspaces/scope/cli/./test2.sh","_time":1609373240.863,"source":"stdout","host":"d55805e5c25e","proc":"sh","cmd":"/bin/sh /workspaces/scope/cli/./test2.sh","pid":15835,"_channel":"1059806938388474","data":"foo 1\n"}}
{"type":"evt","body":{"sourcetype":"console","id":"d55805e5c25e-sh-/workspaces/scope/cli/./test2.sh","_time":1609373240.863,"source":"stdout","host":"d55805e5c25e","proc":"sh","cmd":"/bin/sh /workspaces/scope/cli/./test2.sh","pid":15835,"_channel":"1059806938388474","data":"foo 1\n"}}
{"type":"evt","body":{"sourcetype":"console","id":"d55805e5c25e-sh-/workspaces/scope/cli/./test2.sh","_time":1609373241.872,"source":"stdout","host":"d55805e5c25e","proc":"sh","cmd":"/bin/sh /workspaces/scope/cli/./test2.sh","pid":15835,"_channel":"1059806938388474","data":"foo 2\n"}}
{"type":"evt","body":{"sourcetype":"console","id":"d55805e5c25e-sh-/workspaces/scope/cli/./test2.sh","_time":1609373240.863,"source":"stdout","host":"d55805e5c25e","proc":"sh","cmd":"/bin/sh /workspaces/scope/cli/./test2.sh","pid":15835,"_channel":"1059806938388474","data":"foo 1\n"}}
{"type":"evt","body":{"sourcetype":"console","id":"d55805e5c25e-sh-/workspaces/scope/cli/./test2.sh","_time":1609373241.872,"source":"stdout","host":"d55805e5c25e","proc":"sh","cmd":"/bin/sh /workspaces/scope/cli/./test2.sh","pid":15835,"_channel":"1059806938388474","data":"foo 2\n"}}
{"type":"evt","body":{"sourcetype":"console","id":"d55805e5c25e-sh-/workspaces/scope/cli/./test2.sh","_time":1609373242.884,"source":"stdout","host":"d55805e5c25e","proc":"sh","cmd":"/bin/sh /workspaces/scope/cli/./test2.sh","pid":15835,"_channel":"1059806938388474","data":"foo 3\n"}}
{"type":"evt","body":{"sourcetype":"console","id":"d55805e5c25e-sh-/workspaces/scope/cli/./test2.sh","_time":1609373240.863,"source":"stdout","host":"d55805e5c25e","proc":"sh","cmd":"/bin/sh /workspaces/scope/cli/./test2.sh","pid":15835,"_channel":"1059806938388474","data":"foo 1\n"}}
{"type":"evt","body":{"sourcetype":"console","id":"d55805e5c25e-sh-/workspaces/scope/cli/./test2.sh","_time":1609373241.872,"source":"stdout","host":"d55805e5c25e","proc":"sh","cmd":"/bin/sh /workspaces/scope/cli/./test2.sh","pid":15835,"_channel":"1059806938388474","data":"foo 2\n"}}
{"type":"evt","body":{"sourcetype":"console","id":"d55805e5c25e-sh-/workspaces/scope/cli/./test2.sh","_time":1609373242.884,"source":"stdout","host":"d55805e5c25e","proc":"sh","cmd":"/bin/sh /workspaces/scope/cli/./test2.sh","pid":15835,"_channel":"1059806938388474","data":"foo 3\n"}}
{"type":"evt","body":{"sourcetype":"console","id":"d55805e5c25e-sh-/workspaces/scope/cli/./test2.sh","_time":1609373243.897,"source":"stdout","host":"d55805e5c25e","proc":"sh","cmd":"/bin/sh /workspaces/scope/cli/./test2.sh","pid":15835,"_channel":"1059806938388474","data":"foo 4\n"}}
Upgrade scope binary in place if needed (validate md5).
Explain why you might need --
in case command parsing fails. Show more examples in the help.
To be spec'd out. scope flows
should show all network flow we've seen from metrics and events data and give users the ability to easily cat
the payloads from those flows.
Potentially, we should make it easy to convert flow data to PCAP format as well (we have POCs of this internally) or at least document how to do that.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.