rapid7 / dap Goto Github PK

Data Analysis Pipeline

License: MIT License

Ruby 98.64% Dockerfile 0.06% Shell 1.30%

dap's Introduction

DAP: The Data Analysis Pipeline

DAP was created to transform text-based data on the command-line, specializing in transforms that are annoying or difficult to do with existing tools.

DAP reads data using an input plugin, transforms it through a series of filters, and prints it out again using an output plugin. Every record is treated as a document (aka: hash/dict) and filters are used to reduce, expand, and transform these documents as they pass through. Think of DAP as a mashup between sed, awk, grep, csvtool, and jq, with map/reduce capabilities.

DAP was written to process terabyte-sized public scan datasets, such as those provided by https://scans.io/. Although DAP isn't particularly fast, it can be used across multiple cores (and machines) by splitting the input source and wrapping the execution with GNU Parallel.

Installation

Prerequisites

DAP requires Ruby and is best suited for systems with a relatively current version with 2.6.x being the minimum requirement. Ideally, this will be managed with either rbenv or rvm with the bundler gem also installed and up to date. Using system managed/installed Rubies is possible but fraught with peril.

Maxmind IP Location Databases

If you intend on using any of the geo_ip* or geo_ip2* filters, you must install the databases that provide the data for these filters. If you do not intend on using these filters, you can skip this step.

dap versions 1.4.x and later depend on Maxmind's geoip2/geolite2 databases to be able to append geographic and related metadata to analyzed datasets. In order to use this functionality you must put your copy of the relevant Maxmind databases in the correct location in /var/lib/geoip2 or the data directory of your dap installation or override with an environment variable that specifies the full path to the database in question:

ASN: GeoLite2-ASN.mmdb (environment override: GEOIP2_ASN_DATABASE_PATH)
City: GeoLite2-City.mmdb (environment override: GEOIP2_CITY_DATABASE_PATH)
ISP: GeoIP2-ISP.mmdb (environment override: GEOIP2_ISP_DATABASE_PATH)

NOTE: Prior to dap version 1.4.x there was a dependency on Maxmind's geoip database to be able to append geographic metadata to analyzed datasets. However, since that time Maxmind has dropped support for these legacy databases. If you intend to continue using this deprecated functionality, you must put your copy of the relevant Maxmind legacy databases in the correct location in /var/lib/geoip or the data directory of your dap installation or override with an environment variable that specifies the full path to the database in question:

ASN: GeoIPASNum.dat (environment override in 1.4.x+: GEOIP_ASN_DATABASE_PATH)
City: geoip_city.dat (environment override in 1.4.x+: GEOIP_CITY_DATABASE_PATH)
Org: geoip_org.dat (environment override in 1.4.x+: GEOIP_ORG_DATABASE_PATH)

Ubuntu 16.04+

sudo apt-get install zlib1g-dev ruby ruby-dev gcc make ruby-bundler
gem install dap

OS X

# Install the GeoIP C library required by DAP
brew update
brew install geoip

gem install dap

Usage

In its simplest form, DAP takes input, applies zero or more filters which modify the input, and then outputs the result. The input, filters and output are separated by plus signs (+). As seen from dap -h:

Usage: dap  [input] + [filter] + [output]
       --inputs
       --outputs
       --filters

To see which input/output formats are supported and what filters are available, run dap --inputs,dap --outputs or dap --filters, respectively.

This example reads as input a single IP address from STDIN in line form, applies geo-ip transformations as a filter on that line, and then returns the output as JSON:

$   echo 8.8.8.8 | bin/dap + lines + geo_ip2_city line + json | jq .
{
  "line": "8.8.8.8",
  "line.geoip2.city.city.geoname_id": "0",
  "line.geoip2.city.continent.code": "NA",
  "line.geoip2.city.continent.geoname_id": "6255149",
  "line.geoip2.city.country.geoname_id": "6252001",
  "line.geoip2.city.country.iso_code": "US",
  "line.geoip2.city.country.is_in_european_union": "false",
  "line.geoip2.city.location.accuracy_radius": "1000",
  "line.geoip2.city.location.latitude": "37.751",
  "line.geoip2.city.location.longitude": "-97.822",
  "line.geoip2.city.location.metro_code": "0",
  "line.geoip2.city.location.time_zone": "America/Chicago",
  "line.geoip2.city.postal.code": "",
  "line.geoip2.city.registered_country.geoname_id": "6252001",
  "line.geoip2.city.registered_country.iso_code": "US",
  "line.geoip2.city.registered_country.is_in_european_union": "false",
  "line.geoip2.city.represented_country.geoname_id": "0",
  "line.geoip2.city.represented_country.iso_code": "",
  "line.geoip2.city.represented_country.is_in_european_union": "false",
  "line.geoip2.city.represented_country.type": "",
  "line.geoip2.city.traits.is_anonymous_proxy": "false",
  "line.geoip2.city.traits.is_satellite_provider": "false",
  "line.geoip2.city.continent.name": "North America",
  "line.geoip2.city.country.name": "United States",
  "line.geoip2.city.registered_country.name": "United States"
}

There are also several examples of how to use DAP along with sample datasets here.

dap's People

Contributors

Stargazers

Watchers

dap's Issues

Splitting Subdomains?

End up resorting to awk today, but it'd be nice if I could do everything in dap.

Input:

{  
   "value":"cloud.appspider.rapid7.com"
}

Desired output:

{  
   "value":"cloud.appspider.rapid7.com",
   "value.f1":"cloud",
   "value.f2":"appspider",
   "value.f3":"rapid7",
   "value.f4":"com"
}

LoadError on oj.so

Hey guys, I have installed DAP with ruby 2.2 by bundle install --local, and it looked to be successful:

Using ffi 1.9.3                                                   
Using childprocess 0.5.3                                          
Using builder 3.2.2                                               
Using diff-lcs 1.2.5                                              
Using multi_json 1.10.1                                           
Using gherkin 2.12.2                                              
Using multi_test 0.1.1                                            
Using cucumber 1.3.16                                             
Using rspec-support 3.1.0                                         
Using rspec-expectations 3.1.0                                    
Using aruba 0.6.1                                                 
Using bit-struct 0.15.0                                           
Using geoip-c 0.9.1                                               
Using htmlentities 4.3.2                                          
Using mini_portile 0.6.0                                          
Using net-dns 0.8.0                                               
Using nokogiri 1.6.3.1                                            
Using oj 2.10.2                                                   
Using recog 2.0.2                                                 
Using rspec-core 3.1.1                                            
Using rspec-mocks 3.1.0                                           
Using rspec 3.1.0                                                 
Using bundler 1.10.3                                              
Bundle complete! 10 Gemfile dependencies, 23 gems now installed.  
Bundled gems are installed into ./vendor/bundle.

However, as I run the example given on the README of the repo, the following error is prompted:

$ echo 8.8.8.8 | bin/dap + lines + geo_ip line + json                                                                                         

/home/ubuntu/liulx/dap/vendor/bundle/ruby/2.2.0/gems/oj-2.10.2/lib/oj.rb:42:in `require': libruby.so.2.2: cannot open shared object file: No such file                                                                                                                                          
or directory - /home/ubuntu/liulx/dap/vendor/bundle/ruby/2.2.0/gems/oj-2.10.2/ext/oj/oj.so (LoadError)                                        
from /home/ubuntu/liulx/dap/vendor/bundle/ruby/2.2.0/gems/oj-2.10.2/lib/oj.rb:42:in `<top (required)>'                                        
from /home/ubuntu/liulx/dap/lib/dap/input.rb:4:in `require'                                                                                   
from /home/ubuntu/liulx/dap/lib/dap/input.rb:4:in `<module:Input>'                                                                            
from /home/ubuntu/liulx/dap/lib/dap/input.rb:2:in `<module:Dap>'                                                                              
from /home/ubuntu/liulx/dap/lib/dap/input.rb:1:in `<top (required)>'                                                                          
from /home/ubuntu/liulx/dap/lib/dap.rb:6:in `require'                                                                                         
from /home/ubuntu/liulx/dap/lib/dap.rb:6:in `<module:Dap>'                                                                                    
from /home/ubuntu/liulx/dap/lib/dap.rb:1:in `<top (required)>'                                                                                
from bin/dap:8:in `require'                                                                                                                   
from bin/dap:8:in `<main>'

I had checked the oj.so file and it is present in the according directory. Any thing obvious that I miss?

document dap

hello
I am a researcher and student in the field of collecting and analyzing information.
help me?

how to open or decode data fild the other project sonar ???
please give me doc for decode all project in sonar .

sample is very too low ...

Parsing of non-wdbrpc data with decode_wdbrpc_reply can cause dap to extract invalid fields or consume RAM/CPU indefinitely

Recently, as part of processing 2018-05-07-1525669801-udp_wdbrpc_17185.csv.gz from https://opendata.rapid7.com/sonar.udp/ as part of normal Sonar operations, I noticed that on some replies, dap will consume an extreme amount of CPU, so much that I just ctrl-c'd the process and went hunting for this bug.

It seems like there are several versions of this protocol that are present on the public Internet -- 1.0.x, 2.0 and 4.0. They all seem similar enough and this code handles them well. Unfortunately, the code does not check this version before attempting to go through the rest of the decoding. The result is that when decoding non-WDBRPC data, the fields extracted by dap will be incorrect at best, or, worse, may cause the rest of the code to consume unnecessary resources.

For example, this (base64 encoded) response causes current dap to hang:

2037312e2020362e3231362e2036323a31373138355b31373138355d1a09faba000000000000000255555555000000010000000100000000000000000000000000000000ffff55120000003c000000010000000200000000000000003a22737364703a646973636f766572220d0a4d583a330d0a0d0a0074

Reproduce this with:

echo -n '2037312e2020362e3231362e2036323a31373138355b31373138355d1a09faba000000000000000255555555000000010000000100000000000000000000000000000000ffff55120000003c000000010000000200000000000000003a22737364703a646973636f766572220d0a4d583a330d0a0d0a0074' |  ./bin/dap csv - + rename 1=data + transform data=hexdecode + annotate data=size + decode_wdbrpc_reply data + remove data +  csv

PR incoming.

Geoip support broken due to deprecated Maxmind data usage

These were deprecated in early January, 2019.

https://dev.maxmind.com/geoip/legacy/downloadable/ used to have the old files but https://dev.maxmind.com/geoip/geoip2/geolite2/ is a newer option.

https://github.com/maxmind/MaxMind-DB-Reader-ruby is probably the answer with some updates to https://github.com/rapid7/dap/blob/master/lib/dap/filter/geoip.rb

Extracting last element?

Is it possible to get the last field? e.g. instead of name.f4, could I do name.[-1]?

Desired command:

dap json + field_split_period name + where name.[-1] == com + json | jq

Similar command for future reference:

zcat /tyler/20170204-fdns.json.gz | head -10 | dap json + field_split_period name + where name.f4 == com + json | jq

NameError from sample script

I received the following error when running /samples/http_get_reply_iframes.json.sh

Traceback (most recent call last):
	12: from /usr/local/bin/dap:23:in `<main>'
	11: from /usr/local/bin/dap:23:in `load'
	10: from /var/lib/gems/2.5.0/gems/dap-1.2.6/bin/dap:121:in `<top (required)>'
	 9: from /var/lib/gems/2.5.0/gems/dap-1.2.6/bin/dap:121:in `each'
	 8: from /var/lib/gems/2.5.0/gems/dap-1.2.6/bin/dap:123:in `block in <top (required)>'
	 7: from /var/lib/gems/2.5.0/gems/dap-1.2.6/bin/dap:123:in `collect'
	 6: from /var/lib/gems/2.5.0/gems/dap-1.2.6/bin/dap:123:in `block (2 levels) in <top (required)>'
	 5: from /var/lib/gems/2.5.0/gems/dap-1.2.6/lib/dap/filter/http.rb:66:in `process'
	 4: from /var/lib/gems/2.5.0/gems/dap-1.2.6/lib/dap/filter/http.rb:66:in `each_pair'
	 3: from /var/lib/gems/2.5.0/gems/dap-1.2.6/lib/dap/filter/http.rb:68:in `block in process'
	 2: from /var/lib/gems/2.5.0/gems/dap-1.2.6/lib/dap/filter/http.rb:76:in `extract'
	 1: from /var/lib/gems/2.5.0/gems/dap-1.2.6/lib/dap/filter/http.rb:76:in `each'
/var/lib/gems/2.5.0/gems/dap-1.2.6/lib/dap/filter/http.rb:79:in `block in extract': undefined local variable or method `urls' for #<Dap::Filter::FilterHTMLIframes:0x000055bee91ef700> (NameError)
Did you mean?  url

Documentation

Is there any documentation for the tool outside of the examples?

decode sonar ssl

i downloaded
https://scans.io/data/rapid7/sonar.ssl/20160509/20160509_certs.gz
https://scans.io/data/rapid7/sonar.ssl/20160509/20160509_hosts.gz
https://scans.io/data/rapid7/sonar.ssl/20160509/20160509_names.gz
from scans.io
i readed 20160509/20160509_hosts.gz similar following
zcat 20160509/20160509_hosts.gz |less . but there is unclear strings.
for example:
13.90.214.57,38113203d606e5f5459575d9687e5d849ba7b799
13.90.214.57,948e1652586240d453287ab69caeb8f2f4f02117
23.0.17.189,6ad2b04e2196e48bf685752890e811cd2ed60606
...
i hex decoded some strings. and again got unclear string
now this is my question: how decode or decrypt mentioned unclear string

Error installing dap on MAC

Building native extensions. This could take a while...
ERROR: Error installing dap:
ERROR: Failed to build gem native extension.

current directory: /Library/Ruby/Gems/2.3.0/gems/geoip-c-0.9.1/ext/geoip

/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/ruby -r ./siteconf20190916-24561-1oci3x7.rb extconf.rb
checking for iconv_open() in iconv.h... no
checking for iconv_open() in -liconv... yes
checking for GeoIP_record_by_ipnum() in -lGeoIP... no
you must have geoip c library installed!
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.

Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/$(RUBY_BASE_NAME)
--with-geoip-dir
--without-geoip-dir
--with-geoip-include
--without-geoip-include=${geoip-dir}/include
--with-geoip-lib
--without-geoip-lib=${geoip-dir}/lib
--with-iconvlib
--without-iconvlib
--with-GeoIPlib
--without-GeoIPlib

To see why this extension failed to compile, please check the mkmf.log which can be found here:

/Library/Ruby/Gems/2.3.0/extensions/universal-darwin-18/2.3.0/geoip-c-0.9.1/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in /Library/Ruby/Gems/2.3.0/gems/geoip-c-0.9.1 for inspection.
Results logged to /Library/Ruby/Gems/2.3.0/extensions/universal-darwin-18/2.3.0/geoip-c-0.9.1/gem_make.out

document dap

i want a document from dap in order to learn usage --fiters option
please
thanks

Multiple "Skipping impossibly large" errors when working with http_body data

I want to analyze the body of HTTP responses; however, I am seeing errors which say Skipping impossibly large 26003-byte #1 chunk, at offset 6/21013.

I can reproduce these errors when processing the http_get_reply_iframes.json.bz2 file provided in the samples directory using the following command:

bzcat http_get_reply_iframes.json.bz2 | dap json + select ip data + transform data=base64decode + decode_http_reply data + remove data data.http_raw_body + select ip + json

I am running DAP in Docker and mounting the samples directory. My Dockerfile is a duplicate of this repo's Dockerfile, but I removed the installation of MaxMind as it was throwing an error which I think is due to a licensing change...

How should I structure the DAP query to avoid the skipping?

"duplicated key" warning from htmlentities

$   bundle install  && echo 8.8.8.8 | bin/dap + lines + geo_ip line + json   
Using ffi 1.9.3
Using childprocess 0.5.3
Using builder 3.2.2
Using diff-lcs 1.2.5
Using multi_json 1.10.1
Using gherkin 2.12.2
Using multi_test 0.1.1
Using cucumber 1.3.16
Using rspec-support 3.1.0
Using rspec-expectations 3.1.0
Using aruba 0.6.1
Using bit-struct 0.15.0
Using geoip-c 0.9.1
Using htmlentities 4.3.2
Using mini_portile 0.6.0
Using net-dns 0.8.0
Using nokogiri 1.6.3.1
Using oj 2.10.2
Using recog 2.0.2
Using rspec-core 3.1.1
Using rspec-mocks 3.1.0
Using rspec 3.1.0
Using bundler 1.10.5
Bundle complete! 10 Gemfile dependencies, 23 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
/home/jhart/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/htmlentities-4.3.2/lib/htmlentities/mappings/expanded.rb:465: warning: duplicated key at line 466 ignored: "inodot"
{"line":"8.8.8.8"}

This was fixed in htmlentities 4.3.3

Json deserialization coerces some values into symbols

DAP uses Oj to deserialize json. However, Oj is not set to use 'strict' or 'compat' modes. Thus, when it sees a value in object prefixed with ":", it attempts to convert it to a symbol, which often breaks downstream processing.

Example:

{ "a": ":b" }

:b above will be converted into a ruby symbol named "b". With strict mode (enforces only the 7 json types may be used) enabled, however, :b will remain a string.