andresriancho / collector Goto Github PK

Clear CPU load before w3af start, to get a real CPU load from the OS, that's not related with the installation.

Measure number of w3af HTTP requests that make it to the network, the number returned by the id generator also counts the requests we get from the cache.

on mac Yosemite 10.10.2 can't produce . /tmp/w3af_dependency_install.sh

• Use http://techblog.netflix.com/2013/03/ami-creation-with-aminator.html
• Update https://github.com/andresriancho/collector/wiki/Building-new-AMIs-for-faster-collection

For historical reasons it would be really good to have the data in S3

The file we want to upload is OUTPUT_FILE = 'collect-output.tar.bz2', to make things easier we just:

• Download the file from the collector EC2 instance to our host (where the AWS credentials live)
• Upload collect-output.tar.bz2 to S3

The configuration should be part of main:

main:
 - S3: name-of-bucket/

The file should be renamed to something more unique like: revision-date.tar.bz2

Upload to s3 from EC2 instead of downloading to host running collector and then uploading.

TODO

• config.yml should support calling wpa
• Configure it for the w3af example
• Make sure it works, it requires me to use the patched python to call wpa, most likely will have to use full paths in my config?

on mac Yosemite 10.10.2 can't produce . /tmp/w3af_dependency_install.sh

Clear network stats before w3af run so we get a better/real understanding of how much traffic w3af generated, vs. the traffic generated during install+run

Before starting, the collection tool will make sure that all the scripts referenced by the configuration exist, and are marked +x.

Create docker image to run w3af inside which will speedup the setup phase. Remember that we have to re-compile python now to have support for some of our tools.

TODO

• Create a Dockerfile in collector/examples/w3af based on the w3af docker image but installing all the other things I need for profiling
• Also update the ami to point to one that already has docker installed:

main:
  ami: ami-48eb2f20

• Create a script to run w3af inside docker and save the profiling output to a place where I can easily download afterwards

Measure number of 404 pages generated by w3af during a scan

It is a good idea to also include some generic performance output which the collection tool can gather using the psutils module. Even if w3af can collect it itself, the generic output would be useful to others that don't want to modify their software.

I would like to have a log for the different caches in w3af, at least these caches are important and need to be analyzed:

• HTTP request/response cache
• DocumentParser cache

The log can look like this:

epoch,result,key
123,MISS,http://foo.com/abc
124,HIT,http://foo.com/abc

I'll parse it in the analyzer to get the real stats.

User story

As a user I want to be able to retrieve the collector output and be able to run EXACTLY THE SAME TEST again. In order to do this I need this information to be in the output file:

• The config.yml
• Collector version/git revision
• All files referenced by config.yml
• Any command line arguments passed to the collector

Do not allow runs with revision set as "master" or "develop", make sure that the revision specified in the command line is a hash.