A schedulable OSINT scanner using recon-ng which allows for the analysis of an attack surface over time.
This project comes from a desire to learn how to build applications on AWS, and a practical need to understand the attack surface of an organisation as a function of time. This initial, very basic release does domain enumeration periodically with effectively indefinite persistence of results. This will hopefully serves as a helpful baseline for dcata analysis.
AWS helpfully explains how we should segment large Organisations into multiple Accounts, and I have used a similar architecture for this project. Specifically:
- I have production and non-production workloads in their own accounts; and
- The CICD pipeline and associated components are also has its own acccount.
The pipeline itself comprises a source stage in Github, a build step followed by a deployment to a non-production (I refer to it herein as devtest) environment, and finally: deployment to production.
Fundamentally: recon-ng with a custom workflow run from the shell of a EC2 Instance created for the task. I could not figure out how to make recon-ng serverless, and so I've done what I hope is the next best thing:
- An EC2 Instance which is configured to launch, install recon-ng, clone this rpo, set up cron, set up some environment variables, then stop.
- A scheduled eventbridge task that invokes a Lambda, which itself starts the previously launched EC2 Instance. This act causes the enumeration script to run, resulting in a CSV file which is copied to S3 and then deleted locally. That EC2 Instance is then stopped (because I am cheap / frugal).
- S3 event notifications then cause another Lambda to invoke, which parses the file, loads contents into a DynamoDB table, then deletes the CSV file.
- A secondary DynamoDB table that summarises when hosts were first detected and most recently detected.
- Deploy the crossAccountRoles stack to the Accounts you intend to use for production and devtest.
- Deploy the pipeline stack to the Account you intend to use for CICD.
- The CodePipeline object will run automatically once the pipeline stack reaches CREATE_COMPLETE. This will result in a built application.
OK, so despite my best efforts the application is not perfectly self-contained. I.e. there are some items that need to be set up prior to deploying the pipeline stack.
- The Pipeline's source stage connects to Github through a Codestar Connection. This needs to instantiated, and its ARN given as a parameter.
- Since this repo is currently private, it cannot be accessed without authentication. Loathe as I am to store secrets in repos of any kind - I have placed the require Personal Access Token into the Systems Manager Paramater Store.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent