red-kite-solutions / stalker Goto Github PK

Ensure that it is deployable via a dockerfile
Ensure that it is deployable via a docker-compose file along with the other modules
Ensure that every job is runnable via the adequate environment in the docker container

The goal is to make sure that everything that was asked is completed.

Maybe purge the queue or get it to see if the job that we are about to add is already in the queue.

The HTTP screenshot job will try to take a screenshot of the front-page of a website to help identifying it quickly. The goal is then to store it as base64 in the database. It may be useful to have a web interface to visualize the pictures of a program or something.

Look into aquatone or another tool. It needs to be able to run without a GUI.

Try to query and update only the required domains in place of getting the whole program object for performance reasons.

AC:

• Crawl a HTTP server
• Report its files
• Configure crawling depth

Scan the known open ports with nmap -sV -sC options and store the output. This will give us more details about the service running on the port.

AC:

• Page showing details of a host object

Add a Slack reporting service that will send reports to different Slack channels to be used as a GUI/notification portal. When a new vulnerabiliity if found or a new website is screenshoted, for instance, the data will be sent as a slack message for later consultation.

Report new data to a keybase channel once in a while

Maybe make a boolean on domain objects that marks them as "big"

Big domains should not be queried with their domain array included if possible, but only query their child domains (subdomains). For performance reasons.

For instance, if example.com has 2000 subdomains, query the full domain object as little as possible and try to query sub1.example.com instead.

AC:

• Identify is port is running a FTP server
• Add service to port

Likely solution could be to back up the markdown file created in a file server somewhere.

Add the job to the database when it is created so that we can keep track of it. It will enable the deletion of said task upon completion, guaranteeing that the job will be done or queued (so eventually done), even if the server is stopped mid-job. The job would simply be restarted, granted that the content of the database is preserved and intact.

AC:

• Real paging
• List hosts in the database
• Filter by company

Add an admin UI to the flow manager

Templates with nice features (must include a copy of the MIT license and credit the creators):
https://github.com/akveo/ngx-admin

The UI will obviously need some backend features.

A completed first UI would include:

• A login page
• A landing home page once logged in
• A navigation menu on the left including :
  • An application settings link including :
    • A multi-tabs container with the tabs
      • Application settings
      • Users and roles
  • A home page link that sends you to the landing page
• A profile drop down on the top right including :
  • A link to modify profile (requires password to submit)
    • Change display name
    • Change email
    • Change password
  • A link to logout
• A 404 page

AC:

• Add an IP range to a company
• Scan IP range for live hosts
• Refresh on a time basis like a cron job
• Add the new hosts to the database
• The maximum is /8

AC:

• List a file on a HTTP server
• Report its existence

Lets say you add 5 jobs with a priority of 5, they will get prioritized in FIFO order. If you then add a job with a priority smaller than 5, lets say 3, it will become the next job as its priority is higher. However, the other jobs will lose their FIFO order and will get into a seemingly random order.

Explained visually:

At first:
{ id: 1, priority: 5}
{ id: 2, priority: 5}
{ id: 3, priority: 5}
{ id: 4, priority: 5}
{ id: 5, priority: 5}

Then:
{ id: 6, priority: 3}
{ id: 3, priority: 5}
{ id: 1, priority: 5}
{ id: 2, priority: 5}
{ id: 5, priority: 5}
{ id: 4, priority: 5}

The entended behavior is to keep the FIFO order in same priority jobs. A solution might be to add a queue entry timestamp with milliseconds and do a second prioritization based on that. I do not know if it is supported by the python priority queue.

Make sure that the flow manager is easily deployable via a Dockerfile
Make sure that the flow manager is deployable via a docker-compose file, along with the other modules, and that they interconnect properly

AC:

• Create a company
• Change its details
• Show a logo
• List existing companies

Add a port scanning job with masscan to find open ports really quickly.

AC:

• Scan all TCP ports for a host
• Add the found ports to the host in the database
• When a host is found, start a TCP scan on it

Add the results from the subdomains /recon/subdomains/:id/results to the database upon reception.

Remove the related job from the database.

Deployable via a dockerfile
Deployable via a docker compose

AC:

• Check for existence of a string
• Report its existence

IsNewContentReported: control if new data is reported, boolean. Set to false temporarily to prevent a hug report when we inject data ourselves, for instance.
Maybe add URLs, maybe API keys and stuff

AC:

• Page showing details of a domain object

AC:

• Identify a SMTP server
• Add service name to the port

AC:

• Identify a http(s) server on a port
• Add the service name to the port

Domain objects need to be able to handlea list of Ports that will contain a port number and several details about the running service. Only open ports will be stored.

Add the possibility to handle IP ranges in Program objects to potentially perform port scanning and host discovery on ip ranges

Create the Mongo DB instance in a docker container for easy erase of the data, to start clean everytime for development purposes.

Make sure that authentication is properly setup on it.

Find a convenient Mongo DB client for debugging purposes.

AC:

• Real paging
• List domains in the database
• Filter by company

AC:

• Brute force files and directories
• Report content
• Specify a word list

Make a function that goes through the full domain tree and calls a callback function in every node.

Useful for everything that needs its custom logic in every node, but needs to go through the full tree.

Something like:
function fullTreeAction(callback: function): void

It would call the callback by giving it the current domain node (this).

Implement the function in domain_tree.utils.ts

This job will find out and mark a port as running a HTTP server. Other jobs will rely on this information such as potential dirsearch jobs, web screenshot jobs, etc.

AC:

• Take a screenshot of a website
• Store the image for future use
• See the image in a host page

🎇 Aquatone

More research needs to be performed, but subdomain alteration scanning consists of trying common alternatives of a found subdomain to try to find new ones. For instance, www.example.com could lead to trying the subdomain ww2.example.com.

AC:

• Upload a list of domains for a company
• Add the unique domains to the database.

AC:

• From a host details, access the details of a port
• The UI of the details of a port must be procedurally generated

AC:

• Upload a list of hosts for a company
• Add the unique hosts to the database.
• Reverse DNS lookup

AC:

• List the last X reports
• Download the report