Giter VIP home page Giter VIP logo

codingo / vhostscan Goto Github PK

View Code? Open in Web Editor NEW
1.2K 39.0 232.0 229 KB

A virtual host scanner that performs reverse lookups, can be used with pivot tools, detect catch-all scenarios, work around wildcards, aliases and dynamic default pages.

License: GNU General Public License v3.0

Python 99.44% Dockerfile 0.56%
security-audit penetration-testing penetration-test virtual-hosts web-application-security discovery-service hacking hacking-tool virtual-host vhost vhosts security-tools security hackthebox oscp ctf-tools offensive-security bugbounty reverse-lookups scanner

vhostscan's Introduction

VHostScan

A virtual host scanner that can be used with pivot tools, detect catch-all scenarios, aliases and dynamic default pages. First presented at SecTalks BNE in September 2017 (slidedeck).

Build Status Python 3.2|3.6 PEP8 License Twitter Twitter

Key Benefits

  • Quickly highlight unique content in catch-all scenarios
  • Locate the outliers in catch-all scenarios where results have dynamic content on the page (such as the time)
  • Identify aliases by tweaking the unique depth of matches
  • Wordlist supports standard words and a variable to input a base hostname (for e.g. dev.%s from the wordlist would be run as dev.BASE_HOST)
  • Works over HTTP and HTTPS
  • Ability to set the real port of the webserver to use in headers when pivoting through ssh/nc
  • Add simple response headers to bypass some WAF products
  • Identify new targets by using reverse lookups and append to wordlist

Product Comparisons

VHOSTScan Feature Map

Install on docker (recommended)

  1. ´git clone https://github.com/codingo/VHostScan.git´
  2. ´cd VHostScan´
  3. ´docker build -t vhostscan .`

Then run application `docker run --rm -it vhostscan -t´

Install Requirements

Install using:

$ python3 setup.py install

Dependencies will then be installed and VHostScan will be added to your path. If there is an issue regarding running python3 setup.py build_ext, you will need to reinstall numpy using pip uninstall numpy and pip install numpy==1.12.0. This should resolve the issue as there are sometimes issues with numpy being installed through setup.py.

Usage

Argument Description
-h, --help Display help message and exit
-t TARGET_HOSTS Set the target host.
-b BASE_HOST Set host to be used during substitution in wordlist (default to TARGET).
-w WORDLISTS Set the wordlist(s) to use. You may specify multiple wordlists in comma delimited format (e.g. -w "./wordlists/simple.txt, ./wordlists/hackthebox.txt" (default ./wordlists/virtual-host-scanning.txt).
-p PORT Set the port to use (default 80).
-r REAL_PORT The real port of the webserver to use in headers when not 80 (see RFC2616 14.23), useful when pivoting through ssh/nc etc (default to PORT).
--ignore-http-codes IGNORE_HTTP_CODES Comma separated list of http codes to ignore with virtual host scans (default 404).
--ignore-content-length IGNORE_CONTENT_LENGTH Ignore content lengths of specificed amount.
--prefix PREFIX Add a prefix to each item in the wordlist, to add dev-<word>, test-<word> etc
--suffix SUFFIX Add a suffix to each item in the wordlist, to add <word>dev, <word>dev
--first-hit Return first successful result. Only use in scenarios where you are sure no catch-all is configured (such as a CTF).
--unique-depth UNIQUE_DEPTH Show likely matches of page content that is found x times (default 1).
--ssl If set then connections will be made over HTTPS instead of HTTP.
--fuzzy-logic If set then all unique content replies are compared and a similarity ratio is given for each pair. This helps to isolate vhosts in situations where a default page isn't static (such as having the time on it).
--no-lookups Disbale reverse lookups (identifies new targets and append to wordlist, on by default).
--rate-limit Amount of time in seconds to delay between each scan (default 0).
--random-agent If set, each scan will use a random user-agent from a predefined list.
--user-agent Specify a user agent to use for scans.
--waf If set then simple WAF bypass headers will be sent.
-oN OUTPUT_NORMAL Normal output printed to a file when the -oN option is specified with a filename argument.
-oG OUTPUT_GREPABLE Grepable output printed to a file when the -oG is specified with a filename argument.
-oJ OUTPUT_JSON JSON output printed to a file when the -oJ option is specified with a filename argument.
-v VERBOSE Increase the output of the tool to show progress

Usage Examples

Note that a number of these examples reference 10.10.10.29. This IP refers to BANK.HTB, a retired target machine from HackTheBox (https://www.hackthebox.eu/).

Quick Example

The most straightforward example runs the default wordlist against example.com using the default of port 80:

$ VHostScan -t example.com

Quick Example with SSL

If your connection requires SSL, you can use:

$ VHostScan -t example.com --ssl

VHOSTScan Wordlist example

Port forwarding

Say you have an SSH port forward listening on port 4444 fowarding traffic to port 80 on example.com's development machine. You could use the following to make VHostScan connect through your SSH tunnel via localhost:4444 but format the header requests to suit connecting straight to port 80:

$ VHostScan -t localhost -b example.com -p 4444 -r 80

STDIN

VHostScan Supports piping from other applications and will treat information passed to VHostScan as wordlist data, for example:

$ cat bank.htb | VHostScan -t 10.10.10.29

VHOSTScan STDIN Example

STDIN and WordList

You can still specify a wordlist to use along with stdin. In these cases wordlist information will be appended to stdin. For example:

$ echo -e 'a.example.com\b.example.com' | VHostScan -t localhost -w ./wordlists/wordlist.txt

Fuzzy Logic

Here is an example with fuzzy logic enabled. You can see the last comparison is much more similar than the first two (it is comparing the content not the actual hashes):

VHOSTScan Fuzzy Logic Example

Running the tests

This project includes a small battery of tests. It's really simple to run the tests:

pip install -r test-requirements.txt
python3 setup.py test

If you're thinking of adding a new feature to the project, consider also contributing with a couple of tests. A well-tested codebase is a sane codebase. :)

vhostscan's People

Contributors

aancw avatar arpitx165 avatar bcoles avatar ccsplit avatar codingo avatar dekan avatar diogoosorio avatar ewilded avatar gigkokman avatar gkanishk avatar grimd34th avatar jeffryadones avatar kalhama avatar linted avatar nvolcz avatar prabhath6 avatar sky-code avatar timkent avatar ufocoder avatar vortexau avatar wavvs avatar xuvez avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

vhostscan's Issues

Docker-Image doesn't build

...
x86_64-linux-gnu-gcc: numpy/core/src/multiarray/numpyos.c
numpy/core/src/multiarray/numpyos.c:18:10: fatal error: xlocale.h: No such file or directory
18 | #include <xlocale.h>
| ^~~~~~~~~~~
compilation terminated.

A typo in one of the imports keeps the tool from running

An attempt to run the tool after installation fails:

$ VHostScan

Traceback (most recent call last):
  File "/usr/local/bin/VHostScan", line 11, in <module>
    load_entry_point('VHostScan==1.21', 'console_scripts', 'VHostScan')()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 484, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2707, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2325, in load
    return self.resolve()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2331, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python2.7/dist-packages/VHostScan-1.21-py2.7.egg/VHostScan/VHostScan.py", line 8, in <module>
    from .lib.core.virtual_host_soutput_helpercanner import virtual_host_scanner
ImportError: No module named virtual_host_soutput_helpercanner

Reason: VHostScan.py, line 8:

from .lib.core.virtual_host_soutput_helpercanner import virtual_host_scanner

Do you have an idea for a solution?

Seems like the "output_helper" string crept into the class name.
Replacing the line with from .lib.core.virtual_host_scanner import virtual_host_scanner fixes the problem.

ImportError attempted relative import with no known parent package (is this python2 or python3?)

While following the instructions, I was having issues running python3 setup.py install due to the following error:

running install
running bdist_egg
running egg_info
writing VHostScan.egg-info/PKG-INFO
writing dependency_links to VHostScan.egg-info/dependency_links.txt
writing entry points to VHostScan.egg-info/entry_points.txt
writing requirements to VHostScan.egg-info/requires.txt
writing top-level names to VHostScan.egg-info/top_level.txt
reading manifest file 'VHostScan.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'VHostScan.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/VHostScan
copying build/lib/VHostScan/__init__.py -> build/bdist.linux-x86_64/egg/VHostScan
creating build/bdist.linux-x86_64/egg/VHostScan/wordlists
copying build/lib/VHostScan/wordlists/testing.txt -> build/bdist.linux-x86_64/egg/VHostScan/wordlists
copying build/lib/VHostScan/wordlists/simple.txt -> build/bdist.linux-x86_64/egg/VHostScan/wordlists
copying build/lib/VHostScan/wordlists/virtual-host-scanning.txt -> build/bdist.linux-x86_64/egg/VHostScan/wordlists
copying build/lib/VHostScan/wordlists/hackthebox.txt -> build/bdist.linux-x86_64/egg/VHostScan/wordlists
creating build/bdist.linux-x86_64/egg/VHostScan/lib
copying build/lib/VHostScan/lib/input.py -> build/bdist.linux-x86_64/egg/VHostScan/lib
creating build/bdist.linux-x86_64/egg/VHostScan/lib/helpers
copying build/lib/VHostScan/lib/helpers/output_helper.py -> build/bdist.linux-x86_64/egg/VHostScan/lib/helpers
copying build/lib/VHostScan/lib/helpers/file_helper.py -> build/bdist.linux-x86_64/egg/VHostScan/lib/helpers
copying build/lib/VHostScan/lib/helpers/__init__.py -> build/bdist.linux-x86_64/egg/VHostScan/lib/helpers
copying build/lib/VHostScan/lib/helpers/wordlist_helper.py -> build/bdist.linux-x86_64/egg/VHostScan/lib/helpers
copying build/lib/VHostScan/lib/__init__.py -> build/bdist.linux-x86_64/egg/VHostScan/lib
creating build/bdist.linux-x86_64/egg/VHostScan/lib/core
copying build/lib/VHostScan/lib/core/__version__.py -> build/bdist.linux-x86_64/egg/VHostScan/lib/core
copying build/lib/VHostScan/lib/core/__init__.py -> build/bdist.linux-x86_64/egg/VHostScan/lib/core
copying build/lib/VHostScan/lib/core/discovered_host.py -> build/bdist.linux-x86_64/egg/VHostScan/lib/core
copying build/lib/VHostScan/lib/core/virtual_host_scanner.py -> build/bdist.linux-x86_64/egg/VHostScan/lib/core
copying build/lib/VHostScan/lib/ua-random-list.txt -> build/bdist.linux-x86_64/egg/VHostScan/lib
copying build/lib/VHostScan/VHostScan.py -> build/bdist.linux-x86_64/egg/VHostScan
creating build/bdist.linux-x86_64/egg/tests
copying build/lib/tests/test_input.py -> build/bdist.linux-x86_64/egg/tests
creating build/bdist.linux-x86_64/egg/tests/helpers
copying build/lib/tests/helpers/test_file_helper.py -> build/bdist.linux-x86_64/egg/tests/helpers
copying build/lib/tests/helpers/__init__.py -> build/bdist.linux-x86_64/egg/tests/helpers
copying build/lib/tests/helpers/test_wordlist_helper.py -> build/bdist.linux-x86_64/egg/tests/helpers
copying build/lib/tests/conftest.py -> build/bdist.linux-x86_64/egg/tests
copying build/lib/tests/__init__.py -> build/bdist.linux-x86_64/egg/tests
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/__init__.py to __init__.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/input.py to input.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/helpers/output_helper.py to output_helper.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/helpers/file_helper.py to file_helper.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/helpers/__init__.py to __init__.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/helpers/wordlist_helper.py to wordlist_helper.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/__init__.py to __init__.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/core/__version__.py to __version__.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/core/__init__.py to __init__.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/core/discovered_host.py to discovered_host.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/core/virtual_host_scanner.py to virtual_host_scanner.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/VHostScan.py to VHostScan.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/test_input.py to test_input.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/helpers/test_file_helper.py to test_file_helper.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/helpers/__init__.py to __init__.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/helpers/test_wordlist_helper.py to test_wordlist_helper.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/conftest.py to conftest.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/__init__.py to __init__.cpython-38.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying VHostScan.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying VHostScan.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying VHostScan.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying VHostScan.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying VHostScan.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying VHostScan.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
creating 'dist/VHostScan-1.21-py3.8.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing VHostScan-1.21-py3.8.egg
removing '/usr/local/lib/python3.8/dist-packages/VHostScan-1.21-py3.8.egg' (and everything under it)
creating /usr/local/lib/python3.8/dist-packages/VHostScan-1.21-py3.8.egg
Extracting VHostScan-1.21-py3.8.egg to /usr/local/lib/python3.8/dist-packages
VHostScan 1.21 is already the active version in easy-install.pth
Installing VHostScan script to /usr/local/bin

Installed /usr/local/lib/python3.8/dist-packages/VHostScan-1.21-py3.8.egg
Processing dependencies for VHostScan==1.21
Searching for pandas==0.19.2
Reading https://pypi.org/simple/pandas/
Downloading https://files.pythonhosted.org/packages/08/9d/31ec596099f14528fc6ad39428248ac5360f0bb5205a3ee79a5d1cf260fb/pandas-0.19.2.tar.gz#sha256=6f0f4f598c2b16746803c8bafef7c721c57e4844da752d36240c0acf97658014
Best match: pandas 0.19.2
Processing pandas-0.19.2.tar.gz
Writing /tmp/easy_install-gjwdn1sb/pandas-0.19.2/setup.cfg
Running pandas-0.19.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-gjwdn1sb/pandas-0.19.2/egg-dist-tmp-b1p8avd1
UPDATING build/lib.linux-x86_64-3.8/pandas/_version.py
set build/lib.linux-x86_64-3.8/pandas/_version.py to '0.19.2'

However, I ran across this GitHub issue which mentioned running python2 setup.py install, which is not in the instructions.

While that command completed and everything installed successfully, I now have issues trying to run VHostScan.py with both python2 and python3, as shown in the examples below:

 ⚡ root@kali  ~/test/VHostScan/VHostScan   master  ./VHostScan.py                                                                                                                                                                    5:19PM/06.02
Traceback (most recent call last):
  File "./VHostScan.py", line 8, in <module>
    from .lib.core.virtual_host_scanner import virtual_host_scanner
ImportError: attempted relative import with no known parent package
 ✘ ⚡ root@kali  ~/test/VHostScan/VHostScan   master  python2 ./VHostScan.py                                                                                                                                                          5:19PM/06.02
Traceback (most recent call last):
  File "./VHostScan.py", line 8, in <module>
    from .lib.core.virtual_host_scanner import virtual_host_scanner
ValueError: Attempted relative import in non-package
 ✘ ⚡ root@kali  ~/test/VHostScan/VHostScan   master  python3 ./VHostScan.py                                                                                                                                                          5:21PM/06.02
Traceback (most recent call last):
  File "./VHostScan.py", line 8, in <module>
    from .lib.core.virtual_host_scanner import virtual_host_scanner
ImportError: attempted relative import with no known parent package

Any help would be greatly appreciated.

DNS query issues

Scanning an IP resulted in this:

[+] Starting virtual host scan for 1.2.3.4 using port 80 and stdin, wordlists: 
[>] Ignoring HTTP codes: 404
Traceback (most recent call last):
  File "VHostScan.py", line 117, in <module>
    main()
  File "VHostScan.py", line 90, in main
    for ip in Resolver().query(arguments.target_hosts, 'A'):
  File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1051, in query
    raise NXDOMAIN(qnames=qnames_to_try, responses=nxdomain_responses)
dns.resolver.NXDOMAIN: None of DNS query names exist: 1.2.3.4., 1.2.3.4.dns.suffix.

Possibly because of missing PTR record?

It needs to be okay to have no records returned.

Create a CI recipe for the project

I've just opened a PR that introduces some tests to the project.

The next step would be to make sure that the test battery is run every time a PR is opened. A CI (continuous integration) system is great for automating this kind of job.

Tracis CI appears to be the de-facto service for this kind of work in the Github/open source community.

I'm up for writing the recipe for the system (for now to run the test battery each time a PR is opened), but the task would then require for a maintainer to open up an account on Travis CI (so that it would be associated with this repository).

Is this something worth pursuing? What are your thoughts about this?

Add oG (output grepable) support

After oN (output normal) has been added it will be considered whether an output grepable (oG) solution is also needed. If so this would function much the same as nmap providing an outputted file that can be quickly cut into segments using grep/awk/sed, etc'

Add oN (output normal) format

Two flags to be added to the initial release - oN, output in normal format. This will work similar to nmap in that it will prioritize a human readable format. Given the simplicity of the output this should still be grepable and remove the need for an oG output in the tool (for consideration after implementation).

output files implicit local directory

When specifying an output file with a flag like -oN, the local directory should be implicitly used

Right now you have to manually add the ./ before your output file name if you want it to be in the current directory. If you don't, an exception is thrown and no output file is written.

Check to make sure that directory is not empty before trying to stat/make it

This can easily be achieved by adding an if statement around the try/execpt block on line 18 of file_helper.py. Another option if you assume that most people will be outputing to another dir is to wrap the mkdir() call on line 21 with a similar if statement

Optional flag to cease scan on first 200 response for CTF items?

Although in a commercial setting catch-all's are quite likely this doesn't always appear to be the case in CTF items. In these cases you don't need vhostscan's catch-all/likepage filtering and instead would want to know about the first 200 response code found.

Potentially there's some value in a flag that will stop the scan and surface results on the first 200 response code that's hit? It's a minor item since they would find the content with a full scan anyway, but one to consider.

Compilation Error

Hello,

Getting compilation error while installing VHostScan

pandas/lib.c:4:10: fatal error: Python.h: No such file or directory #include "Python.h" ^~~~~~~~~~ compilation terminated. error: Setup script exited with error: command 'i686-linux-gnu-gcc' failed with exit status 1

Fix formatting on Json Output

We now have working json output but it's all on one line. I resolved this in a prior project (https://github.com/codingo/Ransomware-Json-Dataset) however I think it was a different library, simplejson. It may be solvable in the current implementation without adding new dependencies (current implementation is quite clean and shouldn't be rewritten). Merits investigation to see if cleanly formatted json can be outputted without a rewrite..

Allow wordlists to be piped via stdin for quicker vhost checks

Problem statement

After using the tool for a while I've found cases where I want to scan for a hostname and then run a quick vhost check. I'd also love a quicker way to check for a specific vhost without having to open burp repeater. Currently I've done this using vhostscan by pushing the vhostname to a wordlist file in /tmp, and then referencing it - by the time I've done that I might as well have used burp to do the check, which negates some of the value of vhostscan.

Proposed Solution

I think it would make more sense to allow stdin input for wordlists. This way, you could create / pass content without needing to create a file on the disk. If another wordlist(s) is provided it would be appended to the piped in dataset (however this should not happen with the default list unless it was specifically passed as a flag). #1 should be completed before implementing this as this would refactor a portion of the wordlist handling that would make this feature easier to implement in a logical way.

Usage Example

For example, on something like HackTheBox's retired bank machine I'd love to have been able to do quick checks using:

echo "bank.htb" | vhostscan.py -t 10.10.10.29

This would avoid running a full wordlist and allow for a very quick way to check for the presence of a vhost hosted at bank.htb on this machine, all from the terminal.

Simple waf bypass headers

By adding origin headers to all requests we can bypass some WAF products, for example:

X-Originating-IP: 127.0.0.1
X-Forwarded-For: 127.0.0.1
X-Remote-IP: 127.0.0.1
X-Remote-Addr: 127.0.0.1

Add setup file

Add setup file to add application to $PATH and install dependancies

Add support for aliases during catch-all scenarios

As the current codebase detects catch-all scenarios by hashing the pages and providing feedback to the user of the unique content this can be defeated when aliases are used (as the hash will match).

Support needs to be added to help detect these scenarios to insure incorrect information about unique pages isn't being served back to the end user.

--random-agent can't find file

What's the problem (or question)?

When using the --random-agent flag in release version 1.21, the program returns a FileNotFoundError:

[>] Random User-Agent flag set.
Traceback (most recent call last):
  File "/usr/local/bin/VHostScan", line 11, in <module>
    load_entry_point('VHostScan==1.8.3', 'console_scripts', 'VHostScan')()
  File "/usr/local/lib/python3.6/dist-packages/VHostScan-1.8.3-py3.6.egg/VHostScan/VHostScan.py", line 54, in main
    user_agents = load_random_user_agents()
  File "/usr/local/lib/python3.6/dist-packages/VHostScan-1.8.3-py3.6.egg/VHostScan/lib/helpers/file_helper.py", line 65, in load_random_user_agents
    with open('./lib/ua-random-list.txt') as f:
FileNotFoundError: [Errno 2] No such file or directory: './lib/ua-random-list.txt'

Running VHostScan from a parent directory that contains lib/ua-random-list.txt seems to solve the problem.

Do you have an idea for a solution?

Consider having the user manually specify the file containing the list of user agents, or modify the load_random_user_agents() function to specify the base path of where ua-random-list.txt should exist. One example fix:

def load_random_user_agents():
    dir = os.path.dirname(__file__)

    # in my case, this looks for the file in the directory above /usr/local/lib/python3.6/dist-packages/VHostScan-1.8.3-py3.6.egg/VHostScan/lib/helpers
    with open(os.path.join(dir, '../ua-random-list.txt')) as f:
        return f.readlines()

NOTE: When I modified this on my machine, I then received another error indicating an invalid header value:

Traceback (most recent call last):
  File "/usr/local/bin/VHostScan", line 11, in <module>
    load_entry_point('VHostScan==1.8.3', 'console_scripts', 'VHostScan')()
  File "/usr/local/lib/python3.6/dist-packages/VHostScan-1.8.3-py3.6.egg/VHostScan/VHostScan.py", line 101, in main
    scanner.scan()
  File "/usr/local/lib/python3.6/dist-packages/VHostScan-1.8.3-py3.6.egg/VHostScan/lib/core/virtual_host_scanner.py", line 147, in scan
    res = requests.get(dest_url, headers=headers, verify=False)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.10.0-py3.6.egg/requests/api.py", line 71, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.10.0-py3.6.egg/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.10.0-py3.6.egg/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.10.0-py3.6.egg/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.10.0-py3.6.egg/requests/adapters.py", line 403, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/requests-2.10.0-py3.6.egg/requests/packages/urllib3/connectionpool.py", line 578, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.10.0-py3.6.egg/requests/packages/urllib3/connectionpool.py", line 362, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1280, in _send_request
    self.putheader(hdr, value)
  File "/usr/lib/python3.6/http/client.py", line 1217, in putheader
    raise ValueError('Invalid header value %r' % (values[i],))
ValueError: Invalid header value b'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36\n'

I believe this other issue can be resolved by not reading in newline characters \n when reading the user agent file on line 66 of lib/helpers/file_helper.py. I can create another issue to discuss this if necessary.

How can we reproduce the issue?

  1. Install latest VHostScan release 1.21
  2. Run VHostScan with --random-agent flag against any domain from any directory vs. running it within <install_dir>/VHostScan-1.21/VHostScan/

What are the running context details?

  • Installation method (e.g. pip, apt-get, git clone or zip/tar.gz):

    • Downloaded *.tar.gz release 1.21 from GitHub. Installed with python3 setup.py install.
  • Client OS (e.g. Microsoft Windows 10):

    • Observed on both Kali 2019.2 and Ubuntu 18.04.2 LTS
  • Program version (see banner):

    • 1.21
  • Target Web Server if known (e.g. Apache):

    • Any
  • Detected WAF/IDS/IPS protection (e.g. ModSecurity or unknown):

    • n/a

ImportError: cannot import name parse_http_list

What's the problem (or question)?

After git clone and successful installation of all the requirements getting following error.

What are the running context details?

  • Installation method: git clone
  • Exception traceback (if any):
root@vps:~/VHostScan/# VHostScan
Traceback (most recent call last):
  File "/usr/local/bin/VHostScan", line 11, in <module>
    load_entry_point('VHostScan==1.21', 'console_scripts', 'VHostScan')()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 480, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2693, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2324, in load
    return self.resolve()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2330, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python2.7/dist-packages/VHostScan-1.21-py2.7.egg/VHostScan/VHostScan.py", line 8, in <module>
    from .lib.core.virtual_host_scanner import *
  File "/usr/local/lib/python2.7/dist-packages/VHostScan-1.21-py2.7.egg/VHostScan/lib/core/virtual_host_scanner.py", line 4, in <module>
    import requests
  File "/usr/local/lib/python2.7/dist-packages/requests-2.10.0-py2.7.egg/requests/__init__.py", line 64, in <module>
    from . import utils
  File "/usr/local/lib/python2.7/dist-packages/requests-2.10.0-py2.7.egg/requests/utils.py", line 24, in <module>
    from .compat import parse_http_list as _parse_list_header
  File "/usr/local/lib/python2.7/dist-packages/requests-2.10.0-py2.7.egg/requests/compat.py", line 38, in <module>
    from urllib2 import parse_http_list

Broken for HTTPS

HTTPS is broken because python-requests is handed the target, not the entry from the wordlist. This is fine for non-TLS sites, but breaks SNI.

The downside to handing the wordlist entry over is that python-requests will use the system resolver and go to where that points instead of the specified target.

Using -b (BASE_HOST) is also broken for HTTPS.

I believe it can be made to work with monkey patching as per here:
https://stackoverflow.com/questions/22609385/python-requests-library-define-specific-dns

Add support for random user agent and specified agent

Currently vhs will send a Chrome user-agent string. This issue proposes to have --random-agent to pick from a random array of user agents, along with --user-agent where the user can specify their owner user agent string. These could be approached in independent PR's and this issue split in two if preferable.

Change readme documentation to specifically say python3

The readme says python setup.py install when it should say python3 setup.py install

This will allow for people with both versions of python installed to copy and paste the commands and get the right version installed. The usage of pip should also be changed to pip3 just to avoid ambiguity.

Add verbose mode (-v --verbose)

At the moment feedback is cluttered when the scanner encounters a catch-all scenario. Proposal is to update it to track "Found x/y items" or a similar implementation unless verbose flag is added to the query in which case vhostscan will output all results as they are discovered.

Regardless, all results will be output to output results so a refactor of how the class passes and handles results will need to be completed first.

i have allready package and why error again????

RuntimeError: module compiled against API version 0xc but this version of numpy is 0xa
/usr/local/lib/python3.6/dist-packages/fuzzywuzzy-0.15.1-py3.6.egg/fuzzywuzzy/fuzz.py:35: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
+-+-+-+-+-+-+-+-+-+ v. 1.21
|V|H|o|s|t|S|c|a|n| Developed by @codingo_ & @__timk
+-+-+-+-+-+-+-+-+-+ https://github.com/codingo/VHostScan

usage: VHostScan [-h] -t TARGET_HOSTS [-w WORDLISTS] [-b BASE_HOST] [-p PORT]
[--prefix PREFIX] [--suffix SUFFIX] [-r REAL_PORT]
[--ignore-http-codes IGNORE_HTTP_CODES]
[--ignore-content-length IGNORE_CONTENT_LENGTH] [--first-hit]
[--unique-depth UNIQUE_DEPTH] [--ssl] [--fuzzy-logic]
[--no-lookups] [--rate-limit RATE_LIMIT] [--waf] [-v]
[-oN OUTPUT_NORMAL | -oJ OUTPUT_JSON | -oG OUTPUT_GREPABLE]
[--random-agent | --user-agent USER_AGENT]
VHostScan: error: the following arguments are required: -t

Unclear results (false positives?)

Discovering vhosts

My apologies if I am not using/understanding this tool properly, feel free to correct me and close this issue if irrelevant.
This tool got my interest as I simply wanted to automate dictionary-based detection of virtual hosts existing on a given web server (IP:PORT). So, my guess is that the principle this tool works is to keep connecting to the same webserver using different Host: headers and comparing the responses to each other. So if anything stands out, it indicates a successful detection of a new virtual host (which probably means a separate web root and separate configuration).

So I created a simple test case with a local Apache 2 installation. Below are the contents of the /etc/apache2/sites-enabled/000-default.conf file:

<VirtualHost *:80>
        ServerName 127.0.0.1
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html4
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
<VirtualHost *:80>
        ServerName localhost
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html2
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
<VirtualHost *:80>
        ServerName dev.example.org
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html3
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
<VirtualHost *:80>
        ServerName dev
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

The file defines four separate virtual hosts, each with a different webroot:

  • 127.0.0.1 -> /var/www/html4
  • localhost -> /var/www/html2
  • dev.example.org -> /var/www/html3
  • dev -> /var/www/html

Each webroot contains a different default page, which gets displayed accordingly to how I manipulate the host header in Burp's Repeater, with /var/www/html4 being served for any value different than 'localhost','dev.example.org' and 'dev' (making it the default vhost).

Then, I scanned localhost with a small wordlist, making sure that it contained all the vhosts I defined:

foo
bar
nothing
invalid
localhost
127.0.0.2
example.org
something.something
dev
somethingelse
dev.example.org
andsoon
nosuchhost
blablabla

The result

VHostScan -w wordlist.txt -t 127.0.0.1 -p 80
/usr/local/lib/python2.7/dist-packages/fuzzywuzzy-0.15.1-py2.7.egg/fuzzywuzzy/fuzz.py:35: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
  warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
+-+-+-+-+-+-+-+-+-+  v. 1.21
|V|H|o|s|t|S|c|a|n|  Developed by @codingo_ & @__timk
+-+-+-+-+-+-+-+-+-+  https://github.com/codingo/VHostScan

[+] Starting virtual host scan for 127.0.0.1 using port 80 and wordlists: wordlist.txt
[>] Ignoring HTTP codes: 404
[+] Resolving DNS for additional wordlist entries
[!] Couldn't find any records (NXDOMAIN)
[#] Found: foo (code: 200, length: 399, hash: afd749d3aaab964b10b9bd02aa208a004962a995cdb0bf4b379002e2cbceabf7)
  Date: Thu, 11 Oct 2018 12:17:20 GMT
  Server: Apache/2.4.34 (Debian)
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Length: 399
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html;charset=UTF-8

[#] Found: bar (code: 200, length: 398, hash: bef01edffcad980916a1e6066da7bc044aaefd974c66005cf032e25d84f84cf6)
  Date: Thu, 11 Oct 2018 12:17:20 GMT
  Server: Apache/2.4.34 (Debian)
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Length: 398
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html;charset=UTF-8

[#] Found: nothing (code: 200, length: 405, hash: 58ba922d7580747b9983d1f86a86f6879893d257f056bd4f76f7efccec12afe5)
  Date: Thu, 11 Oct 2018 12:17:20 GMT
  Server: Apache/2.4.34 (Debian)
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Length: 405
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html;charset=UTF-8

[#] Found: invalid (code: 200, length: 400, hash: 1bd6e4a459ec75e597de0ca49f4692dd6bf34bd503d3dcb0055f238e109de6b8)
  Date: Thu, 11 Oct 2018 12:17:20 GMT
  Server: Apache/2.4.34 (Debian)
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Length: 400
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html;charset=UTF-8

[#] Found: localhost (code: 200, length: 5, hash: c9d04c9565fc665c80681fb1d829938026871f66e14f501e08531df66938a789)
  Date: Thu, 11 Oct 2018 12:17:20 GMT
  Server: Apache/2.4.34 (Debian)
  Last-Modified: Wed, 10 Oct 2018 17:20:10 GMT
  ETag: "5-577e311029df9"
  Accept-Ranges: bytes
  Content-Length: 5
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html

[#] Found: 127.0.0.2 (code: 200, length: 403, hash: ef68f0ec1c6c1036c595ec4a9fcd9f93ff268829bfe201beee6a33e4c602f1e4)
  Date: Thu, 11 Oct 2018 12:17:20 GMT
  Server: Apache/2.4.34 (Debian)
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Length: 403
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html;charset=UTF-8

[#] Found: example.org (code: 200, length: 408, hash: bc6eefe1bb3d40fd6eecbdba5231fe62f877e2f8916a007f88dec3db3babc20c)
  Date: Thu, 11 Oct 2018 12:17:20 GMT
  Server: Apache/2.4.34 (Debian)
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Length: 408
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html;charset=UTF-8

[#] Found: something.something (code: 200, length: 409, hash: 5b84da62c2a41e6f44c48f055249a3f235856c4100773f23ff10c9500ff39d2f)
  Date: Thu, 11 Oct 2018 12:17:20 GMT
  Server: Apache/2.4.34 (Debian)
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Length: 409
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html;charset=UTF-8

[#] Found: dev (code: 200, length: 563, hash: ae81f9837428f1aa79b94a3b7f809198074334ba498a87821354806980f7fec4)
  Date: Thu, 11 Oct 2018 12:17:20 GMT
  Server: Apache/2.4.34 (Debian)
  Last-Modified: Tue, 07 Aug 2018 13:40:05 GMT
  ETag: "3a6-572d888143908-gzip"
  Accept-Ranges: bytes
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Length: 563
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html

[#] Found: somethingelse (code: 200, length: 409, hash: 5e6a43b80f4ce19d42df792d4c9a8b954a7d6831bcdcaa273be4d3893f41a80e)
  Date: Thu, 11 Oct 2018 12:17:20 GMT
  Server: Apache/2.4.34 (Debian)
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Length: 409
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html;charset=UTF-8

[#] Found: dev.example.org (code: 200, length: 334, hash: 896877f13d3f78b00dbd7e3c529b8e2f4e3c583e52fefe031df06baee7653f99)
  Date: Thu, 11 Oct 2018 12:17:20 GMT
  Server: Apache/2.4.34 (Debian)
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Length: 334
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html;charset=UTF-8

[#] Found: andsoon (code: 200, length: 405, hash: fb9b03a379483f20f76d8676de17060cecbfc70674243aee81d25d21ac8b6539)
  Date: Thu, 11 Oct 2018 12:17:20 GMT
  Server: Apache/2.4.34 (Debian)
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Length: 405
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html;charset=UTF-8

[#] Found: nosuchhost (code: 200, length: 408, hash: 4e3b1afa8df7ffbf0d65de1bf83cb616d4f38f11df38a7b80660690b769d9ac5)
  Date: Thu, 11 Oct 2018 12:17:20 GMT
  Server: Apache/2.4.34 (Debian)
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Length: 408
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html;charset=UTF-8

[#] Found: blablabla (code: 200, length: 400, hash: 888c4287730e65eded0c3d540a3f9ad3204fc8b8d5edc0b0aca78262c8a014f9)
  Date: Thu, 11 Oct 2018 12:17:20 GMT
  Server: Apache/2.4.34 (Debian)
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Length: 400
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html;charset=UTF-8


[+] Most likely matches with a unique count of 1 or less:
        [>] somethingelse
        [>] bar
        [>] dev.example.org
        [>] andsoon
        [>] invalid
        [>] example.org
        [>] nosuchhost
        [>] dev
        [>] blablabla
        [>] nothing
        [>] something.something
        [>] foo
        [>] 127.0.0.2
        [>] localhost

So, it simply returned the entire wordlist as valid vhosts. The result I would like to see would be rather:

  1. Unique vhosts:
    [>] dev
    [>] localhost
    [>] dev.example.org

  2. Everything else:
    [>] somethingelse
    [>] bar
    [>] andsoon
    [>] invalid
    [>] example.org
    [>] nosuchhost
    [>] blablabla
    [>] nothing
    [>] something.something
    [>] foo
    [>] 127.0.0.2

The result I am getting in its current form is not helping me at all :)

Do you have an idea for a solution?

I know I could grep the output and group it by the length, but this sounds like an overkill and a bit defeats the point of using a tool dedicated for this purpose in the first place.

Before I try to modify the source code, I thought I would ask first - which is what I am doing :)
My guess is that successful identification of unique vhosts should boil down to comparing responses, using combination of basic properties like HTTP status code, length, number of words/letters (a hash will in most cases be always different, e.g. because of the common Date: header).

Another method is to pick a file that exists in the default vhost document root (e.g. /js/jquery.js) and then keep requesting it with different host headers. Once we hit 404/anything else than 200 or not modified, we know we have reached a different webroot (this will still leave other vhosts with the same webroot undetectable, but that's another story).
Please let me know what you think.
Thanks,
Julian

Import error when running vhostscan

vhostscan gives an import error when executed

  • Installation method (git clone): python setup.py install
  • Client OS: Linux loki 4.16.0-kali2-amd64 #1 SMP Debian 4.16.16-2kali2 (2018-07-04) x86_64 GNU/Linux
  • Program version (see banner): VHostScan 1.21
  • Relevant console output (if any):
    Traceback (most recent call last):
    File "./vhostscan.py", line 8, in
    from .lib.core.virtual_host_scanner import *
    ValueError: Attempted relative import in non-package

Detect catch-all scenarios with dynamic data and provide "likely matches" based on page differences

Current codebase helps to detect unique pages in catch-all scenarios by capturing the hashes of pages, and serving back the count of each hash (so you view pages which are unique).

This breaks down if something as simple as the time is served on the page causing it to be more dynamic, causing a difference in hashes. Proposal is to scrape page data using requests and work out the overlap between pages to identify the domains that are distinctly different to the others.

Add test cases for wordlists and STDIN

Now that we have some automated test cases in place it would be good to create some cases for the following scenarios to ensure good code coverage of the wordlist code:

  • STDIN only
  • STDIN + wordlist
  • No STDIN, default wordlist (no flags, will automatically select)
  • No STDIN, provided wordlist

These could be separate, or a single test case.

setup.py cant finish installation

Hi, Im trying to install VHostScan on my kali linux virtual machine with command
python3 setup.py install and im getting an error.
My python 3 version: Python 3.7.4

My versions of pip libs installed:

Name: dnspython
Version: 1.16.0
Name: fuzzywuzzy
Version: 0.17.0
Name: numpy
Version: 1.12.0
Name: pandas
Version: 0.24.2
Name: requests
Version: 2.21.0
Name: simplejson
Version: 3.16.0
Name: urllib3
Version: 1.24.1

And here is the console output after running install command:

running install
running bdist_egg
running egg_info
writing VHostScan.egg-info/PKG-INFO
writing dependency_links to VHostScan.egg-info/dependency_links.txt
writing entry points to VHostScan.egg-info/entry_points.txt
writing requirements to VHostScan.egg-info/requires.txt
writing top-level names to VHostScan.egg-info/top_level.txt
reading manifest file 'VHostScan.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'VHostScan.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/tests
copying build/lib/tests/__init__.py -> build/bdist.linux-x86_64/egg/tests
copying build/lib/tests/test_input.py -> build/bdist.linux-x86_64/egg/tests
creating build/bdist.linux-x86_64/egg/tests/helpers
copying build/lib/tests/helpers/__init__.py -> build/bdist.linux-x86_64/egg/tests/helpers
copying build/lib/tests/helpers/test_file_helper.py -> build/bdist.linux-x86_64/egg/tests/helpers
copying build/lib/tests/helpers/test_wordlist_helper.py -> build/bdist.linux-x86_64/egg/tests/helpers
copying build/lib/tests/conftest.py -> build/bdist.linux-x86_64/egg/tests
creating build/bdist.linux-x86_64/egg/VHostScan
copying build/lib/VHostScan/__init__.py -> build/bdist.linux-x86_64/egg/VHostScan
copying build/lib/VHostScan/VHostScan.py -> build/bdist.linux-x86_64/egg/VHostScan
creating build/bdist.linux-x86_64/egg/VHostScan/lib
copying build/lib/VHostScan/lib/__init__.py -> build/bdist.linux-x86_64/egg/VHostScan/lib
creating build/bdist.linux-x86_64/egg/VHostScan/lib/core
copying build/lib/VHostScan/lib/core/discovered_host.py -> build/bdist.linux-x86_64/egg/VHostScan/lib/core
copying build/lib/VHostScan/lib/core/__init__.py -> build/bdist.linux-x86_64/egg/VHostScan/lib/core
copying build/lib/VHostScan/lib/core/virtual_host_scanner.py -> build/bdist.linux-x86_64/egg/VHostScan/lib/core
copying build/lib/VHostScan/lib/core/__version__.py -> build/bdist.linux-x86_64/egg/VHostScan/lib/core
creating build/bdist.linux-x86_64/egg/VHostScan/lib/helpers
copying build/lib/VHostScan/lib/helpers/__init__.py -> build/bdist.linux-x86_64/egg/VHostScan/lib/helpers
copying build/lib/VHostScan/lib/helpers/wordlist_helper.py -> build/bdist.linux-x86_64/egg/VHostScan/lib/helpers
copying build/lib/VHostScan/lib/helpers/file_helper.py -> build/bdist.linux-x86_64/egg/VHostScan/lib/helpers
copying build/lib/VHostScan/lib/helpers/output_helper.py -> build/bdist.linux-x86_64/egg/VHostScan/lib/helpers
copying build/lib/VHostScan/lib/ua-random-list.txt -> build/bdist.linux-x86_64/egg/VHostScan/lib
copying build/lib/VHostScan/lib/input.py -> build/bdist.linux-x86_64/egg/VHostScan/lib
creating build/bdist.linux-x86_64/egg/VHostScan/wordlists
copying build/lib/VHostScan/wordlists/testing.txt -> build/bdist.linux-x86_64/egg/VHostScan/wordlists
copying build/lib/VHostScan/wordlists/hackthebox.txt -> build/bdist.linux-x86_64/egg/VHostScan/wordlists
copying build/lib/VHostScan/wordlists/simple.txt -> build/bdist.linux-x86_64/egg/VHostScan/wordlists
copying build/lib/VHostScan/wordlists/virtual-host-scanning.txt -> build/bdist.linux-x86_64/egg/VHostScan/wordlists
byte-compiling build/bdist.linux-x86_64/egg/tests/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/test_input.py to test_input.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/helpers/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/helpers/test_file_helper.py to test_file_helper.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/helpers/test_wordlist_helper.py to test_wordlist_helper.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/conftest.py to conftest.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/VHostScan.py to VHostScan.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/core/discovered_host.py to discovered_host.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/core/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/core/virtual_host_scanner.py to virtual_host_scanner.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/core/__version__.py to __version__.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/helpers/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/helpers/wordlist_helper.py to wordlist_helper.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/helpers/file_helper.py to file_helper.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/helpers/output_helper.py to output_helper.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/VHostScan/lib/input.py to input.cpython-37.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying VHostScan.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying VHostScan.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying VHostScan.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying VHostScan.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying VHostScan.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying VHostScan.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
tests.__pycache__.conftest.cpython-37: module references __file__
creating 'dist/VHostScan-1.21-py3.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing VHostScan-1.21-py3.7.egg
removing '/usr/local/lib/python3.7/dist-packages/VHostScan-1.21-py3.7.egg' (and everything under it)
creating /usr/local/lib/python3.7/dist-packages/VHostScan-1.21-py3.7.egg
Extracting VHostScan-1.21-py3.7.egg to /usr/local/lib/python3.7/dist-packages
VHostScan 1.21 is already the active version in easy-install.pth
Installing VHostScan script to /usr/local/bin

Installed /usr/local/lib/python3.7/dist-packages/VHostScan-1.21-py3.7.egg
Processing dependencies for VHostScan==1.21
Searching for pandas==0.19.2
Reading https://pypi.org/simple/pandas/
Downloading https://files.pythonhosted.org/packages/08/9d/31ec596099f14528fc6ad39428248ac5360f0bb5205a3ee79a5d1cf260fb/pandas-0.19.2.tar.gz#sha256=6f0f4f598c2b16746803c8bafef7c721c57e4844da752d36240c0acf97658014
Best match: pandas 0.19.2
Processing pandas-0.19.2.tar.gz
Writing /tmp/easy_install-fxu1xzgv/pandas-0.19.2/setup.cfg
Running pandas-0.19.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-fxu1xzgv/pandas-0.19.2/egg-dist-tmp-hnrj4t85
package init file 'pandas/io/tests/sas/__init__.py' not found (or not a regular file)
warning: no files found matching 'README.rst'
no previously-included directories found matching 'doc/build'
warning: no directories found matching 'examples'
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching '*.pyd' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '#*' found anywhere in distribution
warning: no previously-included files matching '.git*' found anywhere in distribution
warning: no previously-included files matching '.DS_Store' found anywhere in distribution
warning: no previously-included files matching '*.png' found anywhere in distribution
package init file 'pandas/io/tests/sas/__init__.py' not found (or not a regular file)
package init file 'pandas/io/tests/sas/__init__.py' not found (or not a regular file)
UPDATING build/lib.linux-x86_64-3.7/pandas/_version.py
set build/lib.linux-x86_64-3.7/pandas/_version.py to '0.19.2'
In file included from /usr/lib/python3/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
                 from /usr/lib/python3/dist-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /usr/lib/python3/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from pandas/lib.c:274:
/usr/lib/python3/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it with " \
  ^~~~~~~
In file included from pandas/lib.c:280:
pandas/src/numpy_helper.h: In function ‘transfer_object_column’:
pandas/src/numpy_helper.h:152:19: warning: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare]
     for (i = 0; i < length; ++i)
                   ^
In file included from /usr/include/python3.7m/Python.h:120,
                 from pandas/lib.c:4:
pandas/lib.c: In function ‘__pyx_f_6pandas_3lib_slice_get_indices_ex’:
pandas/lib.c:28309:37: warning: passing argument 1 of ‘PySlice_Unpack’ from incompatible pointer type [-Wincompatible-pointer-types]
   __pyx_t_4 = PySlice_GetIndicesEx(((PySliceObject *)__pyx_v_slc), __pyx_v_objlen, (&__pyx_v_start), (&__pyx_v_stop), (&__pyx_v_step), (&__pyx_v_length)); if (unlikely(__pyx_t_4 == -1)) __PYX_ERR(0, 1723, __pyx_L1_error)
/usr/include/python3.7m/sliceobject.h:49:21: note: in definition of macro ‘PySlice_GetIndicesEx’
     PySlice_Unpack((slice), (start), (stop), (step)) < 0 ?                  \
                     ^~~~~
/usr/include/python3.7m/sliceobject.h:53:42: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘PySliceObject *’ {aka ‘struct <anonymous> *’}
 PyAPI_FUNC(int) PySlice_Unpack(PyObject *slice,
                                ~~~~~~~~~~^~~~~
pandas/lib.c: In function ‘__pyx_f_6pandas_3lib_slice_len’:
pandas/lib.c:28532:37: warning: passing argument 1 of ‘PySlice_Unpack’ from incompatible pointer type [-Wincompatible-pointer-types]
   __pyx_t_4 = PySlice_GetIndicesEx(((PySliceObject *)__pyx_v_slc), __pyx_v_objlen, (&__pyx_v_start), (&__pyx_v_stop), (&__pyx_v_step), (&__pyx_v_length)); if (unlikely(__pyx_t_4 == -1)) __PYX_ERR(0, 1747, __pyx_L1_error)
/usr/include/python3.7m/sliceobject.h:49:21: note: in definition of macro ‘PySlice_GetIndicesEx’
     PySlice_Unpack((slice), (start), (stop), (step)) < 0 ?                  \
                     ^~~~~
/usr/include/python3.7m/sliceobject.h:53:42: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘PySliceObject *’ {aka ‘struct <anonymous> *’}
 PyAPI_FUNC(int) PySlice_Unpack(PyObject *slice,
                                ~~~~~~~~~~^~~~~
pandas/lib.c: In function ‘__Pyx__ExceptionSave’:
pandas/lib.c:91809:21: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
     *type = tstate->exc_type;
                     ^~~~~~~~
                     curexc_type
pandas/lib.c:91810:22: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
     *value = tstate->exc_value;
                      ^~~~~~~~~
                      curexc_value
pandas/lib.c:91811:19: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
     *tb = tstate->exc_traceback;
                   ^~~~~~~~~~~~~
                   curexc_traceback
pandas/lib.c: In function ‘__Pyx__ExceptionReset’:
pandas/lib.c:91818:24: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
     tmp_type = tstate->exc_type;
                        ^~~~~~~~
                        curexc_type
pandas/lib.c:91819:25: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
     tmp_value = tstate->exc_value;
                         ^~~~~~~~~
                         curexc_value
pandas/lib.c:91820:22: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
     tmp_tb = tstate->exc_traceback;
                      ^~~~~~~~~~~~~
                      curexc_traceback
pandas/lib.c:91821:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
     tstate->exc_type = type;
             ^~~~~~~~
             curexc_type
pandas/lib.c:91822:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
     tstate->exc_value = value;
             ^~~~~~~~~
             curexc_value
pandas/lib.c:91823:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
     tstate->exc_traceback = tb;
             ^~~~~~~~~~~~~
             curexc_traceback
pandas/lib.c: In function ‘__Pyx__GetException’:
pandas/lib.c:92129:24: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
     tmp_type = tstate->exc_type;
                        ^~~~~~~~
                        curexc_type
pandas/lib.c:92130:25: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
     tmp_value = tstate->exc_value;
                         ^~~~~~~~~
                         curexc_value
pandas/lib.c:92131:22: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
     tmp_tb = tstate->exc_traceback;
                      ^~~~~~~~~~~~~
                      curexc_traceback
pandas/lib.c:92132:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
     tstate->exc_type = local_type;
             ^~~~~~~~
             curexc_type
pandas/lib.c:92133:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
     tstate->exc_value = local_value;
             ^~~~~~~~~
             curexc_value
pandas/lib.c:92134:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
     tstate->exc_traceback = local_tb;
             ^~~~~~~~~~~~~
             curexc_traceback
pandas/lib.c: In function ‘__Pyx__ExceptionSwap’:
pandas/lib.c:92975:24: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
     tmp_type = tstate->exc_type;
                        ^~~~~~~~
                        curexc_type
pandas/lib.c:92976:25: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
     tmp_value = tstate->exc_value;
                         ^~~~~~~~~
                         curexc_value
pandas/lib.c:92977:22: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
     tmp_tb = tstate->exc_traceback;
                      ^~~~~~~~~~~~~
                      curexc_traceback
pandas/lib.c:92978:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
     tstate->exc_type = *type;
             ^~~~~~~~
             curexc_type
pandas/lib.c:92979:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
     tstate->exc_value = *value;
             ^~~~~~~~~
             curexc_value
pandas/lib.c:92980:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
     tstate->exc_traceback = *tb;
             ^~~~~~~~~~~~~
             curexc_traceback
error: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

IGNORE.

Did some research on this specific error, and I simply did as people were saying to do:

Change:
from queue import Queue
To:
from multiprocessing import Queue

Apparently it gets confused. Works now!

HTTPS requests working with NGINX but may have an issue with Apache

HTTPS is working against my nginx server however I tried against apache and received the following response:

b'<!DOCTYPE` HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>400 Bad Request</title>\n</head><body>\n<h1>Bad Request</h1>\n<p>Your browser sent a request that this server could not understand.<br />\n</p>\n</body></html>\n'

Need to investigate what our request is doing and compare to standard.

Automated test case (requirements) is failing

#57 introduced automated test cases but one of these appears to be failing. Despite the correct Python version appearing to be used the numpy install is complaining about it:

$ python --version
Python 3.2.6
$ pip --version
pip 7.1.2 from /home/travis/virtualenv/python3.2.6/lib/python3.2/site-packages (python 3.2)
3.24s$ pip install -r ./requirements.txt
Collecting dnspython==1.15.0 (from -r ./requirements.txt (line 1))
Downloading dnspython-1.15.0-py2.py3-none-any.whl (177kB)
100% |████████████████████████████████| 180kB 1.8MB/s
Collecting fuzzywuzzy==0.15.1 (from -r ./requirements.txt (line 2))
Downloading fuzzywuzzy-0.15.1-py2.py3-none-any.whl
Collecting numpy==1.12.0 (from -r ./requirements.txt (line 3))
Downloading numpy-1.12.0.zip (4.8MB)
100% |████████████████████████████████| 4.8MB 101kB/s
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 20, in
File "/tmp/pip-build-qtb0e3/numpy/setup.py", line 34, in
raise RuntimeError("Python version 2.7 or >= 3.4 required.")
RuntimeError: Python version 2.7 or >= 3.4 required.

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-qtb0e3/numpy

Creation dictionary from SSL cert for inclusion in scans

Often hostnames can be found within SSL certificates. This issue proposes to add a "--include-ssl-terms" or something better named to tell vhostscan to pickup and create a dictionary from located ssl certificates to append to any wordlist scan prior to when matching / filtering is done.

Coloured output

Update output to colourize output and improve feedback to the user. Will also require a --no-color to avoid coloured output.

Issue on Output Json

What's the problem (or question)?

  1. Problem on is_json() function in helpers/file_helper.py

Do you have an idea for a solution?

  1. Adding self because it's function based on class, and removed the open(json_file,'r')
    because OS cannot handled bigger file.
    So the new file_helper.py would be like this:
...
...
def is_json(self, json_file):
        try:
            print(json_file)
            '''
            using open for big file can result Error
            '''
            # with open(json_file, "r") as f:
            #     json_object = json.load(f)
           
           
        except ValueError:
            return False
        return True
...
...

How can we reproduce the issue?

  1. Run the VHostScan and give as the output Json `VHostScan -t -oJ .
  2. It will show an error TypeError: is_json() takes 1 positional argument but 2 were given

What are the running context details?

  • Exception traceback (if any):
if not file.is_json(output):
TypeError: is_json() takes 1 positional argument but 2 were given

Cannot Install

What's the problem (or question)?

I can't install using the command: python3 setup.py install

Do you have an idea for a solution?

Nope

How can we reproduce the issue?

  1. Download and install the tool

What are the running context details?

  • Installation method python3 setup.py install
  • Client OS: Linux kali 5.8.0-kali2-amd64
  • Program version (see banner): Latest (downloaded using git clone)
  • Error: (attached)
    error.txt

Error install requirements.txt for pandas==0.19.2

I always get this issue when installing the requirements.
`Collecting pandas==0.19.2
Using cached pandas-0.19.2.tar.gz (9.2 MB)
ERROR: Command errored out with exit status 1:
command: /data/data/com.termux/files/usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/data/data/com.termux/files/usr/tmp/pip-install-t2nuranx/pandas_e6beddb6780b47f8b67d929c3c7ad4ef/setup.py'"'"'; file='"'"'/data/data/com.termux/files/usr/tmp/pip-install-t2nuranx/pandas_e6beddb6780b47f8b67d929c3c7ad4ef/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /data/data/com.termux/files/usr/tmp/pip-pip-egg-info-gd0_3j11
cwd: /data/data/com.termux/files/usr/tmp/pip-install-t2nuranx/pandas_e6beddb6780b47f8b67d929c3c7ad4ef/
Complete output (113 lines):
ERROR: Exception:
Traceback (most recent call last):
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 171, in _merge_into_criterion
crit = self.state.criteria[name]
KeyError: 'numpy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_vendor/urllib3/response.py", line 438, in _error_catcher
    yield
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_vendor/urllib3/response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 62, in read
    data = self.__fp.read(amt)
  File "/data/data/com.termux/files/usr/lib/python3.9/http/client.py", line 458, in read
    n = self.readinto(b)
  File "/data/data/com.termux/files/usr/lib/python3.9/http/client.py", line 502, in readinto
    n = self.fp.readinto(b)
  File "/data/data/com.termux/files/usr/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/data/data/com.termux/files/usr/lib/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/data/data/com.termux/files/usr/lib/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 189, in _main
    status = self.run(options, args)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/commands/wheel.py", line 158, in run
    requirement_set = resolver.resolve(
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 121, in resolve
    self._result = resolver.resolve(
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 453, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 318, in resolve
    name, crit = self._merge_into_criterion(r, parent=None)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _merge_into_criterion
    crit = Criterion.from_requirement(self._p, requirement, parent)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 82, in from_requirement
    if not cands:
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_vendor/resolvelib/structs.py", line 124, in __bool__
    return bool(self._sequence)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in __bool__
    return any(self)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 38, in _iter_built
    candidate = func()
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 167, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 300, in __init__
    super().__init__(
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 144, in __init__
    self.dist = self._prepare()
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 226, in _prepare
    dist = self._prepare_distribution()
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 311, in _prepare_distribution
    return self._factory.preparer.prepare_linked_requirement(
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 457, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 480, in _prepare_linked_requirement
    local_file = unpack_url(
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 230, in unpack_url
    file = get_http_url(
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 108, in get_http_url
    from_path, content_type = download(link, temp_dir.path)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/network/download.py", line 163, in __call__
    for chunk in chunks:
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_internal/network/utils.py", line 64, in response_chunks
    for chunk in response.raw.stream(
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_vendor/urllib3/response.py", line 576, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_vendor/urllib3/response.py", line 541, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/data/data/com.termux/files/usr/lib/python3.9/contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pip/_vendor/urllib3/response.py", line 443, in _error_catcher
    raise ReadTimeoutError(self._pool, None, "Read timed out.")
pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/setuptools/installer.py", line 128, in fetch_build_egg
    subprocess.check_call(cmd)
  File "/data/data/com.termux/files/usr/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/data/data/com.termux/files/usr/bin/python3', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/data/data/com.termux/files/usr/tmp/tmpltq0oto6', '--quiet', 'numpy>=1.7.0']' returned non-zero exit status 2.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data/data/com.termux/files/usr/tmp/pip-install-t2nuranx/pandas_e6beddb6780b47f8b67d929c3c7ad4ef/setup.py", line 616, in <module>
    setup(name=DISTNAME,
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/setuptools/__init__.py", line 164, in setup
    _install_setup_requires(attrs)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/setuptools/__init__.py", line 159, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/setuptools/dist.py", line 699, in fetch_build_eggs
    resolved_dists = pkg_resources.working_set.resolve(
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 779, in resolve
    dist = best[req.key] = env.best_match(
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1064, in best_match
    return self.obtain(req, installer)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1076, in obtain
    return installer(requirement)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/setuptools/dist.py", line 758, in fetch_build_egg
    return fetch_build_egg(self, req)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/setuptools/installer.py", line 130, in fetch_build_egg
    raise DistutilsError(str(e)) from e
distutils.errors.DistutilsError: Command '['/data/data/com.termux/files/usr/bin/python3', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/data/data/com.termux/files/usr/tmp/tmpltq0oto6', '--quiet', 'numpy>=1.7.0']' returned non-zero exit status 2.
----------------------------------------

WARNING: Discarding https://files.pythonhosted.org/packages/08/9d/31ec596099f14528fc6ad39428248ac5360f0bb5205a3ee79a5d1cf260fb/pandas-0.19.2.tar.gz#sha256=6f0f4f598c2b16746803c8bafef7c721c57e4844da752d36240c0acf97658014 (from https://pypi.org/simple/pandas/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pandas==0.19.2
ERROR: No matching distribution found for pandas==0.19.2`

How to fix this? Because going on python3 setup.py install won't do either.

Add rate limiting feature

Currently VHostScan will fire off requests as fast as possible, this is probably not ideal for all situations.

Rate limiting or a configurable pause between requests would be a good feature to add.

Error: command 'i686-linux-gnu-gcc' failed with exit status 1

Can anyone help me troubleshoot this install issue?

warning: no previously-included files matching '*.png' found anywhere in distribution [388/1509]
package init file 'pandas/io/tests/sas/init.py' not found (or not a regular file)
package init file 'pandas/io/tests/sas/init.py' not found (or not a regular file)
UPDATING build/lib.linux-i686-3.7/pandas/_version.py
set build/lib.linux-i686-3.7/pandas/_version.py to '0.19.2'
In file included from /usr/lib/python3/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
from /usr/lib/python3/dist-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /usr/lib/python3/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
from pandas/lib.c:274:
/usr/lib/python3/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [
-Wcpp]
#warning "Using deprecated NumPy API, disable it with " \
^~~~~~~
In file included from pandas/lib.c:280:
pandas/src/numpy_helper.h: In function ‘transfer_object_column’:
pandas/src/numpy_helper.h:152:19: warning: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘unsigned int’} [-Wsign-compare]
for (i = 0; i < length; ++i)
^
In file included from /usr/include/python3.7m/Python.h:120,
from pandas/lib.c:4:
pandas/lib.c: In function ‘__pyx_f_6pandas_3lib_slice_get_indices_ex’:
pandas/lib.c:28309:37: warning: passing argument 1 of ‘PySlice_Unpack’ from incompatible pointer type [-Wincompatible-pointer-types]
__pyx_t_4 = PySlice_GetIndicesEx(((PySliceObject *)__pyx_v_slc), __pyx_v_objlen, (&__pyx_v_start), (&__pyx_v_stop), (&__pyx_v_step), (&__pyx_v_length)); if (unlikely(__pyx_t_4 == -1)) __PYX_ERR(0, 1723,
__pyx_L1_error)
/usr/include/python3.7m/sliceobject.h:49:21: note: in definition of macro ‘PySlice_GetIndicesEx’
PySlice_Unpack((slice), (start), (stop), (step)) < 0 ? \
^~~~~
/usr/include/python3.7m/sliceobject.h:53:42: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘PySliceObject *’ {aka ‘struct *’}
PyAPI_FUNC(int) PySlice_Unpack(PyObject *slice,
~~~~~~~~~~^~~~~
pandas/lib.c: In function ‘__pyx_f_6pandas_3lib_slice_len’:
pandas/lib.c:28532:37: warning: passing argument 1 of ‘PySlice_Unpack’ from incompatible pointer type [-Wincompatible-pointer-types]
__pyx_t_4 = PySlice_GetIndicesEx(((PySliceObject *)__pyx_v_slc), __pyx_v_objlen, (&__pyx_v_start), (&__pyx_v_stop), (&__pyx_v_step), (&__pyx_v_length)); if (unlikely(__pyx_t_4 == -1)) __PYX_ERR(0, 1747,
__pyx_L1_error)
/usr/include/python3.7m/sliceobject.h:49:21: note: in definition of macro ‘PySlice_GetIndicesEx’
PySlice_Unpack((slice), (start), (stop), (step)) < 0 ? \
^~~~~
/usr/include/python3.7m/sliceobject.h:53:42: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘PySliceObject *’ {aka ‘struct *’}
PyAPI_FUNC(int) PySlice_Unpack(PyObject *slice,
~~~~~~~~~~^~~~~
pandas/lib.c: In function ‘__Pyx__ExceptionSave’:
pandas/lib.c:91809:21: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?

pandas/lib.c: In function ‘__Pyx__ExceptionSave’:
pandas/lib.c:91809:21: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
*type = tstate->exc_type;
^~~~~~~~
curexc_type
pandas/lib.c:91810:22: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
*value = tstate->exc_value;
^~~~~~~~~
curexc_value
pandas/lib.c:91811:19: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
*tb = tstate->exc_traceback;
^~~~~~~~~~~~~
curexc_traceback
pandas/lib.c: In function ‘__Pyx__ExceptionReset’:
pandas/lib.c:91818:24: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
tmp_type = tstate->exc_type;
^~~~~~~~
curexc_type
pandas/lib.c:91819:25: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
tmp_value = tstate->exc_value;
^~~~~~~~~
curexc_value
pandas/lib.c:91820:22: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
tmp_tb = tstate->exc_traceback;
curexc_value
pandas/lib.c:91820:22: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
tmp_tb = tstate->exc_traceback;
^~~~~~~~~~~~~
curexc_traceback
pandas/lib.c:91821:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
tstate->exc_type = type;
^~~~~~~~
curexc_type
pandas/lib.c:91822:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
tstate->exc_value = value;
^~~~~~~~~
curexc_value
pandas/lib.c:91823:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
tstate->exc_traceback = tb;
^~~~~~~~~~~~~
curexc_traceback
pandas/lib.c: In function ‘__Pyx__GetException’:
pandas/lib.c:92129:24: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
tmp_type = tstate->exc_type;
^~~~~~~~
curexc_type
pandas/lib.c:92130:25: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
tmp_value = tstate->exc_value;
^~~~~~~~~
curexc_value
pandas/lib.c:92131:22: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
tmp_tb = tstate->exc_traceback;
^~~~~~~~~~~~~
curexc_traceback
pandas/lib.c:92132:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
tstate->exc_type = local_type;
^~~~~~~~
curexc_type
pandas/lib.c:92133:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
tstate->exc_value = local_value;

error: Setup script exited with error: command 'i686-linux-gnu-gcc' failed with exit status 1

Add command line flags as another object in JSON output

#50 resolves #15 however this output doesn't let you know the command line flags / header information used for the scan like output normal does.

Output normal uses the following code from output helper to do this:

    def generate_header(self):
        output = "VHostScanner Log: {} {}\n".format(time.strftime("%d/%m/%Y"), time.strftime("%H:%M:%S"))
        output += "\tTarget: {}\n\tBase Host: {}\n\tPort: {}".format(self.scanner.target, self.scanner.base_host, self.scanner.port)
        output += "\n\tReal Port {}\n\tIgnore HTTP Codes: {}".format(self.scanner.real_port,self.scanner.ignore_http_codes)
        output += "\n\tIgnore Content Length: {}\n\tWordlist: {}".format(self.scanner.ignore_content_length, self.scanner.wordlist)
        output += "\n\tUnique Depth: {}\n\tSSL: {}\n\t".format(self.scanner.unique_depth, self.scanner.ssl)
        return output

Ideally this would be included within the json file as well (as valid json) without being written to every scan object.

Add oJ (output json) format

Proposal for an oJ for output-json to output results in a json format to allow it to more easily overlap with other tools/dashboards.

Add "likely matches" to output that shows items which have unique hashes to input n (--unique-depth)

In catch-all scenarios vhostscan will first try to hash all of the pages it sees and then outputs "likely matches" based on a unique depth.

Unique depth defaults to one so if it finds a page with "one hit" it will display as a unique page. This breaks down if you have three aliases for the same item so the intent of this flag is to allow you to play with this threshold to identify more pages worth discovery. This doesn't work in cases of dynamic content (such as the time) being on the page as all catch-all wordlist items will have their own hash and @timkent is working on another issue to tackle these scenarios in conjunction with this.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.