Giter VIP home page Giter VIP logo

manspider's Introduction

MANSPIDER

Crawl SMB shares for juicy information. File content searching + regex is supported!

UPDATE 2023-10-15

textract, one of manspider's upstream dependencies, is no longer being updated. For this reason, you may run into problems when installing with pip. The recommended installation method is now Docker:

docker run blacklanternsecurity/manspider --help

Note there is also a helper script manspider.sh which will automatically mount volumes for manspider's loot and logs directories, making it a bit more convenient to run:

./manspider.sh --help

manspider

File types supported:

  • PDF
  • DOCX
  • XLSX
  • PPTX
  • any text-based format
  • and many more!!

MANSPIDER will crawl every share on every target system. If provided creds don't work, it will fall back to "guest", then to a null session.

manspider

Installation:

(Optional) Install these dependencies to add additional file parsing capability:

# for images (png, jpeg)
$ sudo apt install tesseract-ocr

# for legacy document support (.doc)
$ sudo apt install antiword

Install manspider (please be patient, this can take a while):

$ pip install pipx
$ pipx install git+https://github.com/blacklanternsecurity/MANSPIDER

Example #1: Search the network for filenames that may contain creds

NOTE: matching files are automatically downloaded into $HOME/.manspider/loot! (-n to disable)

$ manspider 192.168.0.0/24 -f passw user admin account network login logon cred -d evilcorp -u bob -p Passw0rd

Example #2: Search for spreadsheets with "password" in the filename

$ manspider share.evilcorp.local -f passw -e xlsx csv -d evilcorp -u bob -p Passw0rd

Example #3: Search for interesting file extensions

$ manspider share.evilcorp.local -e bat com vbs ps1 psd1 psm1 pem key rsa pub reg txt cfg conf config -d evilcorp -u bob -p Passw0rd

Example #4: Search for finance-related files

This example searches financy-sounding directories for filenames containing 5 or more consecutive numbers (e.g. 000202006.EFT)

$ manspider share.evilcorp.local --dirnames bank financ payable payment reconcil remit voucher vendor eft swift -f '[0-9]{5,}' -d evilcorp -u bob -p Passw0rd

Example #5: Search for SSH keys by filename

$ manspider share.evilcorp.local -e ppk rsa pem ssh rsa -o -f id_rsa id_dsa id_ed25519 -d evilcorp -u bob -p Passw0rd

Example #6: Search for SSH keys by content

$ manspider share.evilcorp.local -e '' -c 'BEGIN .{1,10} PRIVATE KEY' -d evilcorp -u bob -p Passw0rd

Example #7: Search for password manager files

# .kdbx - KeePass Password Database (KeePass, KeePassXC)
# .kdb - KeePass Classic Database (KeePass 1.x)
# .1pif - 1Password Interchange Format (1Password)
# .agilekeychain - Agile Keychain Format (1Password, deprecated)
# .opvault - OPVault Format (1Password)
# .lpd - LastPass Data File (LastPass)
# .dashlane - Dashlane Data File (Dashlane)
# .psafe3 - Password Safe Database (Password Safe)
# .enpass - Enpass Password Manager Data File (Enpass)
# .bwdb - Bitwarden Database (Bitwarden)
# .msecure - mSecure Password Manager Data File (mSecure)
# .stickypass - Sticky Password Data File (Sticky Password)
# .pwm - Password Memory Data File (Password Memory)
# .rdb - RoboForm Data File (RoboForm)
# .safe - SafeInCloud Password Manager Data File (SafeInCloud)
# .zps - Zoho Vault Encrypted Data File (Zoho Vault)
# .pmvault - SplashID Safe Data File (SplashID Safe)
# .mywallet - MyWallet Password Manager Data File (MyWallet)
# .jpass - JPass Password Manager Data File (JPass)
# .pwmdb - Universal Password Manager Database (Universal Password Manager)
$ manspider share.evilcorp.local -e kdbx kdb 1pif agilekeychain opvault lpd dashlane psafe3 enpass bwdb msecure stickypass pwm rdb safe zps pmvault mywallet jpass pwmdb -d evilcorp -u bob -p Passw0rd

Example #8: Search for certificates

$ manspider share.evilcorp.local -e pfx p12 pkcs12 pem key crt cer csr jks keystore key keys der -d evilcorp -u bob -p Passw0rd

Usage Tip #1:

You can run multiple instances of manspider at one time. This is useful when one instance is already running, and you want to search what it's downloaded (similar to grep -R). To do this, specify the keyword loot as the target, which will search the downloaded files in $HOME/.manspider/loot.

Usage Tip #2:

Reasonable defaults help prevent unwanted scenarios like getting stuck on a single target. All of these can be overridden:

  • default spider depth: 10 (override with -m)
  • default max filesize: 10MB (override with -s)
  • default threads: 5 (override with -t)
  • shares excluded: C$, IPC$, ADMIN$, PRINT$ (override with --exclude-sharenames)

Usage Tip #3:

Manspider accepts any combination of the following as targets:

  • IPs
  • hostnames
  • subnets (CIDR format)
  • files containing any of the above
  • local folders containing files

For example, you could specify any or all of these:

  • 192.168.1.250
  • share.evilcorp.local
  • 192.168.1.0/24
  • smb_hosts.txt
  • loot (to search already-downloaded files)
  • /mnt/share (to recursively search a directory)
    • NOTE: when searching local files, you must specify a directory, not an individual file

Usage:

usage: manspider [-h] [-u USERNAME] [-p PASSWORD] [-d DOMAIN] [-m MAXDEPTH] [-H HASH] [-t THREADS] [-f REGEX [REGEX ...]] [-e EXT [EXT ...]] [--exclude-extensions EXT [EXT ...]]
                 [-c REGEX [REGEX ...]] [--sharenames SHARE [SHARE ...]] [--exclude-sharenames [SHARE ...]] [--dirnames DIR [DIR ...]] [--exclude-dirnames DIR [DIR ...]] [-q] [-n]
                 [-mfail INT] [-o] [-s SIZE] [-v]
                 targets [targets ...]

Scan for juicy data on SMB shares. Matching files and logs are stored in $HOME/.manspider. All filters are case-insensitive.

positional arguments:
  targets               IPs, Hostnames, CIDR ranges, or files containing targets to spider (NOTE: local searching also supported, specify directory name or keyword "loot" to search
                        downloaded files)

optional arguments:
  -h, --help            show this help message and exit
  -u USERNAME, --username USERNAME
                        username for authentication
  -p PASSWORD, --password PASSWORD
                        password for authentication
  -d DOMAIN, --domain DOMAIN
                        domain for authentication
  -m MAXDEPTH, --maxdepth MAXDEPTH
                        maximum depth to spider (default: 10)
  -H HASH, --hash HASH  NTLM hash for authentication
  -t THREADS, --threads THREADS
                        concurrent threads (default: 5)
  -f REGEX [REGEX ...], --filenames REGEX [REGEX ...]
                        filter filenames using regex (space-separated)
  -e EXT [EXT ...], --extensions EXT [EXT ...]
                        only show filenames with these extensions (space-separated, e.g. `docx xlsx` for only word & excel docs)
  --exclude-extensions EXT [EXT ...]
                        ignore files with these extensions
  -c REGEX [REGEX ...], --content REGEX [REGEX ...]
                        search for file content using regex (multiple supported)
  --sharenames SHARE [SHARE ...]
                        only search shares with these names (multiple supported)
  --exclude-sharenames [SHARE ...]
                        don't search shares with these names (multiple supported)
  --dirnames DIR [DIR ...]
                        only search directories containing these strings (multiple supported)
  --exclude-dirnames DIR [DIR ...]
                        don't search directories containing these strings (multiple supported)
  -q, --quiet           don't display matching file content
  -n, --no-download     don't download matching files
  -mfail INT, --max-failed-logons INT
                        limit failed logons
  -o, --or-logic        use OR logic instead of AND (files are downloaded if filename OR extension OR content match)
  -s SIZE, --max-filesize SIZE
                        don't retrieve files over this size, e.g. "500K" or ".5M" (default: 10M)
  -v, --verbose         show debugging messages

manspider's People

Contributors

aconite33 avatar hynekpetrak avatar sebrink avatar thetechromancer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

manspider's Issues

not spidering dfs share

manspider domain.com --exclude-sharenames admin$ c$ d$ ipc$ netlogon p$ sysvol -d domain -u id -p pwd -e cmd bat ps1 -c password

this should find a share named fs, which it does, but it won't go into the share and drill down through sub directories and find files that I know match the -e and -c parameters.

i've tried with hostname.domain.com as well but that doesn't seem to work either. nothing is exported to the loot dir either. nothing interesting in -v output

Package installation with pip --require-hashes mode breaks dependencies on install

Upon attempting a default install with the below pip command:
python3 -m pip install -r requirements.txt
I am receiving the following error:

Collecting tzdata
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    tzdata from https://files.pythonhosted.org/packages/71/9b/8b9fea4f4dc956de76baa291cec1c864a8edadf2950d1740bc386d7fe55a/tzdata-2022.2-py2.py3-none-any.whl#sha256=c3119520447d68ef3eb8187a55a4f44fa455f30eb1b4238fa5691ba094f2b05b (from pytz-deprecation-shim==0.1.0.post0->-r requirements.txt (line 403))

I resolved this issue by flagging my original command with --no-deps, which (accordingly to official pip documentation, will proceed with the install and "Ignore package dependencies". This does not appear to be a long-term fix, but it should resolve the issue for anyone attempting to install requirements for the tool.

[Feature request] Add support for other authentication methods (PtH, OPtH, PtT, ...)

Hey there ๐Ÿ‘‹ Great tool, awesome work!
Since you base your code on Impacket, adding support for Pass-the-Hash, Overpass-the-Hash, Kerberos auth, Pass-the-Ticket shouldn't be too hard and could make this tool a Snaffler-like for UNIX-only penetration testers.
Since you probably know your code more than I do, here is a feature request but I'd be glad to help you out in this quest ๐Ÿ˜‰ do hesitate to reach out (https://twitter.com/_nwodtuhs)

manspider.py Module Import Error

When running man_spider.py, a module import error takes place:

Traceback (most recent call last): File "/root/tools/MANSPIDER/man_spider/manspider.py", line 8, in <module> from .lib import * ImportError: attempted relative import with no known parent package

The fix is to remove the . in .lib

Thanks!

start crawl from specific path

Hi,
Let's say I want to look inside sub-folders and files in this path only:
\\share.evilcorp.local\windows$\users\john\

It is possible? I did not find the correct command syntax for that.

Implement snaffler rules support

I know this is quite a big feature, but it would be nice if manspider and snaffler could share the looting rules.

Indeed Snaffler is very good when a Windows machine is available and Manspider could be the alternative with Linux, taking advantage of the quite extensive ruleset.

--dirnames needs work

Files in the root of the share are parsed even if --dirnames is specified.
If the dirname doesn't match at the root level, it's never found. E.g. if you specify --dirname logs, it will find /apache_logs, but not /var/apache_logs.

Just pulled the docker, no manspider.sh?

Full search of / doesn't find the file anywhere...nowhere on the filesystem or in the directory in which the pull was done (/opt/manspider).

find / -name manspider

/var/lib/docker/overlay2/1b696d278e07143f78a532582be1ecf9130363c46532c978940d9eb162fd390d/diff/usr/local/bin/manspider
/var/lib/docker/overlay2/1b696d278e07143f78a532582be1ecf9130363c46532c978940d9eb162fd390d/diff/usr/local/lib/python3.6/site-packages/man_spider/pycache/manspider.cpython-36.pyc
/var/lib/docker/overlay2/1b696d278e07143f78a532582be1ecf9130363c46532c978940d9eb162fd390d/diff/usr/local/lib/python3.6/site-packages/man_spider/manspider.py
/run/screen/S-root/2306464.manspider
find: โ€˜/run/user/125/gvfsโ€™: Permission denied
/opt/manspider

Ideas?

Licence

Hi Guys,

Could you specify the licence of the sw?

Thanks a lot.
Regards,
m

Got some error with installing with pipx

cat /proc/version
Linux version 5.4.0-121-generic (buildd@lcy02-amd64-013) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)) #137-Ubuntu SMP Wed Jun 15 13:33:07 UTC 2022

pipx install git+https://github.com/blacklanternsecurity/MANSPIDER
installed package man-spider 1.0.4, installed using Python 3.8.10
These apps are now globally available
- manspider
โš ๏ธ Note: '/root/.local/bin' is not on your PATH environment variable. These apps will not be globally accessible until your PATH is updated. Run pipx ensurepath to automatically add it, or manually modify your PATH in
your shell's config file (i.e. ~/.bashrc).
done! โœจ ๐ŸŒŸ โœจ

manspider
Traceback (most recent call last):
File "/root/.local/bin/manspider", line 5, in
from man_spider.manspider import main
File "/root/.local/pipx/venvs/man-spider/lib/python3.8/site-packages/man_spider/manspider.py", line 8, in
from .lib import *
File "/root/.local/pipx/venvs/man-spider/lib/python3.8/site-packages/man_spider/lib/init.py", line 1, in
from .parser import *
File "/root/.local/pipx/venvs/man-spider/lib/python3.8/site-packages/man_spider/lib/parser/init.py", line 1, in
from .parser import *
File "/root/.local/pipx/venvs/man-spider/lib/python3.8/site-packages/man_spider/lib/parser/parser.py", line 2, in
import magic
File "/root/.local/pipx/venvs/man-spider/lib/python3.8/site-packages/magic/init.py", line 209, in
libmagic = loader.load_lib()
File "/root/.local/pipx/venvs/man-spider/lib/python3.8/site-packages/magic/loader.py", line 49, in load_lib
raise ImportError('failed to find libmagic. Check your installation')
ImportError: failed to find libmagic. Check your installation

Top 20 password manager extensions courtesy of chatgpt

  • .kdbx - KeePass Password Database (KeePass, KeePassXC)
  • .kdb - KeePass Classic Database (KeePass 1.x)
  • .1pif - 1Password Interchange Format (1Password)
  • .agilekeychain - Agile Keychain Format (1Password, deprecated)
  • .opvault - OPVault Format (1Password)
  • .lpd - LastPass Data File (LastPass)
  • .dashlane - Dashlane Data File (Dashlane)
  • .psafe3 - Password Safe Database (Password Safe)
  • .enpass - Enpass Password Manager Data File (Enpass)
  • .bwdb - Bitwarden Database (Bitwarden)
  • .msecure - mSecure Password Manager Data File (mSecure)
  • .stickypass - Sticky Password Data File (Sticky Password)
  • .pwm - Password Memory Data File (Password Memory)
  • .rdb - RoboForm Data File (RoboForm)
  • .safe - SafeInCloud Password Manager Data File (SafeInCloud)
  • .zps - Zoho Vault Encrypted Data File (Zoho Vault)
  • .pmvault - SplashID Safe Data File (SplashID Safe)
  • .mywallet - MyWallet Password Manager Data File (MyWallet)
  • .jpass - JPass Password Manager Data File (JPass)
  • .pwmdb - Universal Password Manager Database (Universal Password Manager)

Manspider Loot Files in Docker

Hello,

How do I access the loot files after running manspider in docker?

I tried to find the ./manspider.sh file thats meant to help but cant find it. Can someone please help me?
Or can someone advice me how to access the docker container to grab the loot files?

Note there is also a helper script manspider.sh which will automatically mount volumes for manspider's loot and logs directories, making it a bit more convenient to run:

./manspider.sh --help

extract-mg Dependency Conflict

Running Kali Linux 2022.01 running. Python 3.9.10. APT is fully up-to-date. Proxychains works just fine. I am not using pipx. I activated a venv, git cloned the repo, and then ran the following:

sudo proxychains python3 -m pip install -r requirements
... # everything installed correctly until...
Collecting argcomplete==1.10.3
  Using cached argcomplete-1.10.3-py2.py3-none-any.whl (36 kB)
ERROR: Cannot install -r requirements.txt (line 417) and extract-msg==0.29.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested extract-msg==0.29.0
    textract 1.6.4 depends on extract-msg<=0.29.*

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

If you need more information, let me know.

C$ crawling disabled

Hey!

After using this tool for at least 2 years I just realized it doesn't crawl the c$ share.
Got it, it would be insanly long but I was wondering if it wouldn't be interesting to authorize the c:\users share as it will probably contain a lot of secrets (especially c:\users\whoami\desktop).

I haven't dug into the code but I was wondering two things:

  • Is this something you'd be keen to merge ?
  • How difficult would it be for me to help you on that task ?

Thanks in advance for the answer ;)
Deft_

Issues with packaging library, unable to install via pipx

Running Kali 2023.1 and python 3.11

I tried installing via pipx using the following

sudo apt install pipx -y
pipx install git+https://github.com/blacklanternsecurity/MANSPIDER

I get this error during the installation


Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/packaging/requirements.py", line 35, in __init__
    parsed = parse_requirement(requirement_string)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/packaging/_parser.py", line 64, in parse_requirement
    return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/packaging/_parser.py", line 82, in _parse_requirement
    url, specifier, marker = _parse_requirement_details(tokenizer)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/packaging/_parser.py", line 120, in _parse_requirement_details
    specifier = _parse_specifier(tokenizer)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/packaging/_parser.py", line 206, in _parse_specifier
    with tokenizer.enclosing_tokens("LEFT_PARENTHESIS", "RIGHT_PARENTHESIS"):
  File "/usr/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/usr/lib/python3/dist-packages/packaging/_tokenizer.py", line 183, in enclosing_tokens
    self.raise_syntax_error(
  File "/usr/lib/python3/dist-packages/packaging/_tokenizer.py", line 163, in raise_syntax_error
    raise ParserSyntaxError(
packaging._tokenizer.ParserSyntaxError: Expected closing RIGHT_PARENTHESIS
    extract-msg (<=0.29.*)
                ~~~~~~~^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/pipx", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/lib/python3/dist-packages/pipx/main.py", line 819, in cli
    return run_pipx_command(parsed_pipx_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pipx/main.py", line 202, in run_pipx_command
    return commands.install(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pipx/commands/install.py", line 60, in install
    venv.install_package(
  File "/usr/lib/python3/dist-packages/pipx/venv.py", line 244, in install_package
    self._update_package_metadata(
  File "/usr/lib/python3/dist-packages/pipx/venv.py", line 318, in _update_package_metadata
    venv_package_metadata = self.get_venv_metadata_for_package(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pipx/venv.py", line 300, in get_venv_metadata_for_package
    venv_metadata = inspect_venv(
                    ^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pipx/venv_inspect.py", line 251, in inspect_venv
    app_paths_of_dependencies = _dfs_package_apps(
                                ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pipx/venv_inspect.py", line 142, in _dfs_package_apps
    app_paths_of_dependencies = _dfs_package_apps(
                                ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pipx/venv_inspect.py", line 121, in _dfs_package_apps
    dependencies = get_package_dependencies(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pipx/venv_inspect.py", line 54, in get_package_dependencies
    for req in map(Requirement, dist.requires or []):
  File "/usr/lib/python3/dist-packages/packaging/requirements.py", line 37, in __init__
    raise InvalidRequirement(str(e)) from e
packaging.requirements.InvalidRequirement: Expected closing RIGHT_PARENTHESIS
    extract-msg (<=0.29.*)
                ~~~~~~~^

Is this an issue with the packaging library, and if so, can this be fixed to support later versions?

Improper module import of .lib

Line 8 of manspider.py appears to be referencing lib improperly. Currently it is referenced like so:

from .lib import *

The issue can be resolved by changing the affected import line to the following (aka removing the leading period):

from lib import *

Tag the source

Could you please tag the source? This allows distributions to get the complete source from GitHub if they want.

Thanks

Password special characters

When using -u fakeuser -p fakepass% returns zsh:%with no previous word matched. So a password can't contain certain characters.

Figured out using -p 'fakepass%' allows the special characters

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.