threatexpress / domainhunter Goto Github PK

Checks expired domains for categorization/reputation and Archive.org history to determine good candidates for phishing and C2 domain names

Home Page: http://threatexpress.com

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%

domainhunter's Introduction

Threatexpress Blog

Quickstart

This blog is written in markdown and uses mkdocs to generate the HTML hosted using Git Pages on github

Workflow

Create new content
Push content

git add .
git commit -a -m "new stuff"
git push

Push to Git Pages

mkdocs gh-deploy

Reference

https://www.mkdocs.org/user-guide/deploying-your-docs/

Mkdocs

This wiki use mkdocs and the mkdocs-material theme

MkDocs is a static site generator that's geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file.

Font Awesome

Reference: http://bwmarrin.github.io/MkDocsPlus/fontawesome/

Requirements and Setup

Note: This has been tested with python 3.6 running on MacOS.

Requirements

pip3.6 install mkdocs
pip3.6 install mkdocs-material
pip3.6 install pygments
pip3.6 install pymdown-extensions
pip3.6 install fontawesome-markdown

References

Material Theme Documentation and Reference: http://mkdocs.github.io/mkdocs-bootswatch/#installation-usage
https://squidfunk.github.io/mkdocs-material/getting-started/
https://facelessuser.github.io/pymdown-extensions/

Directory and File Info

./docs/ : mkdocs content
./docs/img/ : Store all course images here
mkdocs.yml : mkdocs configuration file
./site/ : Prod version of site. Not tracked in git. Used for testing or manual deployment

domainhunter's People

Contributors

Stargazers

Watchers

Forkers

minkione quikilr oneiroi vysecurity davehardy20 wineries crackcomm lengyun123456 0utcode techbytom yocoa newaynewlife thecripple kostaki theralfbrown cys3c jack51706 ak47uma666 jbarcia infosecsecurity djenova rubenjpmartins songofhack h1d3r paran0ids0ul olivierh59500 burtay mrmugiwara securitywarrior team-firebugs kbahaxor realmadaha adolfoeliazat techlord-rce 5up3rc clicknull nuadaandre wushenwu h0k5 ankushgoel27 yeyintminthuhtut chaitanyakrishna manuelramirezio vudubond mehrdad-shokri blankxyz vanzhiganov asakasinsky ssundhar tuian tmmmmmr interweblt shantanu561993 eln1x shellgh05t rushmyfile yehgdotnet mgcfish nabz0r dm2333 gavz 4n6strider mrtaheramine reydecopas christruncer p3t3rp4rk3r opt9 akur-8 yogisec tricrosoft forkarc fuckup1337 nickhakkz chouaibhm pktlabs malstor sgachies thor-vath ma5onic dan-sra chrismaddalena dannymas ameeno thedarkcode yifenglv46 marpie attacker34 joaorodolfo rajivraj tim1512 rdsece mohammadobaid1 u53r55 slooppe tsoderstrom seemantbisht secdatahunter chosen1 theseanis vkkan

domainhunter's Issues

License?

Hi there, could you let us know what license is used for this project?

Cisco and Bluecoat issues

Hello, these two checks are giving errors. any ideas?

[*] Fetching domain reputation for: google.com
[*] BlueCoat: google.com
[+] google.com: HTTP Error (400-) - Is your IP blocked?
[*] IBM xForce: google.com
[+] google.com: Search Engines / Web Catalogues / Portals, (Score: 1)
[*] Cisco Talos: google.com
[-] Error retrieving Talos reputation! Expecting value: line 1 column 1 (char 0)
[+] google.com: error

Error when I run the script

Hello,

When I run the script python3 ./domainhunter.py -s mydomain.com I have this error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 158, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/connection.py", line 57, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.9/socket.py", line 953, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 597, in urlopen
    httplib_response = self._make_request(conn, method, url,
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 301, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 167, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fa77e2d0c10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 637, in urlopen
    retries = retries.increment(method, url, error=e, _pool=self,
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='gitlab.com', port=443): Max retries exceeded with url: /gerowen/old-malware-domains-ad-list/-/raw/master/malwaredomainslist.txt (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa77e2d0c10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/PNTST/RECON/Domain-Expirated/domainhunter/./domainhunter.py", line 537, in <module>
    maldomains = downloadMalwareDomains(malwaredomainsURL)
  File "/opt/PNTST/RECON/Domain-Expirated/domainhunter/./domainhunter.py", line 294, in downloadMalwareDomains
    response = s.get(url=url,headers=headers,verify=False,proxies=proxies)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='gitlab.com', port=443): Max retries exceeded with url: /gerowen/old-malware-domains-ad-list/-/raw/master/malwaredomainslist.txt (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa77e2d0c10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

Anyone can help me ?

A greeting and thanks

Error: 'NoneType' object is not callable

I have ran the following commands:

./domainhunter.py -k -r 1000 -c
python domainhunter.py -k -r 1000 -c
python3 domainhunter.py -k -r 1000 -c

There are two separate issues:

Error #1
('[!] Error: ', AttributeError("'NoneType' object has no attribute 'find_all'",))

I changed "findAll" to "find_all" as the previous syntax was for BeautifulSoup3 and it is now importing bs4. After trying to run the command again, I stumbled upon the next error.

Error #2

The assumption is that there is an error within the query, but I have not had much fixing it. Was wondering if you had a quick fix for it. Thank you!

How its possible downloads the expireddomains domain list ?

Because I'm a new user I got the next notification from the website :

"Some features are only available one month after signup. You have to wait another 30 days for them to be enabled automatically for your account.
CSV/Textfile Export
Deleted Archive Lists"

What password to enter?

When I start the script, it asks me for a password, but I have no idea which one to enter.

Can someone guide me?

_   ___  __  __    _    ___ _   _   _   _ _   _ _   _ _____ _____ ____  
|  _ \ / _ \|  \/  |  / \  |_ _| \ | | | | | | | | | \ | |_   _| ____|  _ \ 
| | | | | | | |\/| | / _ \  | ||  \| | | |_| | | | |  \| | | | |  _| | |_) |
| |_| | |_| | |  | |/ ___ \ | || |\  | |  _  | |_| | |\  | | | | |___|  _ < 
|____/ \___/|_|  |_/_/   \_\___|_| \_| |_| |_|\___/|_| \_| |_| |_____|_| \_\ 

Expired Domains Reputation Checker
Authors: @joevest and @andrewchiles

DISCLAIMER: This is for educational purposes only!
It is designed to promote education and the improvement of computer/cyber security.  
The authors or employers are not liable for any illegal act or misuse performed by any user of this tool.
If you plan to use this content for illegal purpose, don't.  Have a nice day :)

[*] Downloading malware domain list from http://mirror1.malwaredomains.com/files/justdomains

Password: 
[!] Login failed

www.expireddomains.net - Does not allow crawlers

For anyone who comes across this.
It is against service terms to run crawlers/bots against this service. Account suspended as a result.

search option (-q) didn't work for me without deleting the additional cookies

a nice script, thankyou

for me, the script didn't find results using the -q option. removing the additional 'annoying' cookies rectified the situation. i didn't submit a pull request because i'm unsure as to whether the fix would be generally good, or is just 'good for me', for whatever reason

[*] Downloading malware domain list from http://mirror1.malwaredomains.com/files/justdomains

Password:

What password? Local? Expireddomains.net? Either way... not working

Error: list index out of range

output just like that, anysolution?>

AttributeError: 'NoneType' object has no attribute 'format'

When I run the program I got
python3 ./domainhunter.py

[-] Error reaching:{} Status: {}
Traceback (most recent call last):
File "./domainhunter.py", line 494, in
maldomains = downloadMalwareDomains(malwaredomainsURL)
File "./domainhunter.py", line 263, in downloadMalwareDomains
print("[-] Error reaching:{} Status: {}").format(url, response.status_code)
AttributeError: 'NoneType' object has no attribute 'format'

How fix it?

Cannot install Pillow on MacOS Catalina

During initial installation of the requirements.txt, I get an error during the install of Pillow. There's LOTS of red text, but at the bottom there's this: "The headers or library files could not be found for zlib, a required dependency when compiling Pillow from source."

'NoneType' object has no attribute 'findAll'

Looks like there's an unhandled beautifulsoup error?

[*] Downloading malware domain list from http://mirror1.malwaredomains.com/files/justdomains
[*] Fetching expired or deleted domains...
[*]  https://www.expireddomains.net/backorder-expired-domains?start=0&o=changed&r=a
[*]  https://www.expireddomains.net/deleted-com-domains/?start=0&o=changed&r=a
[*]  https://www.expireddomains.net/deleted-net-domains/?start=0&o=changed&r=a
[*]  https://www.expireddomains.net/deleted-org-domains/?start=0&o=changed&r=a
[*]  https://www.expireddomains.net/backorder-expired-domains?start=25&o=changed&r=a
[*]  https://www.expireddomains.net/deleted-com-domains/?start=25&o=changed&r=a
[*]  https://www.expireddomains.net/deleted-net-domains/?start=25&o=changed&r=a
[*]  https://www.expireddomains.net/deleted-org-domains/?start=25&o=changed&r=a
[*]  https://www.expireddomains.net/backorder-expired-domains?start=50&o=changed&r=a
[*]  https://www.expireddomains.net/deleted-com-domains/?start=50&o=changed&r=a
[*]  https://www.expireddomains.net/deleted-net-domains/?start=50&o=changed&r=a
[*]  https://www.expireddomains.net/deleted-org-domains/?start=50&o=changed&r=a
[*]  https://www.expireddomains.net/backorder-expired-domains?start=75&o=changed&r=a
[*]  https://www.expireddomains.net/deleted-com-domains/?start=75&o=changed&r=a
[*]  https://www.expireddomains.net/deleted-net-domains/?start=75&o=changed&r=a
[*]  https://www.expireddomains.net/deleted-org-domains/?start=75&o=changed&r=a
'NoneType' object has no attribute 'findAll'

BlueCoat - 400 response

The BlueCoat service seems to have changed. The current payload returns a Bad Request error.

List Index Out of Range - Line 538

Hello everyone,

I apologize if this is an easy question but I am relatively new.

I installed pycharm as my IDE and installed all the packages required (except that I updated pillow to 7.0.0 as I was getting an error while trying to install the 5.0.0) and installed it on Windows.

I successfully ran a command and entered my password and it successfully logged in ExpiredDomains. However, the code gave me the following error (added a line of code that would tell me where the error is)

For the record - I did not know how to install the Dockerfile, if that is the problem maybe someone could explain? would be really appreciated.

The command that I used was: python ./domainhunter.py -r 10

[+] Login successful. ExpiredDomainssessid: ****************** lo4 lo4 [*] https://member.expireddomains.net/domains/combinedexpired/?fwhois=22&fadult=1&start=0&ftlds[]=2&ftlds[]=3&ftlds[]=4&flimit=10&fdomain=&fdomainstart=&fdomainend=&falexa=0 EXCEPTION IN (./domainhunter.py, LINE 538 "tbody = table[0].select("tbody tr")"): list index out of range [-] No domain results found or none are currently available for purchase!
Could someone please help?

Problem with login

Hello,

function:
./domainhunter.py -k apples -c --ocr -t5

i got error mssg:
[-] Error: ExpiredDomains.net requires a username! Use the --username parameter

When i insert login and password in file

Traceback (most recent call last):
File "./domainhunter.py", line 427, in
parser.add_argument("-u", "--username", required=False, default=my_login, type=str, help="username for expireddomain.net")
NameError: name 'my_login' is not defined

However "my_login" working fine when i authorize in browser...

Expired domains blocking accout

Hi!
Thanks for implementing a feature, but as I can see, expireddomains blocking accounts for a several queries from my side via domainhunter.

Maybe it will be good idea to cache all exp domains for one query first and then do search in local disk file?

Can't fetch domain by keyword

I check the code,there is login required when doing domain search,
currently the project can't do domain search
login addr is
https://member.expireddomains.net/login/

Mcafeewg notdefined error

I am getting the following error.

Code shows this for version:

version = "20221025"

[*] https://member.expireddomains.net/domains/combinedexpired/?fwhois=22&fadult=1&start=0&ftlds[]=2&ftlds[]=3&ftlds[]=4&flimit=100&fdomain=&fdomainstart=&fdomainend=&falexa=0
Traceback (most recent call last):
File "domainhunter.py", line 754, in
data.append([domain,birthdate,archiveentries,availabletlds,status,bluecoat,ibmxforce,ciscotalos,umbrella,mcafeewg])
NameError: name 'mcafeewg' is not defined

[-] Missing dependencies: No module named texttable

I have this error

:~$ python ./domainhunter.py
Expired Domains Reputation Check
[-] Missing dependencies: No module named texttable
[*] Install required dependencies by running `pip install -r requirements.txt`

:~$ pip install -r requirements.txt
Requirement already satisfied: requests==2.13.0 in /home/user/.local/lib/python3.5/site-packages (from -r requirements.txt (line 1))
Requirement already satisfied: texttable==0.8.7 in /home/user/.local/lib/python3.5/site-packages (from -r requirements.txt (line 2))
Requirement already satisfied: beautifulsoup4==4.5.3 in /home/user/.local/lib/python3.5/site-packages (from -r requirements.txt (line 3))
Requirement already satisfied: lxml in /usr/lib/python3/dist-packages (from -r requirements.txt (line 4))

:~$ pip install texttable
Requirement already satisfied: texttable in /home/user/.local/lib/python3.5/site-packages

Permissions

chmod 755 *.py

error using argument -w

C:\DL>domainhunter.py -k pro -w 5

| _ \ / _ | / | / \ |_ | \ | | | | | | | | | \ | | | | _
| | | | | | | |/| | / _ \ | || | | | || | | | | | | | | | | | |) |
| || | || | | | |/ ___ \ | || |\ | | _ | || | |\ | | | | || _ <
|/ ___/|| |// ____|| _| || ||_/|| _| || |___|_| _\

Expired Domains Reputation Checker
Authors: @joevest and @andrewchiles

DISCLAIMER: This is for educational purposes only!
It is designed to promote education and the improvement of computer/cyber security.
The authors or employers are not liable for any illegal act or misuse performed by any user of this tool.
If you plan to use this content for illegal purpose, don't. Have a nice day :)

[*] Downloading malware domain list from http://mirror1.malwaredomains.com/files/justdomains

[] Fetching expired or deleted domains containing "pro"
[] https://www.expireddomains.net/domain-name-search/?q=pro&fwhois=22&falexa=1
[] https://www.expireddomains.net/domain-name-search/?start=25&q=pro&fwhois=22&falexa=1
[] https://www.expireddomains.net/domain-name-search/?start=50&q=pro&fwhois=22&falexa=1
[*] https://www.expireddomains.net/domain-name-search/?start=75&q=pro&fwhois=22&falexa=1

[*] 78 of 78 domains discovered with a potentially desireable categorization!

[] Search complete
[] Log written to 20180818_113249_domainreport.html

Traceback (most recent call last):
File "C:\domainhunter-master\domainhunter.py", line 653, in
print(drawTable(header,sortedDomains))
File "C:\domainhunter-master\domainhunter.py", line 283, in drawTable
return(t.draw())
File "C:\Users\jaimin26783\AppData\Local\Programs\Python\Python36-32\lib\site-packages\texttable.py", line 372, in draw
out += self._draw_line(self._header, isheader=True)
File "C:\Users\jaimin26783\AppData\Local\Programs\Python\Python36-32\lib\site-packages\texttable.py", line 547, in _draw_line
line = self._splitit(line, isheader)
File "C:\Users\jaimin26783\AppData\Local\Programs\Python\Python36-32\lib\site-packages\texttable.py", line 586, in _splitit
array.extend(textwrap.wrap(c, width))
File "C:\Users\jaimin26783\AppData\Local\Programs\Python\Python36-32\lib\textwrap.py", line 379, in wrap
return w.wrap(text)
File "C:\Users\jaimin26783\AppData\Local\Programs\Python\Python36-32\lib\textwrap.py", line 354, in wrap
return self._wrap_chunks(chunks)
File "C:\Users\jaimin26783\AppData\Local\Programs\Python\Python36-32\lib\textwrap.py", line 248, in _wrap_chunks
raise ValueError("invalid width %r (must be > 0)" % self.width)
ValueError: invalid width -3 (must be > 0)

MXToolbox reputation checking is broken

The service has removed the original endpoints used to query SPAM and Google Safe Browsing lists. There is a new API that returns JSON objects and is heavily reliant on JS to format the requests correctly. Any malformed request results in an IP block that requires CAPTCHA completion to remove.

All HTTP GETs

https://mxtoolbox.com/domain/apples.com/ -> Initial request to the service
https://mxtoolbox.com/api/v1/user -> Returns JSON that is subsequently used as cookie parameter and also an HTTP header in the GET. Need to parse and manually update requests session cookie jar and create custom HTTP headers
https://mxtoolbox.com/api/v1/lookup/blacklist/apples.com -> Returns JSON with blacklist lookup results. All we really need to check is presence of records in "Failed" or "Warnings"

{
  "UID": null,
  "ArgumentType": "hostname",
  "Command": "blacklist",
  "IsTransitioned": false,
  "CommandArgument": "nwk-aaemail-lapp01.apple.com",
  "TimeRecorded": "2020-02-05T15:54:56.7293086-06:00",
  "ReportingNameServer": null,
  "TimeToComplete": "328",
  "RelatedIP": "17.151.62.66",
  "ResourceRecordType": 0,
  "IsEmptySubDomain": false,
  "IsEndpoint": true,
  "HasSubscriptions": false,
  "AlertgroupSubscriptionId": null,
  "Failed": [],
  "Warnings": [],
  "Passed": [
    {
      "ID": 333,
      "Name": "BSB Domain",
      "Url": "https://mxtoolbox.com/Problem/blacklist/BSB-Domain?page=prob_blacklist&showlogin=1&hidetoc=1&action=blacklist:nwk-aaemail-lapp01.apple.com",
      "PublicDescription": null,
      "BlacklistResponseTime": "0",
      "IsExcludedByUser": false
    },
    SNIP
  ],
  "Errors": [],
  "IsError": false,
  "Information": [
    {
      "DNS Resolution": "nwk-aaemail-lapp01.apple.com was resolved to 17.151.62.66."
    }
  ],
  "MultiInformation": [],
  "IsBruteForce": false,
  "Transcript": [
    {
      "Transcript": "DNS - Load Balancers\r\nLookupServer 328ms\r\n"
    }
  ],
  "MxRep": 100,
  "EmailServiceProvider": null,
  "DnsServiceProvider": null,
  "DnsServiceProviderIdentifier": null,
  "RelatedLookups": [
    {
      "Name": "dns lookup",
      "URL": "https://mxtoolbox.com/api/v1/lookup/a/nwk-aaemail-lapp01.apple.com",
      "Command": "a",
      "CommandArgument": "nwk-aaemail-lapp01.apple.com"
    },
    {
      "Name": "smtp diag",
      "URL": "https://mxtoolbox.com/api/v1/lookup/smtp/nwk-aaemail-lapp01.apple.com",
      "Command": "smtp",
      "CommandArgument": "nwk-aaemail-lapp01.apple.com"
    },
    {
      "Name": "http test",
      "URL": "https://mxtoolbox.com/api/v1/lookup/http/nwk-aaemail-lapp01.apple.com",
      "Command": "http",
      "CommandArgument": "nwk-aaemail-lapp01.apple.com"
    }
  ]
}

Only fetching first domain from each page

Hey everyone,

Ran the code the other day and it was only fetching the first domain from each page, can someone help?

Thank