nsonaniya2010 / subdomainizer Goto Github PK
View Code? Open in Web Editor NEWA tool to find subdomains and interesting things hidden inside, external Javascript files of page, folder, and Github.
License: MIT License
A tool to find subdomains and interesting things hidden inside, external Javascript files of page, folder, and Github.
License: MIT License
It would be good if you add some more cloud storage services like digitaloceanspaces.com,windows.net,storage.googleapis.com etc.
###My Result:
`Got Some IPv4 addresses:
2.343.543.543
1.031.844.254
28.22.5.5
7.2.2.5
22.5.5.5
05.7.18.2
5.24.7.06
2.5.5.5
02.02.04.04
24.33.02.04
2.343.516.516
16.53.05.7
394.394.909.59
2.2.5.2
45.01.81.39
3.996.89.89
382.167.814.178
2.18.52.15
2.343.544.544
5.22.5.5
`
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Smartphone (please complete the following information):
Additional context
Add any other context about the problem here.
Hi @nsonaniya2010
It would be much more useful if you display associated js file from which the secret is exposed.
The following error is thrown:
`Traceback (most recent call last):
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "", line 2, in raise_from
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "C:\Python310\lib\http\client.py", line 1374, in getresponse
response.begin()
File "C:\Python310\lib\http\client.py", line 318, in begin
version, status, reason = self._read_status()
File "C:\Python310\lib\http\client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Python310\lib\socket.py", line 705, in readinto
return self._sock.recv_into(b)
File "C:\Python310\lib\site-packages\urllib3\contrib\pyopenssl.py", line 312, in recv_into
return self.recv_into(*args, **kwargs)
File "C:\Python310\lib\site-packages\urllib3\contrib\pyopenssl.py", line 310, in recv_into
raise timeout('The read operation timed out')
TimeoutError: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python310\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 637, in urlopen
retries = retries.increment(method, url, error=e, _pool=self,
File "C:\Python310\lib\site-packages\urllib3\util\retry.py", line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Python310\lib\site-packages\urllib3\packages\six.py", line 686, in reraise
raise value
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 597, in urlopen
httplib_response = self._make_request(conn, method, url,
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 386, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 306, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='www.fisglobal.com', port=443): Read timed out. (read timeout=20)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\ricky\Tools\SubDomainizer\SubDomainizer.py", line 785, in
subextractor(compiledRegexCloud, compiledRegexSecretList,
File "C:\Users\ricky\Tools\SubDomainizer\SubDomainizer.py", line 618, in subextractor
jsfile.IntJsExtract(url, heads)
File "C:\Users\ricky\Tools\SubDomainizer\SubDomainizer.py", line 201, in IntJsExtract
req = requests.get(url, headers=heads, verify=False, timeout=(20, 20))
File "C:\Python310\lib\site-packages\requests\api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "C:\Python310\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Python310\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python310\lib\site-packages\requests\sessions.py", line 668, in send
history = [resp for resp in gen] if allow_redirects else []
File "C:\Python310\lib\site-packages\requests\sessions.py", line 668, in
history = [resp for resp in gen] if allow_redirects else []
File "C:\Python310\lib\site-packages\requests\sessions.py", line 239, in resolve_redirects
resp = self.send(
File "C:\Python310\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\Python310\lib\site-packages\requests\adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='www.example.com', port=443): Read timeout.`
Please help!
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
y run the tool sudo python3 SubDomainizer.py -u target.com
I get
Traceback (most recent call last):
File "/home/bug/Desktop/tool2/SubDomainizer/SubDomainizer.py", line 13, in
import termcolor
ModuleNotFoundError: No module named 'termcolor'
the module is already installed in my linux mint.
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Smartphone (please complete the following information):
Additional context
Add any other context about the problem here.
Hi ,
Thanks for the great tool .
Given the tool fetches many js files , regex prints the secrets on hits but does not print the URL where it was found . Printing js url wouldn't be a good enhancement ? it can save extra effort back-tracing the url .
It is slow even when given list of 10-20 domains . Can multiprocessing be applied here to make it faster ?
Thanks
Option to have a timeout, some websites keep parsing for 30 min.. Timeout option would be a solution to skip such sites
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Smartphone (please complete the following information):
Additional context
Add any other context about the problem here.
Describe the bug
When testing against an internal site with invalid SSL I get this error:
An error occured while fetching URL, Might be server is down, or domain does not exist, Please check!
Suggested Fix
Add a flag which ignores all ssl errors (-k in this example)
python3 SubDomainizer.py -u -k https://mysite.com
Hi, this is awesome project for recon, May I ask, you have any plan add headless support? Headless mode can detect more domain and js files.
As you mentioned in #13 you'll be re-writing the code to support showing the URL's that secrets are found from. Is it possible to also increase the parallelism of this script?
When running this to scan 100~ url's it takes hours. I created a quick wrapper in golang to have an instance of the script run on all of my cpu's cores and it finished in 15 minutes so there are definitely some bottlenecks in the code slowing it down that could likely be threaded.
Thanks for making this amazing tool!
func SubDomainizer(dir string) {
println("starting SubDomainizer")
if _, err := os.Stat(dir + "/" + date + "/" + "subdomainizer/domains"); os.IsNotExist(err) {
os.MkdirAll(dir+"/"+date+"/"+"subdomainizer/domains", os.ModePerm)
}
if _, err := os.Stat(dir + "/" + date + "/" + "subdomainizer/cloud"); os.IsNotExist(err) {
os.MkdirAll(dir+"/"+date+"/"+"subdomainizer/cloud", os.ModePerm)
}
if _, err := os.Stat(dir + "/" + date + "/" + "subdomainizer/secrets"); os.IsNotExist(err) {
os.MkdirAll(dir+"/"+date+"/"+"subdomainizer/secrets", os.ModePerm)
}
var wg = sync.WaitGroup{}
maxGoroutines := 10
guard := make(chan struct{}, maxGoroutines)
domains := ReadFile(dir + "[redacted]")
for _, domain := range domains {
guard <- struct{}{}
wg.Add(1)
go func(dir string, date string, domain string) {
hash := GenerateRandomString()
cmd := exec.Command("python3", "[redacted]]tools/SubDomainizer/SubDomainizer.py", "-u", domain,
"-o", dir+"/"+date+"/subdomainizer/domains/"+hash+"_domains.txt", "-cop", dir+"/"+date+"/subdomainizer/cloud/"+hash+"_cloud.txt", "-sop", dir+"/"+date+"/subdomainizer/secrets/"+hash+"_secrets.txt",
"-g", "-gt", "[redacted]")
println(cmd.String())
cmd.Start()
cmd.Wait()
<-guard
wg.Done()
}(dir, date, domain)
}
wg.Wait()
}
Would be great if when reading from the subdomain list http or https is pre-appended, so it's easier to import from other tools.
Will be happy to send a PR later if needed.
Is your feature request related to a problem? Please describe.
I know what the secrets are but I need to know where were they found so I can remove the data leak from my server
Describe the solution you'd like
I'd like to see where the items in secretList are found, to make it easier to understand which files/data is their source
i use mitm for check requests, so sometimes script make request with joined strings.
i try using sript for github.com , and i see this:
172.18.0.11:37246: GET https://github.com/
200 OK 39.23k
172.18.0.11:37250: GET https://github.com/
200 OK 39.25k
GET https://github.githubassets.com/assets/compat-bootstrap-edde9aec.jshttps://github.githubassets.com/assets/compat-bootstrap-edde9aec.js
404 Not Found 30b
GET https://github.githubassets.com/assets/github-bootstrap-e9911286.js
200 OK 122.5k
GET https://github.githubassets.com/assets/compat-bootstrap-edde9aec.js
200 OK 9.42k
GET https://github.githubassets.com/assets/frameworks-564f5a6e.jshttps://github.githubassets.com/assets/frameworks-564f5a6e.js
404 Not Found 30b
GET https://github.githubassets.com/assets/frameworks-564f5a6e.js
200 OK 48.33k
GET https://github.githubassets.com/assets/github-bootstrap-e9911286.jshttps://github.githubassets.com/assets/github-bootstrap-e9911286.js
404 Not Found 30b
So few request contains of 2 urls
Hello @nsonaniya2010 ,
Getting this after scanning 10+ subdomains using list function:
Searching for External Javascript links in page..... Successfully got all the external js links SubDomainizer.py:132: DeprecationWarning: invalid escape sequence '\|' content = unquote(requests.get(js).content.decode('unicode-escape'))
Traceback (most recent call last): File "SubDomainizer.py", line 360, in <module> subextractor(compiledRegexCloud, compiledRegexSecretList, compiledRegexDomain,compiledRegexIP, i) File "SubDomainizer.py", line 290, in subextractor threads = ThreadPool(300) File "/usr/lib/python3.7/multiprocessing/dummy/__init__.py", line 124, in Pool return ThreadPool(processes, initializer, initargs) File "/usr/lib/python3.7/multiprocessing/pool.py", line 802, in __init__ Pool.__init__(self, processes, initializer, initargs) File "/usr/lib/python3.7/multiprocessing/pool.py", line 176, in __init__ self._repopulate_pool() File "/usr/lib/python3.7/multiprocessing/pool.py", line 241, in _repopulate_pool w.start() File "/usr/lib/python3.7/multiprocessing/dummy/__init__.py", line 51, in start threading.Thread.start(self) File "/usr/lib/python3.7/threading.py", line 847, in start _start_new_thread(self._bootstrap, ()) RuntimeError: can't start new thread
Hello @nsonaniya2010
SubDomainizer is not parsing amazon bucket from url as the source is like
<a href="https://example.com" target="NEW"><img src="https://s3-us-west-2.amazonaws.com/domain/image.jpg"></a>
Describe the bug
When I run python3 SubDomainizer.py -h, I get the following error message:
Traceback (most recent call last):
File "SubDomainizer.py", line 13, in
import termcolor
ImportError: No module named 'termcolor'
I have verified that the module 'termcolor' is installed
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Currently the output is logged into one file which we can specify with the "-o" flag. Ideally, we could use that same flag and two different files would be generated: one with the results from subdomains another with cloud storage results. The names could be pre-appended with the parameter from the output flag.
This would make much easier to parse results from the tool and integration into different workflows.
Cheres
Hello @nsonaniya2010 ,
Can you please enhance SubDomainizer by adding one more switch to scan already downloaded js files from a folder.
I'm running linux Mint tina
uname:
Linux d0urd3n 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
This is the error I'm getting:
Traceback (most recent call last):
File "/home/n0w0nd3r/tools/SubDomainizer/SubDomainizer.py", line 869, in
gitThread.map(getGithubData, contentApiURLs)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/home/n0w0nd3r/tools/SubDomainizer/SubDomainizer.py", line 617, in getGithubData
_data = base64.b64decode(jsonData['content'])
KeyError: 'content'
Hi,
Firstly, a great tool, so glad I found it!
I have noticed that savesecretsresults()
isn't begin called in the SubDomainizer.py. As a result, nothing is getting saved to -sop --SECRETOP
.
SubDomainizer/SubDomainizer.py
Lines 699 to 705 in ea9a166
I don't like to raise an issue without giving a possible solution but unfortunately, my coding isn't up to scratch.
I love this tools, it's quite faster, and it has a features particularly i love that is
Github data checking scan
External Javascript examination
But particulary, Github scan, so It is possible to add just this few features
Json output
Query subdomain IP
And also add support for module import, that way it will support this tool from being used in
other python scripts. I wanted to intergrate this tool at https://www.nmmapper.com and particulary at it's collection of subdomain tools
So to be something like this
import subdomainizer
scan_result = subdomainizer.subdomain_scan("example.com")
git_scan_result = subdomainizer.git_scan("whatever")
Thank you.
This is a really annoying bug, after I gather subdomains and put the URLS in a list to scan them, subdomainizer completely freezes at specific urls for whatever reasons despite being alive hosts, I delete the URL from the txt file that's causing this and rescan, another URL that causes the tool to completely freeze and I had to remove that too..etc until the output finally shows when there are no more URLs causing a permanent freeze
other than that the tool is an exception and is very useful despite simplicity, thank you very much for publicly posting the project
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.