eth0izzle / bucket-stream Goto Github PK
View Code? Open in Web Editor NEWFind interesting Amazon S3 Buckets by watching certificate transparency logs.
Home Page: https://www.darkport.co.uk
License: MIT License
Find interesting Amazon S3 Buckets by watching certificate transparency logs.
Home Page: https://www.darkport.co.uk
License: MIT License
Hi
I'm not sure if this is still being maintained or not, but for some reason bucket-stream has stopped finding any buckets at all. If I use -l
option, the log file just stays empty, even after a number of days.
I think it stopped finding anything from around 21st Jan 2022 I think.
Regards
Xnl
python bucket-stream.py -h
Traceback (most recent call last):
File "bucket-stream.py", line 11, in
from queue import queue as Queue
ImportError: No module named queue
Hey dude,
This is awesome can you add a feature to log buckets it has found please?
if i get time ill see if i can throw a PR request.
Hi,
maybe I'm just stupid, but when using bucket-stream, I get eg. bucketname.s3.amazonaws.com, with ACLs = AllUsers: READ, READ_ACP | AuthenticatedUsers: (none). But, when I try to list the contents with aws2 CLI, like aws2 s3 ls s3://bucketname.s3.amazonaws.com, I get
An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: The specified bucket does not exist
So what's going on? Am I doing something wrong?
HI,
I am running bucket-stream in Debian 9 environment with Python 3.5.3 installed on system. The script is being run as a root on a test environment. Whenever I run the script, a file not found error is encountered for default.txt.
root@test:/home/bucketstream/bucket-stream# python3 bucket-stream.py --ignore-rate-limiting
It is highly recommended to enter AWS keys in config.yaml otherwise you will be severely rate limited!You might want to run with --ignore-rate-limiting
No AWS keys, reducing threads to 5 to help with rate limiting.
Traceback (most recent call last):
File "bucket-stream.py", line 328, in <module>
main()
File "bucket-stream.py", line 300, in main
.format(ARGS.threads, len([x for x in get_permutations("")])), "green")
File "bucket-stream.py", line 241, in get_permutations
perms.extend([line.strip() % domain for line in open(ARGS.permutations)])
FileNotFoundError: [Errno 2] No such file or directory: 'permutations\\default.txt'
If we explicitly specify the path of default.txt, then an incomplete format error is generated. Am I missing something or does the script requires a specific environment setup?
root@test:/home/bucketstream/bucket-stream# python3 bucket-stream.py --ignore-rate-limiting -s keywords.txt --permutations default.txt
It is highly recommended to enter AWS keys in config.yaml otherwise you will be severely rate limited!You might want to run with --ignore-rate-limiting
No AWS keys, reducing threads to 5 to help with rate limiting.
Traceback (most recent call last):
File "bucket-stream.py", line 328, in <module>
main()
File "bucket-stream.py", line 300, in main
.format(ARGS.threads, len([x for x in get_permutations("")])), "green")
File "bucket-stream.py", line 241, in get_permutations
perms.extend([line.strip() % domain for line in open(ARGS.permutations)])
File "bucket-stream.py", line 241, in <listcomp>
perms.extend([line.strip() % domain for line in open(ARGS.permutations)])
ValueError: incomplete format
Thanks!
bucket-stream looks like a very nice application.
I'm interesting in being quite specific and only checking certain domains from the certstream. An example would be only .org tlds or a list of specific domains eg domain1.com, domain2.com.
I see some code
def get_permutations(domain, subdomain=None):
perms = [
"%s" % domain,
"www-%s" % domain,
"%s-www" % domain,
]
but I don't this this does that.
I'm thinking a command line argument such as:
-l limit to tld or only from list
-l .org
or
-l domain-list.txt
Is this currently possible?
Thanks.
I haven't tested this thoroughly yet but I wanted to get some feedback.
It feels like I'm getting less buckets found when using the extended.txt versus when I use the default.txt which contains less permutations. Does this make sense?
I don't have the skills to actually write the PR myself, but I'd like to request the option to log discovered domains to a file.
I run today like:
python3 bucket-stream.py --skip-lets-encrypt -t 50 &>> out.txt
but it doesn't actually write to the file until I interrupt the app with a control+C ...so not sure if you're doing some buffering in the app that prevents real time writing to a file, but yeah -- feature request.
Result of python3 bucket-stream.py --ignore-rate-limiting
is:
Traceback (most recent call last):
File "/home/platt-sam/bucket-stream/bucket-stream/bucket-stream.py", line 336, in
main()
File "/home/platt-sam/bucket-stream/bucket-stream/bucket-stream.py", line 315, in main
THREADS.extend([CertStreamThread(q)])
File "/home/platt-sam/bucket-stream/bucket-stream/bucket-stream.py", line 71, in init
self.c = CertStreamClient(
TypeError: CertStreamClient.init() missing 1 required positional argument: 'url'
If I'm just being dumb and not understanding something just let me know! 🙏
Thanks in advance if you get time to reply. Im getting a attribute error;
[1m�[33mWaiting for Certstream events - this could take a few minutes to queue up...�[0mTraceback (most recent call last):
File "C:\bucket-stream\bucket-stream.py", line 279, in
main()
File "C:\bucket-stream\bucket-stream.py", line 271, in main
signal.pause() # pause the main thread
AttributeError: module 'signal' has no attribute 'pause'
Any thoughts?
Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
any idea? first two steps worked fine
First off, Great script, appreciate the work. I have used in the past a different script, called Slurp. Primarily for the permutations, it allows the usage of "-" delimiters and even blanks, etc, He has a pretty large list also. But, it's not as fast, and you can not use AWS access creds, nor can it log to any where. Plus, I am really not a fan of the Go Lang. Setting it up in any enviornment then having it spit out the compiled app is just like.. oh, there it is.. way over there, all the time.. ok..
Anyways, I took the time, being bored, to give you the equivlent of what your script would look like if you used the permutations that his does. It's very much larger, duplicate are removed and everything. Probably would want to use an external file to load them.. It would add a LOT more detection on your scripts side, but you'd need to do some obvious changes to the permutation parsing / generating.
Thanks again! Hopefully you'll think about it !
I noticed when running this script without AWS Keys it catches buckets from multiple AWS locations, such as US-East- and US-West-. However, when using AWS Key and Secret it only logs domain at ".s3.amazonaws.com"
Sadly, I don't have a credit card, to test it with the subscription, but, running without, for hours, not a single bucket is found.
Perhaps, Amazon done some changes?
Thanks!
I looked through the README, codebase, and past issues, but I cannot determine a reason to enable the --skip-lets-encrypt
feature flag. What are the pros/cons of this feature flag?
I can update the documentation with any explanation given in this issue. Thanks!
So.. running without aws keys.
(bucketvenv) bob@awweb1:~/bucket-stream$ python3 bucket-stream.py
It is highly recommended to enter AWS keys in config.yaml otherwise you will be severely rate limited! You might want to run with --ignore-rate-limiting
No AWS keys, reducing threads to 5 to help with rate limiting.
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...
With s3 buckets there is an option to allow "Any Authenticated AWS User" right now this tool would not find these open buckets:
Example:
#open to everyone (catches this one)
$ aws s3 ls s3://flaws.cloud/ --no-sign-request --region us-west-2
2017-03-14 04:00:38 2575 hint1.html
2017-03-03 05:05:17 1707 hint2.html
2017-03-03 05:05:11 1101 hint3.html
2017-03-25 21:58:29 2877 index.html
2017-02-27 02:59:28 46 robots.txt
2017-02-27 02:59:30 1051 secret-dd02c7c.html
#closed to anyone not authenticated to AWS (does not matter key)
$ aws s3 ls s3://level2-c8b217a33fcf1f839f6f1f73a00a9ae7.flaws.cloud/ --no-sign-request --region us-west-2
An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
#with AWS key (but not one on the account) would not catch this one
$ aws s3 ls s3://level2-c8b217a33fcf1f839f6f1f73a00a9ae7.flaws.cloud/ --region us-west-2
2017-02-27 03:02:15 80751 everyone.png
2017-03-03 04:47:17 1433 hint1.html
2017-02-27 03:04:39 1035 hint2.html
2017-02-27 03:02:14 2786 index.html
2017-02-27 03:02:14 26 robots.txt
2017-02-27 03:02:15 1051 secret-e4443fc.html
this can be fixed with something like the following:
"""
This will look for the s3 bucket
Requires:
aws credentials set
Takes:
a bucket name
Returns:
True if bucket has directory listing, false otherwise
"""
def try_s3(bucket_name):
s3 = boto3.client('s3')
try:
result = s3.list_objects(Bucket=bucket_name)
print bucket_name
return True
except:
return False
Hi
This script is great, thank you.
However, I am finding occasional buckets, but when I open the link all I get is XML pages everytime, is this correct or is there a way to view the bucket contents of these publicly accesible buckets.
I am researching on behalf of my company as they are looking to invest in AWS
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.