eth0izzle / bucket-stream Goto Github PK

Hi
I'm not sure if this is still being maintained or not, but for some reason bucket-stream has stopped finding any buckets at all. If I use -l option, the log file just stays empty, even after a number of days.
I think it stopped finding anything from around 21st Jan 2022 I think.
Regards
Xnl

python bucket-stream.py -h
Traceback (most recent call last):
File "bucket-stream.py", line 11, in
from queue import queue as Queue
ImportError: No module named queue

Hey dude,

This is awesome can you add a feature to log buckets it has found please?

if i get time ill see if i can throw a PR request.

Hi,
maybe I'm just stupid, but when using bucket-stream, I get eg. bucketname.s3.amazonaws.com, with ACLs = AllUsers: READ, READ_ACP | AuthenticatedUsers: (none). But, when I try to list the contents with aws2 CLI, like aws2 s3 ls s3://bucketname.s3.amazonaws.com, I get

An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: The specified bucket does not exist

So what's going on? Am I doing something wrong?

HI,
I am running bucket-stream in Debian 9 environment with Python 3.5.3 installed on system. The script is being run as a root on a test environment. Whenever I run the script, a file not found error is encountered for default.txt.

root@test:/home/bucketstream/bucket-stream# python3 bucket-stream.py --ignore-rate-limiting 
It is highly recommended to enter AWS keys in config.yaml otherwise you will be severely rate limited!You might want to run with --ignore-rate-limiting
No AWS keys, reducing threads to 5 to help with rate limiting.
Traceback (most recent call last):
  File "bucket-stream.py", line 328, in <module>
    main()
  File "bucket-stream.py", line 300, in main
    .format(ARGS.threads, len([x for x in get_permutations("")])), "green")
  File "bucket-stream.py", line 241, in get_permutations
    perms.extend([line.strip() % domain for line in open(ARGS.permutations)])
FileNotFoundError: [Errno 2] No such file or directory: 'permutations\\default.txt'

If we explicitly specify the path of default.txt, then an incomplete format error is generated. Am I missing something or does the script requires a specific environment setup?

root@test:/home/bucketstream/bucket-stream# python3 bucket-stream.py --ignore-rate-limiting -s keywords.txt --permutations default.txt
It is highly recommended to enter AWS keys in config.yaml otherwise you will be severely rate limited!You might want to run with --ignore-rate-limiting
No AWS keys, reducing threads to 5 to help with rate limiting.
Traceback (most recent call last):
  File "bucket-stream.py", line 328, in <module>
    main()
  File "bucket-stream.py", line 300, in main
    .format(ARGS.threads, len([x for x in get_permutations("")])), "green")
  File "bucket-stream.py", line 241, in get_permutations
    perms.extend([line.strip() % domain for line in open(ARGS.permutations)])
  File "bucket-stream.py", line 241, in <listcomp>
    perms.extend([line.strip() % domain for line in open(ARGS.permutations)])
ValueError: incomplete format

Thanks!

bucket-stream looks like a very nice application.

I'm interesting in being quite specific and only checking certain domains from the certstream. An example would be only .org tlds or a list of specific domains eg domain1.com, domain2.com.

I see some code

def get_permutations(domain, subdomain=None):
    perms = [
        "%s" % domain,
        "www-%s" % domain,
        "%s-www" % domain,
    ]

but I don't this this does that.

I'm thinking a command line argument such as:

-l limit to tld or only from list
-l .org

or 

-l domain-list.txt

Is this currently possible?

Thanks.

I haven't tested this thoroughly yet but I wanted to get some feedback.

It feels like I'm getting less buckets found when using the extended.txt versus when I use the default.txt which contains less permutations. Does this make sense?

I don't have the skills to actually write the PR myself, but I'd like to request the option to log discovered domains to a file.

I run today like:

python3 bucket-stream.py --skip-lets-encrypt -t 50 &>> out.txt

but it doesn't actually write to the file until I interrupt the app with a control+C ...so not sure if you're doing some buffering in the app that prevents real time writing to a file, but yeah -- feature request.

Result of python3 bucket-stream.py --ignore-rate-limiting is:

Traceback (most recent call last):
File "/home/platt-sam/bucket-stream/bucket-stream/bucket-stream.py", line 336, in
main()
File "/home/platt-sam/bucket-stream/bucket-stream/bucket-stream.py", line 315, in main
THREADS.extend([CertStreamThread(q)])
File "/home/platt-sam/bucket-stream/bucket-stream/bucket-stream.py", line 71, in init
self.c = CertStreamClient(
TypeError: CertStreamClient.init() missing 1 required positional argument: 'url'

If I'm just being dumb and not understanding something just let me know! 🙏

Thanks in advance if you get time to reply. Im getting a attribute error;

[1m�[33mWaiting for Certstream events - this could take a few minutes to queue up...�[0mTraceback (most recent call last):

File "C:\bucket-stream\bucket-stream.py", line 279, in
main()
File "C:\bucket-stream\bucket-stream.py", line 271, in main
signal.pause() # pause the main thread
AttributeError: module 'signal' has no attribute 'pause'

Any thoughts?

Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'

any idea? first two steps worked fine

First off, Great script, appreciate the work. I have used in the past a different script, called Slurp. Primarily for the permutations, it allows the usage of "-" delimiters and even blanks, etc, He has a pretty large list also. But, it's not as fast, and you can not use AWS access creds, nor can it log to any where. Plus, I am really not a fan of the Go Lang. Setting it up in any enviornment then having it spit out the compiled app is just like.. oh, there it is.. way over there, all the time.. ok..

Anyways, I took the time, being bored, to give you the equivlent of what your script would look like if you used the permutations that his does. It's very much larger, duplicate are removed and everything. Probably would want to use an external file to load them.. It would add a LOT more detection on your scripts side, but you'd need to do some obvious changes to the permutation parsing / generating.

https://pastebin.com/RpUkywbV

Thanks again! Hopefully you'll think about it !

I noticed when running this script without AWS Keys it catches buckets from multiple AWS locations, such as US-East- and US-West-. However, when using AWS Key and Secret it only logs domain at ".s3.amazonaws.com"

Sadly, I don't have a credit card, to test it with the subscription, but, running without, for hours, not a single bucket is found.
Perhaps, Amazon done some changes?
Thanks!

I looked through the README, codebase, and past issues, but I cannot determine a reason to enable the --skip-lets-encrypt feature flag. What are the pros/cons of this feature flag?

I can update the documentation with any explanation given in this issue. Thanks!

Hey,

I think there might be a rate limit needed on when checking buckets.

So.. running without aws keys.

(bucketvenv) bob@awweb1:~/bucket-stream$ python3 bucket-stream.py
It is highly recommended to enter AWS keys in config.yaml otherwise you will be severely rate limited!  You might want to run with --ignore-rate-limiting
No AWS keys, reducing threads to 5 to help with rate limiting.
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...

With s3 buckets there is an option to allow "Any Authenticated AWS User" right now this tool would not find these open buckets:

Example:

#open to everyone (catches this one)
$ aws s3 ls  s3://flaws.cloud/ --no-sign-request --region us-west-2
2017-03-14 04:00:38       2575 hint1.html
2017-03-03 05:05:17       1707 hint2.html
2017-03-03 05:05:11       1101 hint3.html
2017-03-25 21:58:29       2877 index.html
2017-02-27 02:59:28         46 robots.txt
2017-02-27 02:59:30       1051 secret-dd02c7c.html

#closed to anyone not authenticated to AWS (does not matter key)
$ aws s3 ls  s3://level2-c8b217a33fcf1f839f6f1f73a00a9ae7.flaws.cloud/ --no-sign-request --region us-west-2

An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

#with AWS key (but not one on the account) would not catch this one
$ aws s3 ls  s3://level2-c8b217a33fcf1f839f6f1f73a00a9ae7.flaws.cloud/  --region us-west-2
2017-02-27 03:02:15      80751 everyone.png
2017-03-03 04:47:17       1433 hint1.html
2017-02-27 03:04:39       1035 hint2.html
2017-02-27 03:02:14       2786 index.html
2017-02-27 03:02:14         26 robots.txt
2017-02-27 03:02:15       1051 secret-e4443fc.html

this can be fixed with something like the following:

"""
This will look for the s3 bucket
Requires:
    aws credentials set
Takes: 
    a bucket name
Returns: 
    True if bucket has directory listing, false otherwise
"""
def try_s3(bucket_name):
    s3 = boto3.client('s3')
    try:
        result = s3.list_objects(Bucket=bucket_name)
        print bucket_name
        return True
    except:
        return False

Once running, the only way to stop checking for buckets seems to be to mash Ctrl+C 3-4 times quickly.

Hitting it once yields the following:

Tested using Python 3.6.4

Hi

This script is great, thank you.

However, I am finding occasional buckets, but when I open the link all I get is XML pages everytime, is this correct or is there a way to view the bucket contents of these publicly accesible buckets.

I am researching on behalf of my company as they are looking to invest in AWS