netflix-skunkworks / cloudaux Goto Github PK

Amazon recently announced new S3 features around analytics. There are additional API calls that are required:

- list_bucket_analytics_configurations()
- list_bucket_inventory_configurations()
- list_bucket_metrics_configurations()

This will also require boto's minimum version to be bumped up to 1.4.2.

get_role_managed_policy_documents passes its own kwargs directly to its internal calls to get_role_managed_policies and get_managed_policy_document, but the relevant connection parameters (account_number, assume_role, etc) are stripped by the sts_conn decorator beforehand. The result is that those internal calls receive no explicit connection parameters and fall back to boto3's credential-finding routine -- this may fail completely or pick up credentials that were not intended for this use.

Ran into an issue in security monkey when running the S3 watcher:

cloudaux/orchestration/aws/s3.py', line 69, in get_lifecycle 'prefix': rule['Prefix'], KeyError: 'Prefix' "

Any chance you can update https://github.com/Netflix-Skunkworks/cloudaux/blob/master/cloudaux/orchestration/aws/s3.py#L69 to handle when no prefix exists?

We need to unit test all of our AWS code.

I wonder if all the orchestration commands (get_user, get_role, get_bucket) should take an optional fields parameter.

That way you could use CloudAux and specify some subset of the fields. The default would be all.

Each tech could provide an importable field list so that IDE's could auto-complete.

@mikegrima - Thoughts?

sts assume role fails with following error in GovCloud at

ClientError: An error occurred (AccessDenied) when calling the AssumeRole operation: Access denied

Can we somehow pass / implement ARN_PARTITION / ARN_PREFIX in the likes of what has been done in security_monkey

https://github.com/Netflix/security_monkey/blob/b174a705124f12aeee612f9ef93820f2b4227e0e/security_monkey/__init__.py#L72

Need to document the required IAM permissions for use of this library.

PIN to 'google-cloud-storage==0.22.0'

More info at:

googleapis/google-cloud-python#3235

@supertom

@willbengtson found a bug in the CloudAux class.

You can't have more than one CloudAux object instantiated because the conn_details is currently stored in a class variable, not an instance variable.

So this test would fail:

from cloudaux import CloudAux


def test_cloudaux():
    conn_one = {
        "account_number": "111111111111",
        "assume_role": "role_one",
        "region": "us-east-1",
        "session_name": "conn_one"
    }

    conn_two = {
        "account_number": "222222222222",
        "assume_role": "role_two",
        "region": "us-east-2",
        "session_name": "conn_two"
    }

    ca_one = CloudAux(**conn_one)
    ca_two = CloudAux(**conn_two)

    assert ca_one.conn_details["account_number"] == "111111111111"
    assert ca_one.conn_details["assume_role"] == "role_one"
    assert ca_one.conn_details["region"] == "us-east-1"
    assert ca_one.conn_details["session_name"] == "conn_one"

    assert ca_two.conn_details["account_number"] == "222222222222"
    assert ca_two.conn_details["assume_role"] == "role_two"
    assert ca_two.conn_details["region"] == "us-east-2"
    assert ca_two.conn_details["session_name"] == "conn_two"

I have a PR coming in just a second to fix the CloudAux class and to add the above unit test.

There is a bug in the S3 get_grants() function when capturing canonical IDs. It is incorrectly looking for URI when it's supposed to fetch ID. This creates an incorrect string when the DisplayName isn't captured. The incorrect string looks like this:

Grants": {
    "null": [
        "FULL_CONTROL"
    ]
},

This is also causing downstream issues with Security Monkey where it thinks that there is a new change each time. This is currently the case with "new" regions where the get_bucket_acl API no longer returns the display name (still have not identified why SM thinks a new change keeps occurring, but this only happens on these specific buckets).

The S3 orchestration should include a field for:

• Region
• Replication?
• Events?

@mikegrima - Anything else?

Need to add CloudTrail support for CloudAux.

CloudAux needs to fetch details from Cloud Front. Security Monkey also needs to have watchers for it.

The boto session caching is causing massive memory leaks with newer versions of botocore.

Will need to only cache credentials instead.

Undefined names may raise NameError at runtime.

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./cloudaux/aws/sns.py:132:2: F821 undefined name 'pagiated'
@pagiated('Topics', request_pagination_marker="NextToken", response_pagination_marker="NextToken")
 ^
1     F821 undefined name 'pagiated'

Also see FIXME in .travis.yml