ucopacme / aws-orgs Goto Github PK

To satisfy IS3 policy, we want to demonstrate IAM users are managed by our request tracking system. to do this we will add 2 new attributes to the users spec:

• CN (the 'cn' attribute of AD user)
• RequestId

These attibutes will be used to generate resource tags to the resulting IAM User resource.

Adding these tags to the user lets us abandon the use of 'Teams' for users. this attribute can be dropped from the users spec.

Implementing this change requires editing of in service spec files

after we migrated aws-orgs into ucopacme, the install instructions still point to the old git repo

This will permit us to separate the different specs into multiple git repositories, so that we can enforce separation of duties for IAM admins and operators: admins will have write access to policysets, sc-policies, organizational-units, accounts, custom-policies; operators will have write access to users, groups, delegations, local-users.

Drop the --verbose option and just be verbose by default. add a --quiet option to suppress output. Out actual usage indicates we always want verbose output.

see aws-orgs/docs/source/testing/

add doc for local-users
make sure existing docs cover any new features

awsorgs.auth.manage_custom_policy() uses stupid method to compare existing policy statement with a specified statement. It compares the dictionaries line by line:

  File "/home/ashely/git-repos/github/ucopacme/aws-orgs/awsorgs/auth.py", line 360, in manage_custom_policy
    if current_doc['Statement'][i] != policy_doc['Statement'][i]:
IndexError: list index out of range

I am utterly amazed this did not throw exception previously.

We are encountering use cases for managing IAM users in more than just the auth account:

• local service user required for some SES functions
• local service user required for a 3rd-party monitoring tool
• AWS Eclipse plugin requires local user for Codecommit access
• Maintainers of S3 backed websites need to manage content using sftp client

I wish to expand awsauth cli to accommodate local user management in org accounts. These are handled similar to delegation roles in that I create a IAM user in a listed set of accounts and I apply policies to the user in some fashion. Unlike delegation roles I need to manage authentication methods for local IAM users. These will vary depending on on use case.

example:

[dryrun] urllib3.connectionpool: DEBUG

These messages add a lot of noise to our debug output. it would be nice if we only see them when we run -dd option.

add functionality to awsaccounts to provission arbitraty tags on AWS accounts

add another spec key for 'job_function'
add job_function attr to delegation spec
make job_function and policies attr mutually exclusive
add support in auth.py to support job_function attr in delegation spec
document usage

consider:
tag roles not using job_function as non-is3 compliant
separating spec sources so operators can only edit user/groups/delegations

add check for .git dir in spec.validate_spec(). bypass if found.

this has to do with the ordering of the tags list when it is being compared to deployed tags. I need to sort tags_spec and deployed tags before comparing.

(ucop_accounts) [gmadayag@area51 spec]$ awsauth users --exec

awsorgs.utils: INFO Creating user 'vmutyal'
Traceback (most recent call last):
File "/home/gmadayag/AWS/ucop_accounts/lib/python3.6/site-
packages/awsorgs/auth.py", line 837, in main
create_users(auth_credentials, args, log, deployed, auth_spec)
File "/home/gmadayag/AWS/ucop_accounts/lib/python3.6/site-
packages/awsorgs/auth.py", line 141, in create_users
response = iam_client.create_user(UserName=u_spec['Name'], Path=path)
raise error_class(parsed_response, operation_name)

botocore.errorfactory.EntityAlreadyExistsException: An error occurred
(EntityAlreadyExists) when calling the CreateUser operation: User with
name vmutyal already exists.

We need a way to pre-populate .awsorgs/spec.d and config.yaml with working skeleton files.
spec files must have detailed parameter documentation.

can pip install run post-install actions?

in awsorgs.auth check that all accounts for which the group has a reference actually are in the Org. delete the group policy if not.

Currently when you run:

organization -v -s samples/org-spec.yaml

If the policy is not created then you a RuntimeError is thrown, this should only happen if the --exec flag is passed.

See full stack below:
/usr/local/bin/python2.7 /Users/sionsmith/development/oso/aws-orgs/awsorgs/orgs.py organization -v -s samples/org-spec.yaml
[dryrun] awsorgs.utils: INFO Updating policy 'infrastructure_service_policy'
[dryrun] awsorgs.utils: INFO Updating policy 'logging_service_policy'
[dryrun] awsorgs.utils: INFO Creating policy 'aduit_only_policy'
[dryrun] awsorgs.utils: INFO Creating policy 'iam_only_policy'
[dryrun] awsorgs.utils: INFO Creating policy 'application_account_policy'
[dryrun] awsorgs.utils: WARNING Account 'global_services' not yet in Organization
[dryrun] awsorgs.utils: WARNING Account 'service_catalogue' not yet in Organization
[dryrun] awsorgs.utils: WARNING Account 'management' not yet in Organization
[dryrun] awsorgs.utils: WARNING Account 'iam_master' not yet in Organization
[dryrun] awsorgs.utils: WARNING Account 'logging' not yet in Organization
Traceback (most recent call last):
File "/Users/sionsmith/development/oso/aws-orgs/awsorgs/orgs.py", line 449, in
main()
File "/Users/sionsmith/development/oso/aws-orgs/awsorgs/orgs.py", line 436, in main
org_spec['organizational_units'], 'root')
File "/Users/sionsmith/development/oso/aws-orgs/awsorgs/orgs.py", line 360, in manage_ou
ou_spec['Child_OU'], ou_spec['Name'])
File "/Users/sionsmith/development/oso/aws-orgs/awsorgs/orgs.py", line 379, in manage_ou
deployed, org_spec, ou_spec, ou['Id'])
File "/Users/sionsmith/development/oso/aws-orgs/awsorgs/orgs.py", line 332, in manage_policy_attachments
policy_name)
RuntimeError: spec-file: ou_spec: policy 'aduit_only_policy' not defined

Process finished with exit code 1

max-line-length=90

add facility to awsauth that deletes any custom policies with /awsauth/ path prefix and are not attached to any resources.

`I am getting the following error when I run awsauth local-users.

(env) khalid@ITS-
KAHMADZA-9:~/aws/accounts/Auth/awsorgs_spec_files/spec$ awsauth local-
users

/home/khalid/env/env/lib/python3.6/site-packages/awsorgs/spec.py:30:
YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated,
as the default Loader is unsafe. Please read
https://msg.pyyaml.org/load for full details.

config = yaml.load(f.read())

`

Fake error

remove any refs to real usernames or hostnames from package documentation

group policy remains attached to a group after the policy has been removed from the group in auth-spec.

we can no longer add in-line policy to trusted group when trusting account is set to ALL in delegation spec. we have so many accounts now we hit a limit:

For inline policies You can add as many inline policies as you want to an IAM user, role, or group, but the total aggregate policy size (the sum size of all inline policies) per entity cannot exceed the following limits:
User policy size cannot exceed 2,048 characters

Role policy size cannot exceed 10,240 characters

Group policy size cannot exceed 5,120 characters

add a spec top level parameter for minumum_package_version. add code to compare this to the
aws-orgs version running. send an informative message if package-version is old.
this wll prevent users from getting spec parsing errors if an old package does not support the current spec layout.

we've been tracking business contacts for accounts and users via the teams spec. this is kinda lame. Now that AWS lets us tag IAM users and Org accounts, we should track this info as tags applied directly to the resources.

as of 0.3.0, email and team tags are applied to IAM users. i think we can drop the team tag. but we should start tagging users with our ServiceNow req numbers. but this should probably be a generic tag.

as for accounts, I think we should required tags for business and tech contacts. maybe these are required. we should also tag accounts with our ucop required tags, such as
ucop:service
ucop:protection_level
etc...

to satify IS3 policies we need to show local IAM service user are governed by our intake process and are associated with a real human. to do this we want to add tags to the local user. these tags equate to required attributes in local-users spec:

• Service
• BusinessContactEmail
• TechnicalContactEmail
• RequestId

We also want to track IAM service users by the resource path. to we want to apply a standard path format to generated local users: /<default_path>/service/<service_name>.

by using the 'Service' attribute in the path name, we no longer need the 'Path' attribute in the local-users spec.

[dryrun] awsorgs.utils: WARNING Account UCOP PeopleSoft POC not found in deployed accounts

depends on #63

replaced describe_organization() with list_root() as per AWS support ticket:

#need to use the list_roots() call here.
p_type = org_client.list_roots()['Roots'][0]['PolicyTypes']
if(not p_type or (p_type['Type'][0] == 'SERVICE_CONTROL_POLICY' and p_type[0]['Status'] != 'ENABLED')):
    org_client.enable_policy_type(RootId=root_id, PolicyType='SERVICE_CONTROL_POLICY')

When trying to delete a group, you get a spec validation error if the 'Members' field is empty, but you get a runtime error if you try to set ensure: absent while the group has members. definately a bug.

The work-arround is to delete the 'Members' line entirely and then set ensure: absent.

The solution is to edit group-spec validator config to allow Members field to be a null value.

This needs more investigation. Seems to work ok sometimes and not others.

when creating a new delegation role, I can set a custom path and the role is creatd with that path. but if I then alter the Path attribute in the delegation spec, when I rerun awsauth delegations, no change or update occurs, not even an info message.

This is a bug intruduced by the preceeding patch. Need to check if g_spec contains Ensure: absent prior to scanning for groups.

this will allow us to build a testing suite for interface testing using click CLIrunner

'[dryrun] awsorgs.utils: ERROR validate_spec_file(): schema validation failed for spec_file: /home/agould/git-repos/codecommit/seg-auth/awsorgs_spec_files/spec/groups-spec.yml
[dryrun] awsorgs.utils: DEBUG validate_spec_file(): validator errors:
groups:

• 4:
  • Members:
    • no definitions validate
    • anyof definition 0:
      • null value not allowed
        anyof definition 1:
      • null value not allowed

Date: Mon Jun 3 16:33:39 2019 -0700

bigdev acces for mmacdo and shunter

diff --git a/spec/groups-spec.yml b/spec/groups-spec.yml
index 0a723e9..e852cec 100644
--- a/spec/groups-spec.yml
+++ b/spec/groups-spec.yml
@@ -34,11 +34,11 @@ groups:
- aprayag
- cmathew
- kahmadza

•  - mmacdo
  

•  - shunter
  

  • Name: bigprod
    Ensure: present
    Members:

•  - shunter
  

•  - mmacdo
  

`

File "/home/agould/git-repos/github/ucopacme/aws-orgs/awsorgs/auth.py", line 174, in create_users
user.update(NewPath=path)

when lookup finds multiples, let us know what is was actually looking for.

refacter get_policy_arn and manage_custom_policy functions so that
get_policy_arn no longer calls manage_custom_policy. It only returns a policy arn or None.