binxio / aws-ssm-copy Goto Github PK

I'm sure this is not your problem, but I want to let you know that with botocore==1.34.47 aws-ssm-copy stopped working. With botocore==1.34.46 everything works.

Perhaps something can be done on your end.

The error itself looks like this:

# aws-ssm-copy -r --keep-going --target-path /fr-417-feature/v2-frontend /staging/v2-frontend
Traceback (most recent call last):
  File "/apps/.venv/bin/aws-ssm-copy", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/apps/.venv/lib/python3.11/site-packages/aws_ssm_copy/ssm_copy.py", line 336, in main
    cp.main()
  File "/apps/.venv/lib/python3.11/site-packages/aws_ssm_copy/ssm_copy.py", line 319, in main
    self.copy(
  File "/apps/.venv/lib/python3.11/site-packages/aws_ssm_copy/ssm_copy.py", line 196, in copy
    self.target_ssm.put_parameter(**parameter)
  File "/apps/.venv/lib/python3.11/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apps/.venv/lib/python3.11/site-packages/botocore/client.py", line 962, in _make_api_call
    request_dict = self._convert_to_request_dict(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apps/.venv/lib/python3.11/site-packages/botocore/client.py", line 1036, in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apps/.venv/lib/python3.11/site-packages/botocore/validate.py", line 381, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in input: "ARN", must be one of: Name, Description, Value, Type, KeyId, Overwrite, AllowedPattern, Tags, Tier, Policies, DataType

Installed package versions:

# pip list
Package         Version
--------------- -------
aws-ssm-copy    0.5.2
boto3           1.34.47
botocore        1.34.47
jmespath        1.0.1
pip             23.2.1
python-dateutil 2.8.2
s3transfer      0.10.0
setuptools      65.5.0
six             1.16.0
urllib3         2.0.7

When specifying --target-path, source parameter names that do not start with a slash are not prefixed with the target-path.

$ aws-ssm-copy --dry-run --recursive  --source-profile binx-io --profile integration-test   --overwrite --target-path /copy/  /
DRY-RUN: copying /cfn-deep-security-provider/password to /copy/cfn-deep-security-provider/password
DRY-RUN: copying k11b5acb7-f483-4f5a-980d-9ab63284e40c to k11b5acb7-f483-4f5a-980d-9ab63284e40c
DRY-RUN: copying k540471fb-c9be-464c-99fa-bd6a7273b947 to k540471fb-c9be-464c-99fa-bd6a7273b947

Secured String copy failing

ERROR: An error occurred (InvalidKeyId) when calling the PutParameter operation: Key 'arn:aws:kms:us-east-1:123123123:key/0f9b04ee-ab13-4f35-a2b2-d0902bfe6a1e' does not exist (Service: AWSKMS; Status Code: 400; Error Code: NotFoundException; Request ID: 06817cdd-99f4-4895-a52c-60deb22131c9; Proxy: null)

Is there any special instruction to copy the secure-string or is it possible to skip the failing secure-string

is it possible to support json for migrations, problem is that i have ssms from multiple subdirs, but i don't want to move all of them.
e.g

/foo/1
/foo/2
/foo/3
/bar/1
/bar/2
/bar/3

but i only want to move
/foo/1 -> /hello/1
/bar/2 -> /hello/2
/bar/3 -> /world/3

Hi, this used to work, but now I keep getting MultiFactorAuthentication failed with invalid MFA one time pass code.

$ pip install -U aws-ssm-copy
Requirement already satisfied: aws-ssm-copy in /usr/local/lib/python3.9/site-packages (0.3.4)
Requirement already satisfied: boto3 in /usr/local/lib/python3.9/site-packages (from aws-ssm-copy) (1.20.46)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.9/site-packages (from boto3->aws-ssm-copy) (0.10.0)
Requirement already satisfied: botocore<1.24.0,>=1.23.46 in /usr/local/lib/python3.9/site-packages (from boto3->aws-ssm-copy) (1.23.46)
Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /usr/local/lib/python3.9/site-packages (from boto3->aws-ssm-copy) (0.5.0)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.9/site-packages (from botocore<1.24.0,>=1.23.46->boto3->aws-ssm-copy) (2.8.0)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/local/lib/python3.9/site-packages (from botocore<1.24.0,>=1.23.46->boto3->aws-ssm-copy) (1.26.8)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.24.0,>=1.23.46->boto3->aws-ssm-copy) (1.16.0)

Hi!

I found an error (at least I consider it an error, you might have an explanation for it) in the code that prevented me from using your code directly.

When dry run is enable, copy_tags function is always called which is fine with me but that copy_tags function assumes that the parameter can be found in the target region which is not really the case if you are copying to an empty region. The call at line 121 is the problem.

Let me know if you want me to submit a PR!

BTW thanks for a great tool!

Thanks, great tool to copy all parameters in one go.
However, does not copies tags from source parameters to newly created.

Run recursive as aws-ssm-copy -r --target-path /app/prod /app/dev

See: https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_AddTagsToResource.html

When attempting to move params between regions, it fails with "The security token included in the request is invalid." My AWS CLI works just fine so I'm not sure where your app is failing.

Don't know if this is a bug or I am doing something not supposed to do:

aws-ssm-copy --dry-run --source-profile prof1 --profile prof2 --target-path /pepe/mysql /staging/mysql/MYSQL_OTHERS_PASSWORD
DRY-RUN: copying /staging/mysql/MYSQL_OTHERS_PASSWORD to /staging/mysql/MYSQL_OTHERS_PASSWORD

/staging/mysql/MYSQL_OTHERS_PASSWORD exists on prof1, it is a SecureString

When I try to use this with a profile in my credentials file that's based on SSO, I get:
botocore.exceptions.NoCredentialsError: Unable to locate credentials

When I created an IAM account and used that instead, it works great.

Hi,

This is nice tool I used several times.

I want to add one more feature that user can change parameter name.

For example, I want to change b.pem to c.pem like,

$ aws-ssm-copy --source-profile aaa --source-region ap-northeast-2 --profile aaa --region ap-northeast-2 \
/ec2-keypair/aaa/ap-northeast-2/b.pem \
/ec2-keypair/aaa/ap-northeast-2/c.pem

ERROR: /ec2-keypair/aaa/ap-northeast-2/c.pem not found.

Thanks,

What is the process for copying a secure string encrypted with the default key? That key will be different in every account.

Also - what happens if a region goes down where you have added a key policy to grant access to a DR account?

A scenario:

Account A has encrypted store items using a non-default KMS key. I've copied the store items over to Account B (my DR account), and put a key policy on Account A's KMS key to allow Account B to use. Say the region where the key is located in Account A goes down, what are the implications for Account B and its use of the store items?

Unsure of how to copy parameters from one AWS account to another.

Some google searches indicated installing via:

pip install aws-ssm-copy

However when I try to run it:

$ aws-ssm-copy -h
Traceback (most recent call last):
  File "/Users/igal/.pyenv/versions/2.7.18/bin/aws-ssm-copy", line 11, in <module>
    load_entry_point('aws-ssm-copy==0.5.2', 'console_scripts', 'aws-ssm-copy')()
  File "/Users/igal/.pyenv/versions/2.7.18/lib/python2.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/Users/igal/.pyenv/versions/2.7.18/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
    return ep.load()
  File "/Users/igal/.pyenv/versions/2.7.18/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2443, in load
    return self.resolve()
  File "/Users/igal/.pyenv/versions/2.7.18/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/igal/.pyenv/versions/2.7.18/lib/python2.7/site-packages/aws_ssm_copy/__init__.py", line 1, in <module>
    from aws_ssm_copy.ssm_copy import main
  File "/Users/igal/.pyenv/versions/2.7.18/lib/python2.7/site-packages/aws_ssm_copy/ssm_copy.py", line 53
    result["Name"] = re.sub(regex, f"/{tp}/", parameter["Name"])
                                           ^
SyntaxError: invalid syntax

Any tips?

Would be great to have an option to ignore error's in the copy.
I have a few environments that are a mirror of each other, I have found this tool super handy for keeping all environments the same. However when I wanted to do a mass copy without overwriting values (so not using force) the first error causing the program to quit.

ERROR: An error occurred (ParameterAlreadyExists) when calling the PutParameter operation: The parameter already exists. To overwrite this value, set the overwrite option in the request to true.

I would love to be able to tell the app to ignore these types of issues and continue to loop to ensure I have matching parameters in both locations.

I have installed the utility from pip. While performing --dry-run it is showing correct values but executing the command without --dry-run it is throwing an exception.

Command with a dry run and it's result
Command: aws-ssm-copy --source-profile test1 --recursive --overwrite /test/ --source-region us-west-2 --region us-west-2 --profile test1 --target-path /test-v2/ --dry-run

Result: INFO: copying /test/first to /test-v2/first

Command without dry run
Command: aws-ssm-copy --source-profile test1 --recursive --overwrite /test/ --source-region us-west-2 --region us-west-2 --profile test1 --target-path /test-v2/

Exception:
aws-ssm-copy` --source-profile test1 --recursive --overwrite /test/ --source-region us-west-2 --region us-west-2 --profile test1 --target-path /test-v2/
INFO: copying /test/second to /test-v2/second
Traceback (most recent call last):
File "/usr/local/bin/aws-ssm-copy", line 11, in
load_entry_point('aws-ssm-copy==0.2.2', 'console_scripts', 'aws-ssm-copy')()
File "/usr/local/lib/python2.7/site-packages/aws_ssm_copy/copy.py", line 182, in main
cp.main()
File "/usr/local/lib/python2.7/site-packages/aws_ssm_copy/copy.py", line 173, in main
options.overwrite,
File "/usr/local/lib/python2.7/site-packages/aws_ssm_copy/copy.py", line 97, in copy
self.target_ssm.put_parameter(**parameter)
File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 634, in _make_api_call
api_params, operation_model, context=request_context)
File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 682, in _convert_to_request_dict
api_params, operation_model)
File "/usr/local/lib/python2.7/site-packages/botocore/validate.py", line 297, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter Policies, value: [{u'PolicyStatus': u'Pending', u'PolicyText': u'{"Type":"NoChangeNotification","Version":"1.0","Attributes":{"After":"5","Unit":"Days"}}', u'PolicyType': u'NoChangeNotification'}, {u'PolicyStatus': u'Pending', u'PolicyText': u'{"Type":"Expiration","Version":"1.0","Attributes":{"Timestamp":"2019-06-01T12:00:00Z"}}', u'PolicyType': u'Expiration'}], type: <type 'list'>, valid types: <type 'basestring'>
[ec2-user@ip-172-31-27-112 .aws]$ clear

[ec2-user@ip-172-31-27-112 .aws]$ aws-ssm-copy --source-profile test1 --recursive --overwrite /test/ --source-region us-west-2 --region us-west-2 --profile test1 --target-path /test-v2/ --dry-run
INFO: copying /test/first to /test-v2/first
[ec2-user@ip-172-31-27-112 .aws]$ aws-ssm-copy --source-profile test1 --recursive --overwrite /test/ --source-region us-west-2 --region us-west-2 --profile test1 --target-path /test-v2/
INFO: copying /test/first to /test-v2/first
Traceback (most recent call last):
File "/usr/local/bin/aws-ssm-copy", line 11, in
load_entry_point('aws-ssm-copy==0.2.2', 'console_scripts', 'aws-ssm-copy')()
File "/usr/local/lib/python2.7/site-packages/aws_ssm_copy/copy.py", line 182, in main
cp.main()
File "/usr/local/lib/python2.7/site-packages/aws_ssm_copy/copy.py", line 173, in main
options.overwrite,
File "/usr/local/lib/python2.7/site-packages/aws_ssm_copy/copy.py", line 97, in copy
self.target_ssm.put_parameter(**parameter)
File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 634, in _make_api_call
api_params, operation_model, context=request_context)
File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 682, in _convert_to_request_dict
api_params, operation_model)
File "/usr/local/lib/python2.7/site-packages/botocore/validate.py", line 297, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter Policies, value: [], type: <type 'list'>, valid types: <type 'basestring'>