Giter VIP home page Giter VIP logo

iidy's Introduction

iidy (Is it done yet?) -- a CloudFormation CLI tool

iidy improves the developer experience with CloudFormation and CloudFormation templates.

  • It provides immediate, readable feedback about what CloudFormation is doing and any errors it encounters.
  • It is simple to learn, understand, and use. There are fewer moving parts and its commands map directly to CloudFormation's.
  • It has simple, reliable support for AWS profiles.
  • It supports the full range of CloudFormation operations, including changesets without requiring any code beyond what is required to create a stack.
  • It does some template validation, guards against common issues, and will be extended over time to validate our best practices and security policies.
  • It provides seamless integration with AWS Parameter Store.
  • It includes an optional YAML pre-processor which allows CloudFormation templates to be abstracted and simplified in ways not possible with vanilla CloudFormation templates. The pre-processor language supports importing data from a variety of external sources and this allows a better separation of concerns than is possible with stock CloudFormation without resorting to Lambda-backed custom resources. See the pre-processor documentation below for more details.
  • It has bash command completion support.

Table of Contents

Pronunciation

iidy is pronounced "eye-dee", like the audience's response to Cab Calloway in Minnie the Moocher.

Demo

asciicast

Installation

iidy is distributed as a self-contained executable (via pkg).

Binary Install for macOS via Homebrew

Use Unbounce's custom Homebrew Tap to install iidy. This is the preferred method for macOS.

brew tap unbounce/homebrew-taps
brew update
brew install iidy

Binary Installation on Other Platforms

# Grab the appropriate binary from the releases page.
wget https://github.com/unbounce/iidy/releases/download/v1.6.7/iidy-linux-amd64.zip
# or wget https://github.com/unbounce/iidy/releases/download/v1.6.7/iidy-macos-amd64.zip
unzip iidy*.zip
chmod +x iidy
mv iidy /usr/local/bin/ # or somewhere more appropriate

Installation from Source

Counter-intuitively, this requires more disk space than the binary install. You need Node 7+ and npm 4+ installed.

git clone [email protected]:unbounce/iidy.git
cd iidy
npm install && npm run build # to compile our source first
ln -s $(pwd)/bin/iidy /usr/local/bin/
# or npm install -g .

Usage

Help

$ iidy help
iidy - CloudFormation with Confidence                    An acronym for "Is it done yet?"

Commands:
  iidy create-stack     <argsfile>                            create a cfn stack based on stack-args.yaml
  iidy update-stack     <argsfile>                            update a cfn stack based on stack-args.yaml
  iidy create-or-update <argsfile>                            create or update a cfn stack based on stack-args.yaml
  iidy estimate-cost    <argsfile>                            estimate aws costs based on stack-args.yaml

  iidy create-changeset           <argsfile> [changesetName]  create a cfn changeset based on stack-args.yaml
  iidy exec-changeset             <argsfile> <changesetName>  execute a cfn changeset based on stack-args.yaml

  iidy describe-stack      <stackname>                        describe a stack
  iidy watch-stack         <stackname>                        watch a stack that is already being created or updated
  iidy describe-stack-drift <stackname>                       describe stack drift
  iidy delete-stack        <stackname>                        delete a stack (after confirmation)
  iidy get-stack-template  <stackname>                        download the template of a live stack
  iidy get-stack-instances <stackname>                        list the ec2 instances of a live stack
  iidy list-stacks                                            list all stacks within a region

  iidy param                                                  sub commands for working with AWS SSM Parameter Store

  iidy template-approval                                      sub commands for template approval

  iidy render <template>                                      pre-process and render yaml template
  iidy get-import <import>                                    retrieve and print an $import value directly
  iidy demo   <demoscript>                                    run a demo script
  iidy lint-template   <argsfile>                             lint a CloudFormation template
  iidy convert-stack-to-iidy <stackname> <outputDir>          create an iidy project directory from an existing CFN stack
  iidy init-stack-args                                        initialize stack-args.yaml and cfn-template.yaml

  iidy completion                                             generate bash completion script. To use: "source <(iidy completion)"

AWS Options:
  --client-request-token  a unique, case-sensitive string of up to 64 ASCII characters used to ensure idempotent retries.         [string] [default: null]
  --region                AWS region. Can also be set via --environment & stack-args.yaml:Region.                                 [string] [default: null]
  --profile               AWS profile. Can also be set via --environment & stack-args.yaml:Profile. Use --profile=no-profile to override values in
                          stack-args.yaml and use AWS_* env vars.                                                                 [string] [default: null]
  --assume-role-arn       AWS role. Can also be set via --environment & stack-args.yaml:AssumeRoleArn. This is mutually exclusive with --profile. Use
                          --assume-role-arn=no-role to override values in stack-args.yaml and use AWS_* env vars.                 [string] [default: null]

Options:
  --environment, -e  used to load environment based settings: AWS Profile, Region, etc.                                  [string] [default: "development"]
  --debug            log debug information to stderr.                                                                           [boolean] [default: false]
  --log-full-error   log full error information to stderr.                                                                      [boolean] [default: false]
  -v, --version      show version information                                                                                                    [boolean]
  -h, --help         show help                                                                                                                   [boolean]

Status Codes:
  Success (0)       Command successfully completed
  Error (1)         An error was encountered while executing command
  Cancelled (130)   User responded 'No' to iidy prompt or interrupt (CTRL-C) was received

The Args File

Metadata for a CloudFormation stack is kept in a file called the "argsfile" (typically called stack-args.yaml). This file is a parameter for commands like iidy create-or-update or iidy create-changeset and is the main source of data for creating or updating a CloudFormation stack.

Required Properties

Property Description Example
StackName CloudFormation stack name my-stack
Template Local, https or s3 location of CloudFormation template cfn-template.yaml

Optional Properties

Property Description Example
Region AWS Region to use, overridden by --region and AWS_REGION us-east-1
Profile AWS profile to use, overridden by --profile and AWS_PROFILE
AssumeRoleARN IAM Role ARN to assume arn:aws:iam::1234567890:role/my-role
ServiceRoleARN CloudFormation Service Role to use
Tags Tags to assign to CloudFormation stack { service: my-app }
Parameters CloudFormation stack parameters to use
Capabilities List of CloudFormation capabilities to use CAPABILITY_NAMED_IAM
NotificationARNs List of [SNS Topic ARNs to send CloudFormation notifications to] arn:aws:sns:us-east-1:123467890:my-topic
TimeoutInMinutes Number of minutes to provide as timeout to the CloudFormation service 10
OnFailure options: ROLLBACK, DELETE, DO_NOTHING, default ROLLBACK DELETE
DisableRollback boolean defaulting to false
EnableTerminationProtection boolean defaulting to false
StackPolicy Location, https, or s3 location of CloudFormation policy document policy.json
ResourceTypes List of allowed resource types to create in CloudFormation stack ['AWS::EC2::*']
CommandsBefore List of commands to run before create- and update-stack commands ['make build']
UsePreviousTemplate for updates, boolean defaulting to false
UsePreviousParameterValues list of parameter names for updates, equivalent to UsePreviousValue

Example

StackName: my-stack

Template: cfn-template.yaml

Tags:
  project: iidy-docs

Parameters:
  Param1: value1
  Param2: value2

Capabilities:
  - CAPABILITY_IAM
  - CAPABILITY_NAMED_IAM

TimeoutInMinutes: 10

OnFailure: DELETE

StackPolicy: policy.json

CommandsBefore:
  # a list of shell commands to run prior the cfn stack operation
  # /bin/bash is used if found.
  # handlebars templates in the command strings are preprocessed prior to execution.

  # E.g.
  - make build

Importing Data

Data can be imported into the stack args file using the $imports: block.

Import Type Description Example
file Load a local file, JSON and YAML files will be parsed into a data structure vars.yaml
filehash Compute a SHA 256 hash of a file's contents filehash:path, filehash:?filepath (? prefix allows file to be empty)
filehash-base64 Compute a SHA 256 hash of a file's contents, returns Base64 encoded hash filehash-base64:path, filehash-base64:?filepath (? prefix allows file to be empty)
s3 Fetch an object from S3, JSON and YAML files will be parsed into a data structure s3://bucket-name/path/to/file
cfn:export Fetch a CloudFormation Export cfn:export:export-name, cfn:export:export-name?region=us-east-1
cfn:output Fetch the Output of a CloudFormation stack cfn:export:stack-name, cfn:export:stack-name?region=us-east-1
cfn:parameter Fetch the value of a parameter of a CloudFormation stack cfn:parameter:stack-name/parameter-name, cfn:parameter:stack-name/parameter-name?region=us-east-1
cfn:tag Fetch the value of a tag of a CloudFormation stack cfn:tag:stack-name/tag-name, cfn:tag:stack-name/tag-name?region=us-east-1
cfn:resource Fetch a resource of a CloudFormation stack cfn:resource:stack-name/resource-name, cfn:resource:stack-name/resource-name?region=us-east-1
cfn:stack Fetch all data of a CloudFormation stack cfn:stack:stack-name, cfn:stack:stack-name?region=us-east-1
http Fetch a file over HTTP. JSON and YAML files will be parsed into a data structure https://example.com/vars.yaml
env Fetch value from environment variable env:VERSION, env:VERSION:default-value
git:branch Get current git branch name (via git rev-parse --abbrev-ref HEAD) git:branch
git:describe Get current git description (via git describe --dirty --tags) git:describe
git:sha Get current git sha name (via git rev-parse HEAD) git:sha
random:dashed-name Generate a random dashed name, for example: 'uptight-guitar' random:dashed-name
random:name Generate a random dashed name, for example: 'uptightguitar' random:name
random:int Generate a random integer, between 1 and 1000 random:int
ssm Fetch a SSM Parameter Store value ssm:/path/to/param
ssm-path Fetch all SSM Parameter Store values at a given path ssm-path:/path/to/params

iidy imports can be accessed using the !$ YAML tag or using Handlebars templating. !$ will insert the data assigned to that variable. Handlebars templated strings can be used to interpolate values into a string.

iidy parses .yaml or .json imports and makes them available as a map. It uses either the file extension or the mime-type of remote files to detect these file types.

$imports:
  params: ssm-path:/my-services/config

Parameters:
  Timeout: $! params.Timeout

Tags:
  StackName: 'my-service'

Implicit Variables

Some data is automatically set by iidy.

Variable Description
iidy.region The current AWS region in use
iidy.environment Value of the --environment flag
StackName: 'my-service-{{ iidy.environment }}'

Defining Variables

Local variables with file can be specified using the $defs block.

$defs:
  serviceName: my-service

StackName: '{{ serviceName }}-{{ iidy.environment }}'

AWS IAM Settings

iidy supports loading AWS IAM credentials/profiles from a) the cli options shown above, b) Region and Profile or AssumeRoleArn settings in stack-args.yaml, or c) the standard AWS environment variables. You will also need the correct level of IAM permissions for iidy to perform CloudFormation API calls. This information can be provided in a number of different ways (ordered from highest to lowest precedence):

  • CLI options (see above)
  • AWS_REGION and AWS_PROFILE environment variables
  • standard AWS environment variables
  • Region and Profile or AssumeRoleArn in args file

These credentials are used to interact with the CloudFormation API and to perform and AWS-related imports.

If your profile requires an MFA token, iidy will prompt for it.

? MFA token for arn:aws:iam::001234567890:mfa/example.user: ____

If you've assumed a profile prior to running iidy and want to it to ignore what's specified as Profile in stack-args.yaml and instead use AWS_* environment variables, set the CLI option --profile no-profile.

Environment Variables

Any parameter used by iidy can be set using IIDY_{{argname}}. An example of this would be changing the default environment from development to production.

export IIDY_ENVIRONMENT=production

Examples

See the examples/ directory.

Development

iidy is coded in Typescript and compiles to ES2015 JavaScript using commonjs modules (what Node uses). See the Makefile and the script commands in package.json for details about the build process.

Please format all files with tsfmt and remove extra whitespace before submitting a PR.

Releasing

  • Run npm version minor|patch
  • Run git push --tags
  • Run make prepare_releaseto generate the release artifacts for upload
  • Create a GitHub release following the template of previous releases
  • Update the homebrew formula (double check the sha)

Global Configuration

Some iidy features can be configured globally via SSM Parameter Store configurations.

Parameter Value Description
/iidy/default-notification-arn An ARN, eg. "arn:aws:sns:us-east-1:0123456789:mytopic" Provides a value for NotificationARNs if it is not set
/iidy/disable-template-approval Must be the string "true" Unsets ApprovedTemplateLocation to disable the template approval process

Note that SSM Parameter Store values are isolated to the AWS Account and Region they are set in.

License

MIT.

iidy's People

Contributors

cspicer avatar danielleb-r avatar dannosaur avatar dependabot[bot] avatar frodeaa avatar gpritchi-unbounce avatar icecream-monster avatar jpalardy avatar jpb avatar kalupa avatar kazzer avatar mctaylorpants avatar pauloancheta avatar penelopefudd avatar rymndhng avatar scottbrown avatar suchlike avatar tavisrudd avatar tuff avatar unbouncerabbit avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

iidy's Issues

create-stack for unapproved template produces unhelpful error

iidy create-stack stack-args.yaml

Command Metadata:
 CFN Operation:           CREATE_STACK
 Region:                  us-east-1
 Profile:                 None
 CLI Arguments:           region=null, profile=staging, argsfile=stack-args.yaml
 IAM Service Role:        None
 Current IAM Principal:   ...

error: S3 error: The specified key does not exist.
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

Region Map no longer works in iidy 1.6.4

In iidy 1.6.0-rc5, you were able to specify Region as an environment map:

Region:
  integration: us-west-2
  production: us-east-1

This no longer works in iidy 1.64, instead you see this error:

error: Missing region in config

Improve create-stack error message when missing key arguments

Expecting to get a stack name missing argument, got this instead:

iidy create-stack target/infrastructure/resources.yml --profile sandbox --region us-west-2

Command Metadata:
Cloudformation operation: CREATE_STACK
Command line arguments: region=us-west-2, profile=sandbox, argsfile=target/infrastructure/resources.yml
Region: us-west-2
Profile: sandbox
Current Account / IAM Arn: 894790724410
 arn:aws:sts::894790724410:assumed-role/cross-account-access-AdminCrossAccountRole-150VE9IYNKQJ4/aws-sdk-js-1501739750740
IAM Service Role: None

error: Cannot read property 'trim' of undefined
error:  TypeError: Cannot read property 'trim' of undefined
    at loadCFNTemplate (/snapshot/cf_templates/lib/index.js:0:0)
    at stackArgsToCreateStackInput (/snapshot/cf_templates/lib/index.js:0:0)
    at CreateStack._run (/snapshot/cf_templates/lib/index.js:0:0)
    at CreateStack.run (/snapshot/cf_templates/lib/index.js:0:0)

Formatting templates converted from JSON

Feature: To re-order the keys that come out of exporting a stack and converting it to YAML.

For IAM Statements, I think the general convention is to order (in this order):

  • SID
  • Effect
  • Action
  • Resource

It'd be nice if iidy could normalize that on the output :D

install command

It would be great to be able to run iidy install, it should create stack_args.yaml in the current directory.

  • Should include everything on the example template.
  • The template should be populated with $PWD/template.yaml.
  • Tags should be populated with:
    Tags: # aws tags to apply to the stack
      owner: $(whomai)
      environment: $environment || development
      project: $(basename $PWD)
      lifetime: short
  • Include everything else should be commented.

`GetAtt` in custom resources does not prefix resource name

In a custom resource (imported as LongLivedBucket) I have:

Resources:
  LongLivedBucket:
    Type: "AWS::S3::Bucket"
    ...
Outputs:
  Name:
    Description: "The bucket's name"
    Value:
      !Ref LongLivedBucket
  ARN:
    Description: "The bucket's ARN"
    Value:
      !GetAtt LongLivedBucket.Arn

In the template I have:

Resources:
  ConfigBucket:
    Type: LongLivedBucket

This renders as:

Outputs:
  ConfigBucketName:
    Description: The bucket's name
    Value: !Ref ConfigBucketLongLivedBucket
  ConfigBucketARN:
    Description: The bucket's ARN
    Value: !GetAtt LongLivedBucket.Arn

The first output correctly prefixes the reference to LongLivedBucket with ConfigBucket, but the second output does not, which results in this template failing validation:
error: Template error: instance of Fn::GetAtt references undefined resource LongLivedBucket

Until this is fixed, a workaround is to change the !GetAtt to the following:
!GetAtt '{{Prefix}}LongLivedBucket.Arn'

Pre-process StackName

Pre-processing the stack name would make it easier to have multiple variants of the same stack, for example having the stack name be based on the branch name - allowing per-branch releases for reviews.

Add ability to assume a role

It would be nice if, in addition to being able to use Cloudformation Service Roles, iidy could assume a role, without having to specify a profile.
Profiles require each user of iidy to have correctly configured profiles. Specifying a role arn in the stack args would make it more portable.

iidy create-stack prompts for MFA token multiple times

iidy create-stack --profile profile-with-mfa prompts for a MFA token more then once. Entering the same token at each prompt results in the following error:

error MultiFactorAuthentication failed with invalid MFA one time pass code.

as MFA tokens are only valid for a single use.

iidy should not prompt for an MFA code more than once.

`iidy update-stack --changeset` fails when there are no changes

Expected no-op message and exit with status code 0, actual:

$ iidy --region us-west-2 update-stack --changeset stack-args.yaml
Stack Details:
 Name:                    foo
 Status                   CREATE_COMPLETE
 Capabilities:            CAPABILITY_IAM
...

Stack Template Diff:

info: Templates are the same


Command Metadata:
 CFN Operation:           CREATE_CHANGESET
 Region:                  us-west-2
...
error: The submitted information didn't contain changes. Submit different information to create a change set. Deleting failed changeset.

Add option to immediately watch a stack that's pending a changeset

When a changeset is created, the stack enters REVIEW_IN_PROGRESS.

Should iidy include an option to immediately watch the stack until it enters a completion phase?

This isn't a difficult thing to accomplish by chaining two iidy commands. The benefit here is implicitly knowing the --stack-name argument needed by watch-stack.

Generated command from `create-changeset` missing profile and region

The values region and profile should be carried through to the exec-changeset command.

The current output looks like this:

Your new stack is now in REVIEW_IN_PROGRESS state. To create the resources run the following
  iidy exec-changeset --stack-name your-stack iam/stack-args.yml sturdy-rat

It would be preferrable pass through the region and profile of the previous command if possible:

iidy --region REGION --profile PROFILE exec-changeset --stack-name your-stack iam/stack-args.yml sturdy-rat

Parameter renamed in custom resource template

I'm having an issue trying to parameterize a redrive policy whithin a custom resource template. iidy is prepending the name of the resource to the parameter (DeliveryDeadLetterQueue.Arn).

iidy version: 1.6.5

# delivery-queue-template.yaml

$params:
  ...
  - Name: RedrivePolicy
    Default: !Ref 'AWS::NoValue'
...

Resources:
  Queue:
    Type: AWS::SQS::Queue
    Properties:
      ...
      RedrivePolicy: !$ RedrivePolicy
# cfn-template.yaml

Resources:
  ...
  DeliveryDeadLetter:
    Type: DeliveryQueue
    Properties:
      EventType: DeliveryDeadLetter
  ...
  ProcessLeadEventV1:
    Type: DeliveryQueue
    Properties:
      ...
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt DeliveryDeadLetterQueue.Arn
        maxReceiveCount: 2

Expected output:

Resources:
  ...
  ProcessLeadEventV1Queue:
    Type: 'AWS::SQS::Queue'
    Properties:
      ...
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt DeliveryDeadLetterQueue.Arn
        maxReceiveCount: 2

Current output:

Resources:
  ...
  ProcessLeadEventV1Queue:
    Type: 'AWS::SQS::Queue'
    Properties:
      ...
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt ProcessLeadEventV1DeliveryDeadLetterQueue.Arn
        maxReceiveCount: 2

Is this the expected behavior? Is there a way to avoid renaming the parameter?

Final binary is not named "iidy"

When you unzip the final binary from the releases page, the binary is called iidy-macos instead of iidy. This is confusing for new users and it makes the Homebrew configuration confusing.

Consider renaming the binary to iidy regardless of OS/Architecture.

Add create-or-update command

Create CLI command to "create or update" a stack (such as iidy apply-stack stack-args.yaml or iidy create-or-update stack-args.yaml).

describe-stack <argsfile>

Currently, to describe a stack, the user has to add iidy describe-stack <stackname> and if it's on a different region, the have to add it as --region us-west-1. And if they are more unlucky, they will have to add a profile which makes the command longer.

It would be great if iidy detects the current stack_args.yaml in the file. It should assume that in the current directory, there's only one stack_args.yaml and template.yaml. Convention over configuration

With this, after they have configured stack_args.yaml, they can run commands without further configuration. iidy describe-stack should just work, but still supporting --stackName and other -- configuration

Document required IAM permissions

Document the base IAM permissions to do iidy {create, update, delete}-stack etc. Optional: fail gracefully if permission are missing for "nice to have" output, such as describing the stack that was created.

Document additional permission required for imports, for example:

  • cfn:output:... requires cloudformation:DescribeStack
  • cfn:{export, parameter, tag, resource, stack}:... requires cloudformation:ListExports

Make setting params easier based on stack-args

Given a stack args like:

$imports:
  params: ssm-path:/path/to/params/

...
Parameters:
  ImageId: !$ params.ImageId

allow something like this to work:

iidy param set stack-args.yaml params.ImageId ami-000000

where params is the $imports variable name that points to a SSM parameter path. This would set /path/to/params/ImageId to ami-000000.

This should work with iidy param --environment ... or other imports that are used to produce the ssm-path.

Related, it would be nice to have iidy param update that fails if it isn't overwriting a parameter (the opposite of iidy set without --overwrite). This, in conjunction with the above, would prevent users from setting the wrong parameter (eg. setting the wrong path or the correct path in the wrong region).

Concat with variable does not "flatten"

$ cat concat.yaml
$defs:
  a: [4,5,6]

foo: !$concat
  - [1,2,3]
  - !$ a
  - [7,8,9]

$ idy render concat.yaml
foo:
  - 1
  - 2
  - 3
  - - 4
    - 5
    - 6
  - 7
  - 8
  - 9

Expected:

foo:
  - 1
  - 2
  - 3
  - 4
  - 5
  - 6
  - 7
  - 8
  - 9

Show time to create or update each resource

Add the time it took to create or update each resource on it's CREATE_COMPLETE/UPDATE_COMPLETE line (maybe delete and rollback too?). For example (times added in parentheses at the end of the line):

 Wed Oct 24 2018 21:30:30 CREATE_IN_PROGRESS AWS::CloudFormation::Stack               my-stack
 Wed Oct 24 2018 21:30:44 CREATE_IN_PROGRESS AWS::IAM::Role                           MyRole
 Wed Oct 24 2018 21:30:44 CREATE_IN_PROGRESS AWS::IAM::Role                           MyRole
 Wed Oct 24 2018 21:30:57 CREATE_IN_PROGRESS AWS::EKS::Cluster                        EKSCluster
 Wed Oct 24 2018 21:31:01 CREATE_IN_PROGRESS AWS::EKS::Cluster                        EKSCluster
 Wed Oct 24 2018 21:31:04 CREATE_COMPLETE    AWS::IAM::Role                           MyRole (20s)
 Wed Oct 24 2018 21:40:48 CREATE_COMPLETE    AWS::EKS::Cluster                        EKSCluster (9m 51s)
 Wed Oct 24 2018 21:40:54 CREATE_COMPLETE    AWS::CloudFormation::Stack               my-stack (10m 24s)

This will aid the identification of slow-to-create resources in a CloudFormation stack.

`import` behaviour missing documentation

The templating import mechanism documentation in the README.md is missing the need to include render: <template> in the stack-args.yaml file.

Bonus points: if the tool could detect that the render is not included in the description and explicitly warn the user, that would be double-plus delightful.

Parameter store approval process

Add an optional "approval" process for changing parameter store values. This purpose of this mechanism would be to get a second set of eyes when changing values that affect production as well as allow for rollouts of parameter changes to be accepted before they are applied and affect an application's stack.

Something like:

iidy param set --with-approval /some/path/to/param value
iidy param review /some/path/to/param
iidy param review-by-path /some/path/to/

This could be implemented simply by prefixing the pending parameter's path with some key. For example, set --with-approval would write to _pending/some/path/to/param. param review would pull from _pending/some/path/to/param and /some/path/to/param and show both to the user to approve or reject. review-by-path would do the same, but for all pending parameters under a path.

iidy fails to run if .aws directory or profile does not exist

iidy should be able to run using only the AWS_* env vars.

For example, if the .aws folder does not exist, it should not fail.

(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: ENOENT: no such file or directory, open '/root/.aws/credentials'
1) If you want to compile the package/file into executable, please pay attention to compilation warnings and specify a literal in 'require' call. 2) If you don't want to compile the package/file into executable and want to 'require' it from filesystem (likely plugin), specify an absolute path in 'require' call using process.cwd() or process.execPath.
(node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Expose region as variable

Expose what iidy has determined to be the current region (via --region or environment variables) as a variable to be used in stack args file (similar to how environment is exposed).

delete-stack hangs on polling

When executing delete-stack the program hangs on polling state

Platfrom: Linux

? Are you sure you want to DELETE the stack roman-1? Yes
Live Stack Events (2s poll):
<HANGS>

Support rendering of multiple YAML documents

Support rendering of multiple YAML documents within a single file, for example:

---
foo: bar
---
baz: qux

This would be very useful when using iidy rendering for non-Cloudformation YAML files.

Watch of CREATE_COMPLETE stack polls forever

$ iidy --profile data --region us-east-1 watch-stack verify-data-cluster-slow-test

Stack Details:
 Name:                    verify-data-cluster-slow-test
 Status                   CREATE_COMPLETE
 Capabilities:            CAPABILITY_IAM,CAPABILITY_NAMED_IAM
 Service Role:            None
 Tags:                    owner=dataops, environment=test, project=ub-data-infrastructure/cluster
 DisableRollback:         false
 Creation Time:           Thu Aug 10 2017 21:21:03
 Last Update Time:        Thu Aug 10 2017 21:21:03
 Timeout In Minutes:      None
 NotificationARNs:        None
 ARN:                     arn:aws:cloudformation:us-east-1:514588561872:stack/verify-data-cluster-slow-test/d0a09c60-7e11-11e7-8c6a-503acac41efd
 Console URL:             https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stack/detail?stackId=arn%3Aaws%3Acloudformation%3Aus-east-1%3A514588561872%3Astack%2Fverify-data-cluster-slow-test%2Fd0a09c60-7e11-11e7-8c6a-503acac41efd

Previous Stack Events (max 10):
 Thu Aug 10 2017 21:21:11 CREATE_COMPLETE                     AWS::EC2::SecurityGroup                  EC2InstanceSecurityGroup
 Thu Aug 10 2017 21:21:14 CREATE_IN_PROGRESS                  AWS::EC2::Instance                       EC2Instance
 Thu Aug 10 2017 21:21:16 CREATE_IN_PROGRESS                  AWS::EC2::Instance                       EC2Instance
 Thu Aug 10 2017 21:21:26 CREATE_COMPLETE                     AWS::EC2::Volume                         Volume
 Thu Aug 10 2017 21:21:31 CREATE_COMPLETE                     AWS::S3::Bucket                          S3Bucket
 Thu Aug 10 2017 21:21:34 CREATE_IN_PROGRESS                  AWS::IAM::Role                           IAMRole
 Thu Aug 10 2017 21:21:35 CREATE_IN_PROGRESS                  AWS::IAM::Role                           IAMRole
 Thu Aug 10 2017 21:21:49 CREATE_COMPLETE                     AWS::IAM::Role                           IAMRole
 Thu Aug 10 2017 21:21:52 CREATE_COMPLETE                     AWS::EC2::Instance                       EC2Instance
 Thu Aug 10 2017 21:21:56 CREATE_COMPLETE                     AWS::CloudFormation::Stack               verify-data-cluster-slow-test
 10 of 17 total events shown

Live Stack Events (2s poll):
⢂⠀ 58 seconds elapsed

Leveraging parameter store history

Leverage parameter store history to debug and resolve issues due to changes in parameter store values.

Running existing commands, with parameters as they were at a given point in time:

iidy create-stack --point-in-time '2018-01-01 3:30' stack-args.yaml
iidy update-stack --point-in-time '2018-01-01 3:30' stack-args.yaml
iidy param get-by-path --point-in-time '2018-01-01 3:30' /some/path

Viewing how values have changed:

# Show changes since date
iidy param diff /some/path 2018-01-01 
# Show changes between dates
iidy param diff /some/path '2018-01-01 3:30' '2018-01-01 4:30' 

Creating a `changeset` from a non-existent stack shows warning messages

Steps:

  1. Call create-changeset with a stack name that does not exist
  2. See error message:
(node:77684) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): ValidationError: Stack [YOUR_STACK_NAME] does not exist
(node:77684) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
  1. Pending changeset still executes and works as expected

The error message demonstrated in (2) should probably be hidden since this isn't actually a failure.

iidy fails silently if AWS metadata service 404s

If you happen to be running iidy in an environment where the credentials are not found, iidy will fail silently. In particular, I noticed this running iidy under AWS Cloud9. Running iidy render on a template would produce no output.

Manually configuring AWS credentials with aws configure works around the issue.

Non-octal strings are not interpolated properly in stack args properties

Importing a value that is an octal-like string does not interpolate properly into stack args properties, for example:

$defs:
  accountId: 00123456789

ServiceRoleArn: arn:aws:iam::{{ accountId }}:role/role-name

Output from iidy create-stack:

...
 IAM Service Role:        arn:aws:iam::$0string 00123456789:role/role-name
...

Related to #69, #126

Nested lookup can fail within $defs

If a variable (eg. foo) and a nested lookup using that variable (eg. foo.bar) are both in $defs, the lookup can fail.

$ cat list.yaml
$defs:
  a: !$split
    - ','
    - 1,2,3

foo: !$ a.0
$ iidy render list.yaml
foo: '1'
$ cat list.yaml
$defs:
  a: !$split
    - ','
    - 1,2,3
  foo: !$ a.0

foo: !$ foo
$ iidy render list.yaml
error: Could not find "0" at Root.foo}

It appears to be fine if the variable is a static value:

$ cat list.yaml
$defs:
  a:
    - 1
    - 2
    - 3
  foo: !$ a.0

foo: !$ foo
$ iidy render list.yaml
foo: 1

MFA requirement on IAM role trust policies are not supported by iidy

iidy does not support the functionality of MFA requirements on role assumptions (--profile in AWS CLI parlance). This is used to enforce role assumptions constraints when working with escalated IAM role privileges or working in a multi-account structure.

iidy ... --profile other-account

will return an STS assume role error.

Documentation for Conventions for organizing iidy

Given all the features in iidy, it'd be good to have a succinct document that highlights the conventions to recommend users of iidy:

Some things I can think of:

  • naming of stack-args file name(stack-args.yml, stack-args.yaml)
  • how to make stack-args.yml reusable
    • making use of -e or for multi-environment setups
  • conventions for working with Templates (i.e. name them some-resource-template.yaml)

`!Sub` does expansion incorrectly in included files

Take a template, input-template.yaml:

$imports:
  Include: include.yaml
AWSTemplateFormatVersion: 2010-09-09

Resources:
  SomeBucket:
    Type: "Include"
    Properties:
      Suffix: SomeSuffix

With include.yaml:

$params:
  - Name: Suffix

Resources:
  Something:
    Type: "AWS:Something"
    Properties:
      Property1: !Sub
        - "{{Prefix}}-${Foo}"
        -
          - Foo: !$ Suffix
      Property2:
        Fn::Sub:
          - "{{Prefix}}-${Foo}"
          -
            - Foo: !$ Suffix

When input-template.yaml is rendered, the output from Fn::Sub and !Sub are quite different:

Resources:
  SomeBucketSomething:
    Type: 'AWS:Something'
    Properties:
      Property1: !Sub 
        - '{{Prefix}}-${SomeBucketFoo}'
        - - Foo: !$include Suffix
      Property2:
        'Fn::Sub':
          - 'SomeBucket-${Foo}'
          - - Foo: SomeSuffix

Fn::Sub is correctly replacing {{Prefix}}, and not touching ${Foo}.

!Sub should replace {{Prefix}} and not touch embedded ${...} expressions unless the match the name of an in scope variable. (Like from a $param, $import, or $defs.)

iidy does not support STS assumeUser credentials

Expected AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SECURITY_TOKEN and iidy fails at the $import step.

See this error trace:

debug: loading import: cfn:output:some-export-name
error: The security token included in the request is invalid.
error:  InvalidClientTokenId: The security token included in the request is invalid.
    at Request.extractError (/snapshot/iidy/node_modules/aws-sdk/lib/protocol/query.js:47:29)
    at Request.callListeners (/snapshot/iidy/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/snapshot/iidy/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/snapshot/iidy/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/snapshot/iidy/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/snapshot/iidy/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /snapshot/iidy/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/snapshot/iidy/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/snapshot/iidy/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/snapshot/iidy/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
    at Request.emit (/snapshot/iidy/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/snapshot/iidy/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/snapshot/iidy/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/snapshot/iidy/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /snapshot/iidy/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/snapshot/iidy/node_modules/aws-sdk/lib/request.js:38:9)
make: *** [launch-stack] Error 1

Missing $import variables silently fails

Given a template like the following:

$import:
  something: my-thing-{{ other }}

where other is never defined, no error is shown and errors can show up far from actual error site. Specifically in my case, a variable used in an ssm-path import was never set and the error was:

error: Error in string template at Root.Parameters.Something:
       "something" not defined in [object Object]
       Something: {{ params.something }}

Parameter Store always defaults to SecureString

Many of the strings set in parameter store do not need to be secure strings, and ones that are often are echoed by default.

I think a couple changes might be nice:

  • String is the default type
  • Strings are echoed
  • a flag exists (eg. --encrypt) to upload as SecureString
  • SecureStrings are not echoed

!GetAtt assumes it's a string

Getting an error when !GetAtt is throwing an error when it is passed an array:

# cfn-template.yaml
Resources:
  MyCodePipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      RoleArn: !GetAtt [MyRole, Arn]
error: ref.trim is not a function
error:  TypeError: ref.trim is not a function
    at maybeRewriteRef (/snapshot/iidy/lib/preprocess/index.js:827:30)
    at Object.exports.visitGetAtt (/snapshot/iidy/lib/preprocess/index.js:835:60)
    at visitYamlTagNode (/snapshot/iidy/lib/preprocess/index.js:645:24)
    at result (/snapshot/iidy/lib/preprocess/index.js:882:20)
    at visitNode (/snapshot/iidy/lib/preprocess/index.js:899:7)
    at visitMapNode (/snapshot/iidy/lib/preprocess/index.js:969:46)
    at visitPlainMap (/snapshot/iidy/lib/preprocess/index.js:947:16)
    at result (/snapshot/iidy/lib/preprocess/index.js:888:20)
    at visitNode (/snapshot/iidy/lib/preprocess/index.js:899:7)
    at visitMapNode (/snapshot/iidy/lib/preprocess/index.js:969:46)
    at visitPlainMap (/snapshot/iidy/lib/preprocess/index.js:947:16)
    at result (/snapshot/iidy/lib/preprocess/index.js:888:20)
    at visitNode (/snapshot/iidy/lib/preprocess/index.js:899:7)
    at _.fromPairs._.map (/snapshot/iidy/lib/preprocess/index.js:576:24)
    at arrayMap (/snapshot/iidy/node_modules/lodash/lodash.js:660:23)
    at Function.map (/snapshot/iidy/node_modules/lodash/lodash.js:9571:14)

As a workaround, I'm using a string instead

Resources:
  MyCodePipeline:
    Properties:
      RoleArn: !GetAtt MyRole.Arn

create-stack targeting wrong region

Expected: stack created in us-west-2
Actual: stack created in us-east-1

$ env | grep AWS
$ cat stack-args.yaml
$imports:
  stackLabel: env:STACK_LABEL:slim

Region: us-west-2
StackName: 'my-stack-{{ stackLabel }}'
Template: cfn-template.yaml
Parameters:
  ...
$ iidy --debug create-stack stack-args.yaml
debug: argsdata -> stackArgs { '$imports': { stackLabel: 'env:STACK_LABEL:slim' },
  Region: 'us-west-2',
  StackName: 'my-stack-{{ stackLabel }}',
  Template: 'cfn-template.yaml',
  Parameters:
   { ... },
  '$envValues':
   { region: 'us-east-1',
     environment: 'development',
     iidy: { environment: 'development', region: 'us-east-1' },
     stackLabel: 'slim' } }
...

Command Metadata:
 CFN Operation:           CREATE_STACK
 Region:                  us-west-2
 CLI Arguments:           region=null, profile=null, argsfile=infrastructure/ssh_security_group/stack-args.yaml
...
ARN:                     arn:aws:cloudformation:us-east-1:...:stack/...
 Console URL:             https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stack/detail?stackId=...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.