Giter VIP home page Giter VIP logo

magic-cfn-resources's Introduction

magic-cfn-resources

Build Status

Builds Lambda-backed custom Cloudformation resources. When you use magic-cfn-resources's build method, a Lambda function is created in your stack and used to build a custom resource. Resources that can be built with magic-cfn-resources are: SnsSubscription, DynamoDBStreamLabel, StackOutputs, and SpotFleet.

Provided resources in detail:

SNS Subscriptions

This allows you to manage SNS subscriptions as though they are first-class CloudFormation resources.

DynamoDB Stream Labels

This does not actually create any backend resource, but looks up the label for the stream associated with a DynamoDB table.

This resource will use the stream's label as its PhysicalResourceId, so you can then access the label itself in your template via:

{ "Ref": "LogicalNameOfYourCustomResource" }

CloudFormation StackOutputs

Looks up the Outputs for an existing CloudFormation stack.

You can access the values of the stack's outputs with Fn::GetAtt

{ "Fn::GetAtt": ["LogicalNameOfYourCustomResource", "LogicalNameOfStackOutput"] }

SpotFleet

Makes SpotFleet requests.

DefaultVpc

Looks up the default VPC in the region you've launched your stack in, and provides information about the VPC via Fn::GetAtt.

{ "Fn::GetAtt": ["LogicalNameOfYourCustomResource", "VpcId"] }

You can use Fn::GetAtt to obtain the following data:

  • VpcId: the default VPC's ID
  • AvailabilityZones: an array of strings representing the VPC's availability zones
  • AvailabilityZoneCount: the number of availability zones
  • PublicSubnets: an array of strings representing the VPC's public subnets
  • RouteTable: the ID for the first route table in the VPC
  • RouteTables: IDs of all of the route tables in the VPC

S3NotificationTopicConfig

Creates a notification topic configuration on an S3 bucket.

S3Inventory

Creates an InventoryConfiguration for an existing S3 bucket that is not defined in CloudFormation. If you are defining an S3 bucket in CloudFormation, you can use the native CloudFormation support for this configuration.

To create a magical resource in your own CloudFormation template:

In an existing script or in a new script (i.e. sns-subscription.js):

// Purpose: create a handler for your Lambda function to reference

const magicCfnResources = require('@mapbox/magic-cfn-resources');
// export the custom function needed for your stack.
module.exports.SnsSubscription = magicCfnResources.SnsSubscription;

Another example: module.exports.SpotFleet = magicCfnResources.SpotFleet;

In the CloudFormation template of your stack:

const magicCfnResources = require('@mapbox/magic-cfn-resources');

Then, pass in the necessary parameters to magicCfnResources.build. These are the parameters needed for each resource:

SnsSubscription

const SnsSubscription = magicCfnResources.build({
  CustomResourceName: 'SnsSubscription',
  LogicalName: 'Logical Name', // a name to refer to the custom resource being built
  S3Bucket: 'Bucket Name', // the S3 bucket the code for the handler lives in
  S3Key: 'Key', // the S3 key for where the handler lives
  Handler: 'sns-subscription.SnsSubscription', // references the handler created in the repository
  Properties: {
    SnsTopicArn: 'Topic Arn', // the ARN of the SNS Topic you are subscribing to
    Protocol: 'Protocol', // the SNS protocol, i.e. 'sqs', 'email'
    Endpoint: 'Endpoint' // the endpoint you are subscribing
  }
});

DynamoDBStreamLabel

const DynamoDBStreamLabel = magicCfnResources.build({
  CustomResourceName: 'DynamoDBStreamLabel',
  LogicalName: 'Logical Name', // a name to refer to the custom resource being built
  S3Bucket: 'Bucket Name', // the S3 bucket the code for the handler lives in
  S3Key: 'Key', // the S3 key for where the handler lives
  Handler: 'dynamodb-stream-label.DynamoDBStreamLabel', // references the handler created in the repository
  Properties: {
    TableName: 'Name of Table', // the name of the DynamoDB table
    TableRegion: 'Region' // the region of the DynamoDB table i.e.: 'us-east-1'
  }
});

StackOutputs

const StackOutputs = magicCfnResources.build({
  CustomResourceName: 'StackOutputs',
  LogicalName: 'Logical Name', // a name to refer to the custom resource being built
  S3Bucket: 'Bucket Name', // the S3 bucket the code for the handler lives in
  S3Key: 'Key', // the S3 key for where the handler lives
  Handler: 'stack-outputs.StackOutputs', // references the handler created in the repository
  Properties: {
    StackName: 'Name', // name of the CloudFormation stack
    StackRegion: 'region' // region of the CloudFormation stack i.e.: 'us-east-1'
  }
});

SpotFleet

const SpotFleet = magicCfnResources.build({
  CustomResourceName: 'SpotFleet',
  LogicalName: 'Logical Name', // a name to refer to the custom resource being built
  S3Bucket: 'Bucket Name', // the S3 bucket the code for the handler lives in
  S3Key: 'Key', // the S3 key for where the handler lives
  Handler: 'spot-fleet.SpotFleet', // references the handler created in the repository
  Properties: {
    SpotFleetRequestConfigData: { }, // object with SpotFleet configuration specifics
    Region: 'region', // region of the SpotFleet i.e.: 'us-east-1'
  }
});

DefaultVpc

const DefaultVpd = magicCfnResources.build({
  CustomResourceName: 'DefaultVpc',
  LogicalName: 'Logical Name', // a name to refer to the custom resource being built
  S3Bucket: 'Bucket Name', // the S3 bucket the code for the handler lives in
  S3Key: 'Key', // the S3 key for where the handler lives
  Handler: 'index.DefaultVpc', // references the handler created in the repository
  Properties: {}
});

S3NotificationTopicConfig

const S3TopicConfig = magicCfnResources.build({
  CustomResourceName: 'S3NotificationTopicConfig',
  LogicalName: 'Logical Name', // a name to refer to the custom resource being built
  S3Bucket: 'Bucket Name', // the S3 bucket the code for the handler lives in
  S3Key: 'Key', // the S3 key for where the handler lives
  Handler: 'index.DefaultVpc', // references the handler created in the repository
  Properties: {
    Id: 'notif-id', // an id to identify your notification config
    SnsTopicArn: 'topic arn', // topic arn to subscribe (must be in the same region as bucket)
    Bucket: 'bucket name', // name of bucket configuration is placed on,
    BucketRegion: 'us-east-1', // the region the bucket is in
    EventTypes: ['s3:ObjectCreated:*'], // the types of event to notify about
    PrefixFilter: 'prefix', // a prefix to filter notifications on (optional)
    SuffixFilter: '.jpg', // a suffix to filter notifications on (optional)
    BucketNotificationResources: [ 'bucket Arn' ]
    // if Bucket permissions need to be scoped, default is access to all resources (optional)
  }
});

S3Inventory

const S3InventoryConfig = magic.build({
  CustomResourceName: 'S3Inventory',
  LogicalName: 'Logical Name', // a name to refer to the custom resource being built
  S3Bucket: 'Bucket Name', // the S3 bucket the code for the handler lives in
  S3Key: 'Key', // the S3 key for where the handler lives
  Handler: 'index.S3Inventory', // references the handler created in the repository
  Properties: { // Properties here are identical to those in the CloudFormation-native InventoryConfiguration
    BucketRegion: 'us-east-1', // the only additional property required
    Bucket: 'my-bucket',
    Id: 'my-inventory-id',
    InventoryConfiguration: {
      Schedule: { Frequency: 'Daily' },
      IsEnabled: true,
      Destination: {
        S3BucketDestination: {
          Bucket: cf.getAtt('my-inventory-bucket', 'Arn'),
          Format: 'ORC'
        }
      },
      OptionalFields: ['Size', 'LastModifiedDate', 'EncryptionStatus'],
      IncludedObjectVersions: 'Current',
      Id: 'my-inventory-id'
    }
  }
});

Optional Condition

A Condition from your template can also be passed into build. i.e.:

const SpotFleet = magicCfnResources.build({
  CustomResourceName: 'SpotFleet',
  LogicalName: 'Logical Name', // a name to refer to the custom resource being built
  S3Bucket: 'Bucket Name', // the S3 bucket the code for the handler lives in
  S3Key: 'Key', // the S3 key for where the handler lives
  Handler: 'spot-fleet.SpotFleet', // references the handler created in the repository
  Properties: {
    SpotFleetRequestConfigData: { }, // object with SpotFleet configuration specifics
    Region: 'region', // region of the SpotFleet i.e.: 'us-east-1'
  },
  Condition: 'Condition' // the Logical ID of a condition
});

Merge the resources created with build with the resources already in the stack's template:. i.e.:

const cloudfriend = require('@mapbox/cloudfriend');
const magicCfnResources = require('@mapbox/magic-cfn-resources');

module.exports = cloudfriend.merge(SnsSubscription, <Stack Resources>);

To build new functions

Check out contributing.md for a discussion of the framework this library provides for writing other functions.

magic-cfn-resources's People

Contributors

brendanmcfarland avatar hwrdwlf avatar jjanczyszyn avatar k-mahoney avatar kellyoung avatar rclark avatar vsmart avatar xianny avatar zmully avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

magic-cfn-resources's Issues

Upgrade node runtime

Node 6 is now EOL. This project currently uses the Node 6.10 runtime, it needs to be upgraded:

Runtime: 'nodejs6.10',

The AWS docs claim they support Node 10, but this was only announced two days ago so we should confirm that it has full Cloudformation support.

Next Steps

  • Confirm Node 10 support
  • Upgrade to Node 10 (or Node 8 if necessary)
  • Release new version

Bucket permissions default to a scope

If the parameter BucketNotificationResources is not set, the stack creates a policy giving read/write notification permissions on all buckets within the account. Can this logic be inverted by making the BucketNotificationResources parameter is required, and removing the * default? Is there a use case for * that prevents this inversion?

Update lambda runtime to nodejs14.x

👋 Hey mapboxers! The nodejs10.x lambda runtime EOLs Nov 30, 2021 but I've already started getting errors preventing stacks from launching. This looks like a simple change but I've not permissions to PR it.

21:13:27Z us-east-1: CREATE_FAILED StackOutputsFunction: Resource handler returned message: "The runtime parameter of nodejs10.x is no longer supported for creating or updating AWS Lambda functions. We recommend you use the new runtime (nodejs14.x) while creating or updating functions. (Service: Lambda, Status Code: 400, ...

[discussion] bundled code can potentially be too big for Lambda function.

Right now, magic-cfn-resources has users create the handler for the Lambda function in the repository they are working in. Therefore, because the handler is in the repository, the code for the Lambda function is bundled with the repository, and the entirety of the repository is part of the bundled code referenced in the Lambda function. This can lead to the Lambda function that creates the magic resource to fail to create because it has a requirement of Unzipped size must be smaller than 262144000 bytes.

The actual resources to make a magic-cfn-resource is not anywhere near the limit, but the current set up makes it so it's possible code from the repository a magic-cfn-resource is used in can breach the size limit. Therefore, one idea is to bundle the code for magic-cfn-resources and place it on S3. Them, refer to where the code is when creating a magic-cfn-resource.

However, the issue is how do we access the magic-cfn-resource if we are using multiple AWS accounts? The goal of creating magic-cfn-resources was so all the resources necessary to create a magic-cfn-resource lived really close to the repository that uses it. By creating a designated location for magic-cfn-resource code as mentioned in the solution above, it defeats the original intent.

@jakepruitt, @rclark, and I are still discussing how to move forward. Will leave this ticket open for discussion.

add S3BucketNotificationConfiguration resource

Add the ability to configure a topic to get notifications from an S3 bucket on CloudFormation stack creation, update and delete.

For creation and update:
it will make a call to getNotificationConfiguration to see what configurations are there and putNotificationConfiguration with the new Topic, event, and filters.

For delete:
the resource will remove the configuration created in the stack.

cc: @rclark @jakepruitt

remove SnsSubscriptions from oldTopic

The SnsSubscription function manages to remove old subscriptions in the current topic when searching for subscriptions here but it does not remove subscriptions if the topic has changed in a stack update.

For example:

  1. stack update subscribes Endpoint1 to TopicA
  2. then stack is updated to subscribe Endpoint1 to TopicB instead of TopicA

The delete method does not search TopicA for subscriptions to remove, only TopicB to avoid duplicate subscriptions. This means Endpoint1 will continue to receive messages from TopicA until it the subscription is manually deleted by an engineer.

We should update the lambda function to search for subscriptions in the oldTopic and newTopic.

Issues with spot fleet resource during update rollback

I encountered a scenario last night where a stack update attempted to change the fleet's Launch Specification. However there were syntax errors, and so the update to the custom resource failed. The result was that when the update rollback was complete, the new fleet was considered obsolete and the old fleet request was canceled. This led our system to terminate EC2s in the new spot fleet as soon as they launched.

The steps that led to this were complex, but after conversation with AWS support, the takeaways are:

  1. When the custom resource's lambda function receives an UPDATE request, it must compare the incoming set of properties to the existing spot fleet. If they are identical, then it should not create a new spot fleet, and should not provide CloudFormation with a new spot fleet request ID.

  2. "Ref" to custom cfn resources is actually not supported or documented. You're supposed to use "Fn::GetAtt:" for any and all information that you need to pull from the custom resource. The fact that we use a "Ref" to the spot fleet resource as a way to derive the existing fleet's request ID is "a loophole" and it can't be trusted.

cc @mapbox/platform

to version 1.1.0

changing the API to require customResourceName instead of customFunctionName

Engineering standards inventory

Required Elements

If any elements in the below list are not checked, this repo will fail standards compliance.

  • Not running node 4 or below
  • Has at least some test coverage?
  • Has a README?
  • Has no hard-coded critical secrets like API keys?

Rubric

  • 1 pt Is in Version Control/Github ✅ (free points)
  • 1-2 pt node version:
    • 2 pt Best: running node 8+ 🏅
    • 1 pt Questionable: node 6
    • 0 pt Not ok: running node4 or below ⛔️
  • 1 pt No hard-coded config parameters?
  • 1 pt No special branches that need to be deployed?
  • 1 pt All production stacks on latest master?
    • Is not deployed to AWS, no associated stacks.
  • 1 pt No hard-coded secrets like API keys?
  • 1 pt No secrets in CloudFormation templates that don’t use [secure]?
  • 1 pt CI enabled for repo?
  • 1 pt Not running Circle CI version 1? (Point awarded if using Travis)
  • 1 pt nyc integrated to show test coverage summary?
  • 1-3 pt test coverage percentage from nyc?
    • 3 pt High coverage: > 90%
    • 2 pt Moderate coverage: between 75 and 90% total coverage
    • 1 pt 0 - 74% test coverage
  • 1-2 pt evidence of bug fixes/edge cases being tested?
    • 2 pt Strong evidence/several instances noted
    • 1 pt Some evidence
  • 1 pt no flags to enable different functionality in non-test environments?
  • 1 pt Has README?
  • 1-2 pt README explains purpose of a project and how it works to some detail?
    • 2 pt High (but appropriate) amount of detail about the project
    • 1 pt Some detail about the project documented, could be more extensive
  • 1 pt README contains dev install instructions?
  • 1 pt README contains CI badges, as appropriate?
  • 1-2 pt Code seems self-documenting: file/module names, function names, variables? No redundant comments to explain naming conventions?
    • 2 pt Strongly self-documented code, little to no improvements needed
    • 1 pt Some evidence of self-documenting code
  • 1 pt No extraneous permissions in IAM roles?
  • 1 pt Stack has alarms for AWS resources used routed to PagerDuty? (CPU utilization, Lambda failures, etc.)
    • Not deployed, no associated alarms.
  • 1 pt Stack has other appropriate alarms routed to PagerDuty? (Point awarded if no other alarms needed)
  • 1 pt Alarms documented?
  • master branch protected?
    • 1 pt PRs can only be merged if tests are passing?
    • 1 pt PRs must be approved before merging?
  • 2 pt BONUS: was this repo covered in a deep dive at some point?

Total possible: 30 points (+2 bonus)
Grading scale:

Point Total Qualitative Description Scaled Grade
28+ points Strongly adheres to eng. standards 5
23-27 points Adheres to eng. standards fairly well 4
18-22 points Adheres to some eng. standards 3
13-17 points Starting to adhere to some eng. standards 2
9-12 points Following a limited number of eng. standard practices 1
< 9 points Needs significant work, does not follow most standards 0

Repo grade: 22 - Scaled Grade 3

Suggestions

Not ideal for the standard scoring schema - magic-cfn-resources are a set of utilities, not a deployed stack. Next steps:

  • Remove Node 4 from travis.yml, retain 6 and consider adding 8
  • Update all dependencies

/cc @mapbox/assembly-line

Issue with custom resources

Name: params.LogicalName,

This gets rejected by CloudFormation: Name is not valid in this part of a resource. I think that the Logical Name that gets provided by the caller needs to be used as the Resource Key for each of the resources that .build() is going to provide.

For example, the SpotFleet resource is gonna give you a

  • AWS::IAM::Role
  • AWS::Lambda::Function
  • Custom::SpotFleetFunction

Imagine I want to build a stack with 2 spot fleets. In my final template I can't have a Resources block that looks like this:

{
  Resources: {
    SpotFleetFunction: { // for my first fleet
      Type: 'AWS::Lambda::Function'
      Properties: { ... } 
    },
    ... other resources ...
    SpotFleetFunction: { // for my second fleet
      Type: 'AWS::Lambda::Function'
      Properties: { ... } 
    }
  }
}

They have to have two different "logical names":

{
  Resources: {
    MyFirstSpotFleetFunction: { // for my first fleet
      Type: 'AWS::Lambda::Function'
      Properties: { ... } 
    },
    ... other resources ...
    MySecondSpotFleetFunction: { // for my second fleet
      Type: 'AWS::Lambda::Function'
      Properties: { ... } 
    }
  }
}

Having the caller provide a LogicalName option is the right step to take in order to make this happen -- you just need to apply to the keys for each resource that .build() is creating.

cc @kellyoung

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.