Giter VIP home page Giter VIP logo

irvinlim / es2017-lambda-boilerplate Goto Github PK

View Code? Open in Web Editor NEW
174.0 8.0 26.0 371 KB

AWS Lambda boilerplate for Node.js 6.10, adding ES2018/7/6 features, Docker-based unit testing and various CI/CD configurations

Home Page: https://irvinlim.com/blog/async-await-on-aws-lambda/

License: MIT License

JavaScript 100.00%
es2017 es2016 aws-lambda lambda serverless async-await docker-lambda aws-sdk aws-codebuild travis-ci

es2017-lambda-boilerplate's Introduction

es2017-lambda-boilerplate

es2017-lambda-boilerplate

Travis CI David David Greenkeeper badge GitHub The MIT License

Note: As of April 2018, AWS has announced support for Lambda functions on Node.js 8.10, which already supports all ES2018-2016 features added by this boilerplate (and more). You can however, still use this boilerplate on Node.js 8.10 to make use of the unit testing features.


This is a boilerplate for AWS Lambda Node.js 6.10.0 functions, which allows you to use the latest JavaScript ES2016, ES2017 and ES2018 features. The boilerplate also allows you to test your function in a Docker container (thanks to docker-lambda), and also includes common configurations for CI/CD, for both Travis CI and AWS CodeBuild + AWS CloudFormation.

Latest JavaScript features

This boilerplate adds support for the following most commonly used JavaScript features that are not natively supported on Node.js 6.10:

Feature Supported?
ES2016/ES7
Exponentiation operator (**)
Array.prototype.includes
ES2017/ES8
Object.values, Object.entries
Trailing commas in function syntax
async/await
ES2018/ES9
Object rest/spread properties

Note: Most ES2015/ES6 are earlier features are supported.

Usage

Edit your Lambda function under src/main.js, and run:

npm run build

This will transpile your functions down to ES5 using Babel, so that it can be executed using the Node.js 6.10.0 runtime.

For convenience, the following command will create an artifact.zip file which you can upload to AWS Lambda:

npm run package

Testing

You can run automated tests for your Lambda function inside of a Docker container using docker-lambda:

npm run test

All files in the test/ directory which end with .test.js will be interpreted as a test suite. A sample unit test is provided under test/example.test.js to get you started.

The test runner used is Jest (with Jasmine). Using docker-lambda also requires Docker to be installed on your host; see the docs for docker-lambda for more instructions.

Specification tests

In order to ensure that the Babel configuration works and is following the spec, the boilerplate also runs several automated tests to catch any Babel misconfigurations.

  • Functional testing: Runs the relevant spec tests from Test262 (actual tests taken from node.green) on docker-lambda to mock the AWS Lambda environment
  • Snapshot testing: Unit testing strategy by storing snapshots of Babel-transformed source code and running unit tests against them

You can find the spec tests under spec/functional and spec/snapshot respectively.

If you are not going to modify .babelrc, you can choose to skip these tests by omitting the npm run spec script in .travis.yml. This will help to speed up your builds by a bit.

Deployment

Deployment using the AWS SDK

You can automatically deploy to AWS Lambda locally or through CI (e.g. Travis CI) using the AWS SDK, as long as you provide an access key for an IAM user that has write access to AWS Lambda. A single NPM script allows you to deploy using this method:

npm run deploy

See Environment variables for the list of environment variables that are required for SDK deployment.

Deployment through CloudFormation + CodeBuild

Instead of depending on external tools like Travis CI, you can also choose to use AWS CloudFormation to bootstrap the relevant AWS resources, integrated with AWS CodeBuild and AWS CodePipeline. Alternatively, deployment via AWS CodeStar may also be supported out of the box.

To modify the build process, you can update the CodeBuild configuration file at buildspec.yml. To modify the properties of the resultant Lambda function, you can update the CloudFormation configuration file at samTemplate.yml.

If you are new to AWS CI/CD tools, you can follow the official AWS tutorial to set up a build pipeline using CodePipeline. Take note of the following:

  • Set up a S3 bucket for uploading CodeBuild artifacts to.
    • If the CodeBuild build fails, you may need to set the S3_BUCKET environment variable within CodeBuild directly.
  • Ensure that the IAM roles have the necessary permissions to access required resources, including the S3 bucket.
  • The CloudFormation template filename under CodePipeline settings should be template.yml.

Using the AWS SDK

You can write Lambda functions that make use of the AWS SDK by simply import-ing aws-sdk. The package is installed globally within the AWS Lambda environment, so you don't need to add it to your package.json.

Also make sure that your function has Internet connectivity (i.e. not within a VPC without a NAT gateway). The internetConnectivityTest.js utility is included to help to debug such problems early when deploying to AWS Lambda.

Environment variables

If you plan to use the AWS SDK, either for deployment (using npm run deploy), or within your function itself, you need to pass the following environment variables:

  • AWS_ACCESS_KEY_ID: IAM user access key ID
  • AWS_SECRET_ACCESS_KEY: IAM user secret access key
  • AWS_REGION: AWS region where the Lambda function resides in (required for SDK deployment only)
  • LAMBDA_FUNCTION_NAME: Name or ARN of the Lambda function (required for SDK deployment only)

This will work if you store it in a .env file in the root of the project (see dotenv), or if you define it within Travis CI itself (see Travis CI docs).

IAM user permissions

The minimum permissions required for the IAM user for SDK deployment are:

Remember to add more permissions as required if you need to access the SDK in your function.

Why?

Reduce callback hell with async/await

The highest version of Node.js supported on AWS Lambda is 6.10.0, which supports only features up to ES2015/ES6. Newer features in ES2017, such as async/await, are incredibly useful when performing network requests, such as when used with the AWS SDK:

const EC2 = new AWS.EC2();
const Route53 = new AWS.Route53();

// Get instance by ID.
const instances = await EC2.describeInstances({ InstanceIds: 'i-abcdef01' }).promise();

// Get public IP address.
const publicIpAddress = instances.Reservations[0].Instances[0].PublicIpAddress;

// Do something else with the IP address...
await Route53.changeResourceRecordSets({
    // ...
}).promise();

Run automated tests locally/through CI

Instead of testing your Lambda function by uploading to AWS Lambda every single time, running automated tests in conjunction with CI is a better option. By using Docker to mock the AWS Lambda environment locally, you can write test cases to verify the correctness of your function, given an input (the Lambda event):

import run from './util/runner';

it('should work', function() {
    // Sample event from SNS.
    const event = {
        Records: [
            {
                EventVersion: '1.0',
                EventSource: 'aws:sns',
                Sns: {
                    MessageId: '95df01b4-ee98-5cb9-9903-4c221d41eb5e',
                    Message: 'Hello from SNS!',
                    ...
                },
            },
        ],
    };

    // Run the Lambda function against this event.
    const result = run(event);

    expect(result).toEqual(true);
});

This strategy also does not utilise your AWS Lambda invocation credits - meaning you are free to run as many tests as often as you like!

Acknowledgements

This boilerplate was first inspired from this post by Jesse Cascio.

License

MIT

es2017-lambda-boilerplate's People

Contributors

greenkeeper[bot] avatar greenkeeperio-bot avatar irvinlim avatar masterful avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

es2017-lambda-boilerplate's Issues

An in-range update of aws-sdk is breaking the build 🚨

The devDependency aws-sdk was updated from 2.350.0 to 2.351.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

aws-sdk is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

Release Notes for Release v2.351.0

See changelog for more information.

Commits

The new version differs by 3 commits.

  • 70cbaa7 Updates SDK to v2.351.0
  • 4073d3d Merge pull request #2342 from srchase/npmignore-additions
  • 13b0dd3 updated npmignore, fix for issue #2341

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

An in-range update of aws-sdk is breaking the build 🚨

The devDependency aws-sdk was updated from 2.368.0 to 2.369.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

aws-sdk is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

Release Notes for Release v2.369.0

See changelog for more information.

Commits

The new version differs by 1 commits.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

An in-range update of aws-sdk is breaking the build 🚨

The devDependency aws-sdk was updated from 2.323.0 to 2.324.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

aws-sdk is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

Release Notes for Release v2.324.0

See changelog for more information.

Commits

The new version differs by 1 commits.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

pb when testing aws-sdk via npm run test

Hi Irvin,

I'm having troubles when testing via "npm run test" (docker-lambda)
I'm trying to return the aws ecs cluster list of my aws account/region, which contains 1 cluster.
but when displaying the cluster list returned by:

const ECS = new AWS.ECS();
var listClusters = await ECS.listClusters().promise();
console.log("PF cluster list: " + util.inspect(listClusters, false, null));

I get an empty list:

START RequestId: e2253bfa-dd82-13ca-7600-1290dc20ae81 Version: $LATEST
2018-04-02T11:06:19.472Z	e2253bfa-dd82-13ca-7600-1290dc20ae81	Event type: aws-ecs-example
2018-04-02T11:06:20.368Z	e2253bfa-dd82-13ca-7600-1290dc20ae81	PF cluster list: { clusterArns: [] }

When I check directely via aws cli, my cluster is returned correctly

11:52 $ aws ecs list-clusters --profile vetup-prod-user-terraform --region eu-west-1
{
    "clusterArns": [
        "arn:aws:ecs:eu-west-1:xxxxxxxxx:cluster/vma-cluster"
    ]
}

Note: My .env file contains the same credential refering by the above "vetup-prod-user-terraform" aws profile:

.env:

AWS_ACCESS_KEY_ID=xxxxxxxx
AWS_SECRET_ACCESS_KEY=xxxxxxxxx
AWS_REGION=eu-west-1

When I execute the lambda directly via aws lambda dashboard, the cluster is also correctly returned (specified the correct lambda "ecs:ListClusters" policy before):

START RequestId: caea0b50-3666-11e8-8584-d19946e08f2c Version: $LATEST
2018-04-02T11:12:55.583Z	caea0b50-3666-11e8-8584-d19946e08f2c	Event type: aws-ecs-example
2018-04-02T11:12:55.825Z	caea0b50-3666-11e8-8584-d19946e08f2c	PF cluster list: { clusterArns: [ 'arn:aws:ecs:eu-west-1:xxxxxxxxxxx:cluster/vma-cluster' ] }

Any idea what I'm missing ?
Thanks you

Capture Lambda output in docker-lambda

In order to simulate the development experience on AWS Lambda itself, some developers might use console.log or similar methods to debug their code on a live Lambda function itself.

While the recommended development workflow for using this boilerplate is TDD (through the use of Jest), it will be incredibly useful to be able to see console.log output in the Jest output as well.

Currently, I think that the system logs are being suppressed at the docker-lambda stage, and I will need to find out how they can be extracted and logged in the test runner.

Unexpected token import

When I tried your boilerplate I get:

{
  "errorMessage": "Unexpected token import",
  "errorType": "SyntaxError",
  "stackTrace": [
    "Object.runInThisContext (vm.js:97:10)",
    "Module._compile (module.js:542:28)",
    "Object.Module._extensions..js (module.js:579:10)",
    "Module.load (module.js:487:32)",
    "tryModuleLoad (module.js:446:12)",
    "Function.Module._load (module.js:438:3)",
    "Module.require (module.js:497:17)",
    "require (internal/module.js:20:19)"
  ]
}

What is wrong ?

An in-range update of aws-sdk is breaking the build 🚨

Version 2.312.0 of aws-sdk was just published.

Branch Build failing 🚨
Dependency aws-sdk
Current Version 2.311.0
Type devDependency

This version is covered by your current version range and after updating it in your project the build failed.

aws-sdk is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

Release Notes Release v2.312.0

See changelog for more information.

Commits

The new version differs by 6 commits.

  • 4ed5104 Updates SDK to v2.312.0
  • 3839328 Create no-response.yml
  • 59f1eb0 remove stalebot
  • 23701ac enable stale bot
  • f5a3482 update typings for credentialsProviderChain & httpOptions (#2240)
  • 167bc31 retry request for Batch TooManyRequestsException (#2246)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

babel configuration

Hi there,
I'm sorry I'm quite new to nodejs and other tools,
tried to run "npm run build" then realised I had not installed babel
After installing babel globally, I realised I needed to install:

env preset
and other babel plugins refered in .babelrc.

         "add-header-comment",
        "transform-es2015-modules-commonjs",
        "transform-es2017-object-entries",
        "transform-object-rest-spread",
        "transform-es2015-destructuring",
        "array-includes"

what would be the right way to install all this component ? (--save-dev ? something else ?)

Thanks you for your help in advance

"npm run package" returns "npm ERR!" with Object.values & Object.entries

Hi Irvin. Glad I came across your package as callback hell was really slowing down my current lambda project.

I receive the following error when attempting "npm run package" on a script which utilizes Object.values || Object.entries. The entire script + async / await appear to work fine when uploaded to AWS without either Object methods.
Terminal output:

npm run package

[email protected] package /Users/arg/CloudDrive/Projects/Personal/CryptoLAMBDA/Binance_CLI
npm run build && npm run package:pack

[email protected] build /Users/arg/CloudDrive/Projects/Personal/CryptoLAMBDA/Binance_CLI
npm run build:init && npm run build:js && npm run build:install

[email protected] build:init /Users/arg/CloudDrive/Projects/Personal/CryptoLAMBDA/Binance_CLI
rm -rf dist && mkdir dist

[email protected] build:js /Users/arg/CloudDrive/Projects/Personal/CryptoLAMBDA/Binance_CLI
cd src && babel . -d ../dist

index.js -> ../dist/index.js
TypeError: main.js: Property body[1] of BlockStatement expected node to be of a type ["Statement"] but instead got "FunctionExpression"
at validate (/Users/arg/CloudDrive/Projects/Personal/CryptoLAMBDA/Binance_CLI/node_modules/babel-types/lib/definitions/in>dex.js:109:13)
at validator (/Users/arg/CloudDrive/Projects/Personal/CryptoLAMBDA/Binance_CLI/node_modules/babel-types/lib/definitions/in>dex.js:57:7)
at Object.validate (/Users/arg/CloudDrive/Projects/Personal/CryptoLAMBDA/Binance_CLI/node_modules/babel-types/lib/definitions/in>dex.js:190:10)
at validate (/Users/arg/CloudDrive/Projects/Personal/CryptoLAMBDA/Binance_CLI/node_modules/babel-types/lib/index.>js:505:9)
at Object.builder (/Users/arg/CloudDrive/Projects/Personal/CryptoLAMBDA/Binance_CLI/node_modules/babel-types/lib/index.>js:466:7)
at NodePath.replaceExpressionWithStatements (/Users/arg/CloudDrive/Projects/Personal/CryptoLAMBDA/Binance_CLI/node_modules/babel-traverse/lib/path/replac>ement.js:206:54)
at NodePath.insertBefore (/Users/arg/CloudDrive/Projects/Personal/CryptoLAMBDA/Binance_CLI/node_modules/babel-traverse/lib/path/modifi>cation.js:52:10)
at NodePath.unshiftContainer (/Users/arg/CloudDrive/Projects/Personal/CryptoLAMBDA/Binance_CLI/node_modules/babel-traverse/lib/path/modifi>cation.js:239:15)
at File.addHelper (/Users/arg/CloudDrive/Projects/Personal/CryptoLAMBDA/Binance_CLI/node_modules/babel-core/lib/transformation/>file/index.js:388:17)
at PluginPass.addHelper (/Users/arg/CloudDrive/Projects/Personal/CryptoLAMBDA/Binance_CLI/node_modules/babel-core/lib/transformation/>plugin-pass.js:46:42)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build:js: 'cd src && babel . -d ../dist'
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] build:js script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /Users/arg/.npm/_logs/2018-01-27T07_33_28_532Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build: 'npm run build:init && npm run build:js && npm run build:install'
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /Users/arg/.npm/_logs/2018-01-27T07_33_28_548Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] package: 'npm run build && npm run package:pack'
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] package script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /Users/arg/.npm/_logs/2018-01-27T07_33_28_565Z-debug.log

I've added your email to my private repo if you need to checkout the source file.

An in-range update of aws-sdk is breaking the build 🚨

The devDependency aws-sdk was updated from 2.330.0 to 2.331.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

aws-sdk is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Release Notes for Release v2.331.0

See changelog for more information.

Commits

The new version differs by 1 commits.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

Pass AWS credentials to automated tests

There should be a way for automated tests to access the AWS SDK with credentials.

Normally on AWS Lambda, you can specify the IAM role, but through docker-lambda, we can pass the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables using the addEnvVars option, which has to be a IAM user.

For passing this to the test runner, Travis accepts environment variables which will be forwarded to process.env. We can use dotenv, which will work for local test runs as well.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.