streambot

Deprecated

As of 11/18/2016, AWS introduced a native mechanism for providing environment variables to lambda functions, which is also supported by CloudFormation. These features negate the gap that streambot existed to fill. You should not use streambot to configure new Lambda functions.

A Dear Bot's Farewell

oh faithful streambot
supplying runtime config
to stateless functions

without fail you kept
functions wrapped with care, and safe
from context dot done

a misnomer, sure
your creator may chuckle
but things change, you know

now we bid adieu
to that one point of failure
the streambot table

event source mappings
service tokens, streambot envs
farewell our dear bot

no more consumers
deprecation awaits you
we'll see you never

by @emilymcafee

Streambot is a tool to help you manage AWS Lambda functions as persistent services via AWS CloudFormation templates. It provides assistance for common scenarios where existing CloudFormation support is still inadequate.

Runtime configuration

Lambda functions are inherently stateless, but a well-designed Twelve-factor app separates configuration from code. The existing Lambda API offers no way for you to configure your application differently across multiple deploys. Streambot provides helper functions to manage and load runtime configuration parameters separately from your code.

EventSourceMappings

An EventSourceMapping connects a Kinesis or DynamoDB stream to a Lambda function, causing records in the stream to trigger Lambda invocations. Presently, there is no support for EventSourceMappings as standalone CloudFormation resources. Streambot provides a way to manage EventSourceMappings through a custom CloudFormation resource.

Node.js function bundling

In order to manage a Lambda function via CloudFormation, the code behind your function must be bundled into a .zip file and uploaded to S3. If your function has dependencies that include C++ addons precompiled via node-pre-gyp, Streambot will help you bundle your code with dependencies for the correct platform/architecture to run on AWS Lambda, regardless of your native OS.

Installation

The Streambot stack

To use Streambot, you must run a single CloudFormation stack in your AWS account using Streambot's primary template. To do this, clone and build Streambot:

$ git checkout https://github.com/mapbox/streambot
$ cd streambot
$ npm install
$ npm run-script build
# Template file is written to `./cloudformation/streambot.template`
# See ./docs/streambot.template.html for details of the resources that are created

Now you can start a new CloudFormation stack using the template. You may do this using the web console or using an automation tool such as cfn-config. Note: you must create this template in the us-east-1 region. If you don't, your Lambda functions will not know where to find their runtime configuration.

Your service stack

Once the Streambot stack has been created, you're ready to write your own Lambda-backed services. The streambot-example folder provides an example of how a service might look. There are a number of concepts to grasp as you embark on this journey.

Bundling your code with streambot

Streambot provides a wrapper function for you to put around your code. This wrapper makes sure that any runtime configuration for your function is loaded into the environment before your function runs. To use this streambot wrapper, you'll adjust your module's index.js file to look like:

var streambot = require('streambot');

function myFunction(event, callback) {
  // This is your custom code to process an event.
  // fire the `callback` when finished, in familiar Node.js-style (err first,
  // result after).

  // Using the streambot wrapper means that `process.env` will be populated with
  // your runtime configuration.
}

module.exports.streambot = streambot(myFunction);

If your function has dependencies that include C++ addons precompiled via node-pre-gyp, you can use Streambot's bin/bundle script to bundle your code with dependencies for the correct platform/architecture to run on AWS Lambda, regardless of your native OS. Once you've bundled your function, you need to upload it to S3.

If your function doesn't have dependencies that rely on C++ addons, you can create a ZIP file yourself with any archiving tool and upload it to S3.

Remember where you upload the bundles, because you'll need to specify their location when you create the Lambda function.

Writing your CloudFormation template

Your CloudFormation template will need to create a Lambda function and also an IAM role defining that function's permissions. Using Streambot may mean adding a couple more resources, depending on how you wish to use it.

IAM Role

You'll have to make an IAM role, and it should completely cover any permissions that your function needs in order to do what it is supposed to do. Additionally, you need to make sure that your function has permission to read runtime configuration from the DynamoDB table created by Streambot's primary template.

{
    "Effect": "Allow",
    "Action": [
        "dynamodb:GetItem"
    ],
    "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/streambot-env*"
    }
}

StreambotEnv

If your Lambda function needs runtime configuration, you'll need to create a a custom CloudFormation resource backed by the StreambotEnv Lambda function created as part of the primary Streambot template. This custom resource will manage a record in DynamoDB that your Lambda functions can read their runtime configuration from.

{
    "Type": "Custom::StreambotEnv",
    "Properties": {
        "ServiceToken": "arn:aws:lambda:us-east-1:123456789012:function:StreambotEnvFunction",
        "FunctionName": "my-lambda-function",
        "EventBucket": "my-event-bucket",
        "EventPrefix": "my-event-prefix"
    }
}

StreambotConnector

If your Lambda function intends to read from a Kinesis or DynamoDB stream, you'll need to create an EventSourceMapping between your stream and your Lambda function. There is not presently CloudFormation support for these resources. Instead, you can use another a custom CloudFormation resource, this one backed by the StreambotConnector Lambda function created as part of the primary Streambot template.

{
        "Type": "Custom::StreambotConnector",
        "DependsOn": "StreambotExampleConfiguration",
        "Properties": {
            "ServiceToken": "arn:aws:lambda:us-east-1:123456789012:function:StreambotConnectorFunction",
            "FunctionRegion": "us-east-1",
            "FunctionName": "my-lambda-function",
            "StreamArn": "arn:aws:kinesis:us-east-1:123456789012:stream/my-kinesis-stream",
            "BatchSize": 1,
            "StartingPosition": "LATEST"
        }
    }

Example

The example service in the streambot-example folder demonstrates a Lambda-backed service that uses Streambot. The purpose of the service is simply to read records from a Kinesis stream, and write them verbatim to files on S3. See ./docs/streambot-example.template.html and ./docs/streambot-example.index.html for detailed discussion of the example template and code.