CodeDeploy is stuck in middle of pre-validation step and then rolls back.

any clue?

We found that when we have a lambda function which is scheduled through a cloud watch event rule of every x minutes. the lambda fails to trigger and we see a specific error as shown below and it needs to be fixed manually in the AWS console

serverless.yml code:

Sample function:
handler: src/index.js
description: refresh tokens
events:
- schedule: rate(5 minutes)
deploymentSettings:
type: AllAtOnce #Canary10Percent5Minutes, Linear10PercentEvery1Minute, AllAtOnce
alias: Live
preTrafficHook: preHook
postTrafficHook: postHook

Hi,

I'd love the idea of the plugin, but I can't seem to get it to work based on Alarms I created. I do not use the serverless-plugin-aws-alerts.

plugins:
  - serverless-plugin-split-stacks
  - serverless-plugin-warmup
  - serverless-prune-plugin
  - serverless-plugin-include-dependencies
  - serverless-plugin-canary-deployments

 functionName:
    deploymentSettings:
      type: Linear10PercentEvery1Minute
      alias: functionNameLive
      alarms:
        - TestAlarm

with the following policy attached

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "cloudwatch:DescribeAlarms",
                "lambda:*",
                "sns:Publish"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": "arn:aws:s3:::*/CodeDeploy/*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "s3:ExistingObjectTag/UseWithCodeDeploy": "true"
                }
            },
            "Effect": "Allow"
        }
    ]
}

The alarm is based on a metric filter and named 'TestAlarm'.

On sls deploy the following error is thrown
The CloudFormation template is invalid: Template format error: Unresolved resource dependencies [TestAlarm] in the Resources block of the template

I first used an alarm name with dashes (-) in it, figured that might be the problem. So I created a new alarm named 'TestAlarm', but still can't deploy. Started looking into logical IDs and tried some different variations, but with no success.

Does anyone have an idea what goes wrong?

Thank you

Nice work,

So If my serverless configuration included a number of SNS, SQS, Lambda resources, would each of these resources be deployed with a new version as new resources, fully isolated from the previous version?

If yes, is the deploy atomic? I.e. if there are any errors, will the entire deploy rollback?

Repository that reproduces the problem: https://github.com/humodz-forks/serverless-canary-concurrency-issue

What are the steps to reproduce this issue?

1. In serverless.yml, add provisionedConcurrency: 1 to the function's configuration
2. Deploy
3. Attempt to call the function via API Gateway

What happens?

It returns an internal server error

In the linked repository, this endpoint doesn't use provisionedConcurency and is able to invoke the function:

curl https://API-ID.execute-api.us-east-1.amazonaws.com/dev/works

This one doesn't use provisionedConcurency and always returns Internal Server Error:

curl https://API-ID.execute-api.us-east-1.amazonaws.com/dev/broken

What were you expecting to happen?

It should be able to invoke the function, even with provisionedConcurrency

Any logs, error output, etc?

When provisionedConcurrency is set, Serverless creates a "provisioned" alias and an AWS::Lambda::Permission resource that looks like this:

"Type": "AWS::Lambda::Permission",
"Properties": {
  "FunctionName": {
    "Fn::Join": [
      ":",
      [
        {
          "Fn::GetAtt": [
            "BrokenLambdaFunction",
            "Arn"
          ]
        },
        "provisioned"
      ]
    ]
  },

Since this plugin creates another alias and makes API Gateway use it instead of the above one, API Gateway won't have permission to invoke the function.

Any other comments?

The linked repository has a plugin (commented out in serverless.yml) that attempts to fix the issue, by modifying the Permission resource.

What versions of software are you using?

Node v10.18.1
[email protected]
[email protected]

Love the plugin - but is there a way to use the pre/post traffic hook but not the traffic shifting? I want the canary option but not to wait a long time for every deployment; sometimes I need to deploy things quickly.

Thanks!

P.S. it doesn't seem to be creating the CodeDeploy IAM policy automatically for me:

[ERROR] ClientError: An error occurred (AccessDeniedException) when calling the PutLifecycleEventHookExecutionStatus operation: User: arn:aws:sts::898466151470:assumed-role/myapp-mish-eu-central-1-lambdaRole/myapp-mish-preHookCanary is not authorized to perform: codedeploy:PutLifecycleEventHookExecutionStatus on resource: arn:aws:codedeploy:eu-central-1:898466741470:deploymentgroup:myapp-mish-myappmishDeploymentApplication-DP39TAO2TZ28/myapp-mish-AppLambdaFunctionDeploymentGroup-FUGL97F0ZMEU
Traceback (most recent call last):
  File "/var/task/myapp/handler/canary.py", line 33, in pre_traffic_handler
    codedeploy.put_lifecycle_event_hook_execution_status(**params)
  File "/var/runtime/botocore/client.py", line 320, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 623, in _make_api_call
    raise error_class(parsed_response, operation_name)

Hi, first of all thank you for creating this plugin. It will make my life so much easier when I get it working. Not sure if this is a bug or I'm not doing it properly but every time I try to deploy using the plugin i get the following:

Serverless: Validating template...
 
  Error --------------------------------------------------
 
  The CloudFormation template is invalid: [/Resources/GetDashtickerDashdropdownsLambdaFunctionAliasLive/Type/FunctionVersion/Fn::GetAtt/0] 'null' values are not allowed in templates
 
     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
 
  Stack Trace --------------------------------------------
 
Error: The CloudFormation template is invalid: [/Resources/GetDashtickerDashdropdownsLambdaFunctionAliasLive/Type/FunctionVersion/Fn::GetAtt/0] 'null' values are not allowed in templates
    at provider.request.catch (/usr/lib/node_modules/serverless/lib/plugins/aws/deploy/lib/validateTemplate.js:25:13)
From previous event:
    at AwsDeploy.validateTemplate (/usr/lib/node_modules/serverless/lib/plugins/aws/deploy/lib/validateTemplate.js:20:12)
From previous event:
    at AwsDeploy.BbPromise.bind.then (/usr/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:121:39)
From previous event:
    at Object.aws:deploy:deploy:validateTemplate [as hook] (/usr/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:117:10)
    at BbPromise.reduce (/usr/lib/node_modules/serverless/lib/classes/PluginManager.js:390:55)
From previous event:
    at PluginManager.invoke (/usr/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.spawn (/usr/lib/node_modules/serverless/lib/classes/PluginManager.js:408:17)
    at AwsDeploy.BbPromise.bind.then (/usr/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:95:48)
From previous event:
    at Object.deploy:deploy [as hook] (/usr/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:91:10)
    at BbPromise.reduce (/usr/lib/node_modules/serverless/lib/classes/PluginManager.js:390:55)
From previous event:
    at PluginManager.invoke (/usr/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.run (/usr/lib/node_modules/serverless/lib/classes/PluginManager.js:421:17)
    at variables.populateService.then.then (/usr/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:121:23)
From previous event:
    at Serverless.run (/usr/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/usr/lib/node_modules/serverless/bin/serverless:43:50)
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           10.12.0
     Serverless Version:     1.32.0
 
Exited with code 1

The deployment settings for my function are:

deploymentSettings:
      type: Linear10PercentEvery1Minute
      alias: Live

Would anyone know why? Thanks

The readme.md states "Linear10PercentEvery10Minutes: shifts 10 percent of traffic every 30 minutes until all traffic is shifted."

Should it be "Linear10PercentEvery10Minutes: shifts 10 percent of traffic every 10 minutes until all traffic is shifted."?

Or the Configuration name should be Linear10PercentEvery30Minutes

We suddenly noticed the following error on our serverless deployment build script, this was working until yesterday and there were no changes. Any idea why we get this error please? We execute python script on docker to install serverless plugins before we execute sls deploy command

Error: Cannot find module 'lodash/fp' at Function.Module._resolveFilename (internal/modules/cjs/loader.js:668:15) at Function.Module._load (internal/modules/cjs/loader.js:591:27) at Module.require (internal/modules/cjs/loader.js:723:19) at require (internal/modules/cjs/helpers.js:14:16) at Object.<anonymous> (/tmp/build/put/jar-repository/node_modules/serverless-plugin-canary-deployments/serverless-plugin-canary-deployments.js:1:11)

Operating System: linux Node Version: 11.15.0 Framework Version: 1.74.1 Plugin Version: 3.6.14 SDK Version: 2.3.1 Components Version: 2.31.7

serverless/serverless#7923

Hi. Thanks for this plugin cant be overstated how useful it is. I am having trouble defining and global alias and type. I have the following setup but it does not create the code deploy applications. When I place the deploymentSettings inside each function however it works perfectly. If this is not the current use case I think it should be an option and if its not desired behavior I think the documentation should be updated to be more clear since it implies that the setup below should work.

provider:
  name: aws
  runtime: python3.6
  region: us-west-2
  endpointType: regional
  managedPolicyArns:
    - "arn:aws:iam::aws:policy/service-role/AWSCodeDeployRoleForLambda"


plugins:
  - serverless-attach-managed-policy
  - serverless-plugin-canary-deployments

custom:
  deploymentSettings:
    type: Canary10Percent30Minutes
    alias: Production
    stages:
      - prod

  stage: ${opt:stage}


functions:
  someFunc:
    handler: someFunc.someFunc
    runtime: python3.6
    events:
      - http:
          path: someFunc
          method: POST
          cors: true
          authorizer:
            arn: ${self:custom.cognitoArn}```

(Thanks for reporting an issue! Please, then fill out the blanks below.)

What are the steps to reproduce this issue?

1. Go to https://github.com/davidgf/serverless-plugin-canary-deployments/releases

What happens?

Does not see versions newer than 0.5.0

What were you expecting to happen?

at least see 0.6.0 ?

Any logs, error output, etc?

(If it’s long, please paste to https://pastebin.com/ and insert the link here.)

Any other comments?

…

What versions of software are you using?

I would like to run tests against the canary env when deployed so I assume the easiest way to do this is to transfer 100% of the traffic over for testing, is this possible?

(Thanks for reporting an issue! Please, then fill out the blanks below.)

What are the steps to reproduce this issue?

sls deploy

What happens?

[11:19:22][Step 1/3] Serverless Error ----------------------------------------
[11:19:22][Step 1/3]
[11:19:22][Step 1/3] An error occurred: CodeDeployServiceRole - Policy arn:aws:iam::aws:policy/AWSLambdaFullAccess does not exist or is not attachable. (Service: AmazonIdentityManagement; Status Code: 404; Error Code: NoSuchEntity; Request ID: a40989a6-a677-41a1-88db-c0c455317843; Proxy: null).
[11:19:22][Step 1/3]
…
(After March 1, 2021, the AWS managed policies AWSLambdaReadOnlyAccess and AWSLambdaFullAccess will be deprecated and can no longer be attached to new IAM users. For more information about policy deprecations, see Deprecated AWS managed policies in the IAM User Guide.)
https://docs.aws.amazon.com/lambda/latest/dg/security_iam_troubleshoot.html#security_iam_troubleshoot-admin-deprecation

The AWSLambdaFullAccess is being used in the current release.
check here
https://github.com/davidgf/serverless-plugin-canary-deployments/search?q=AWSLambdaFullAccess

What were you expecting to happen?

…

Any logs, error output, etc?

(If it’s long, please paste to https://pastebin.com/ and insert the link here.)

Any other comments?

…

What versions of software are you using?

v0.5.0

(Thanks for reporting an issue! Please, then fill out the blanks below.)

What are the steps to reproduce this issue?

1. create a deploy with alarms
2. have the alarm fire
3. When a rollback is executed because of the alarm no rollback event is fired

What happens?

When a rollback is executed because of the alarm no rollback event is fired

What were you expecting to happen?

would expect a rollback:initialize or rollback:rollback to fire when a rollback is started

Any logs, error output, etc?

https://pastebin.com/kfHg0V9g

Any other comments?

I am using a simple plugin to echo all the life cycle events

'use strict';
const _ = require('lodash');

class LifecyclePrinter {
    constructor(serverless, options) {
        this.serverless = serverless;
        this.options = options;

        const service = this.serverless.service;
        this.hooksToWatch = _.get(service, 'custom.lifeCycle.hooks');
        this.debug = _.get(service, 'custom.lifeCycle.debug');

        this.provider = this.serverless.getProvider('aws');

        this.hooks = {};

        for (let event in this.serverless.pluginManager.hooks) {
            if (event.startsWith('before:') || event.startsWith('after:')) {
                this.hooks[event] = this.hook.bind(this, event);
            } else {
                const beforeEvent = 'before:' + event;
                this.hooks[beforeEvent] = this.hook.bind(this, beforeEvent);

                const afterEvent = 'after:' + event;
                this.hooks[afterEvent] = this.hook.bind(this, afterEvent);
            }
        }

    }

    hook(event) {
        // eslint-disable-next-line no-console
        console.log(' IN: %s', event);
    }
}

module.exports = LifecyclePrinter;

I also tried all of these in the debug plugin

this.hooks = { 'after:webpack:package:packExternalModules': this.packExternalModules.bind(this), 'before:aws:deploy:deploy:updateStack': this.updateStack.bind(this), 'after:rollback:rollback': this.rollback.bind(this), 'before:rollback:rollback': this.rollback.bind(this), 'before:rollback:initialize': this.rollback.bind(this), 'after:rollback': this.rollback.bind(this), 'rollback': this.rollback.bind(this), 'aws:rollback:initialize': this.rollback.bind(this), 'aws:rollback': this.rollback.bind(this), 'before:aws:rollback': this.rollback.bind(this), 'before:aws:rollback:rollback': this.rollback.bind(this) };

None of these fired

thanks for your help

What versions of software are you using?

"serverless-plugin-canary-deployments": "^0.4.8",

What are the steps to reproduce this issue?

1. Create a function with deploymentSettings including a preTrafficHook:

functions:
  http:
    handler: dist/lambda/http.http
    events:
      - httpApi: '*'
    deploymentSettings:
      type: AllAtOnce
      alias: Live
      preTrafficHook: deployPreTraffic

  deployPreTraffic:
    handler: dist/lambda/deploy.preTraffic
    timeout: 300 # 5m

2. Make deployPreTraffic notify CodeDeploy of success:

const { CodeDeploy } = require('aws-sdk');
const codeDeploy = new CodeDeploy({ apiVersion: "2014-10-06" });

module.exports.preTraffic = async (event) => {
  await codeDeploy
    .putLifecycleEventHookExecutionStatus({
      deploymentId: event.DeploymentId,
      lifecycleEventHookExecutionId: event.LifecycleEventHookExecutionId,
      status: "Succeeded",
    })
    .promise();
};

3. Deploy the application
4. Check the logs of the deployPreTraffic function

What happens?

The deployPreTraffic function fails because the Serverless-generated Lambda execution role doesn't grant it permission to set the deployment status. The CodeDeploy hook therefore times out after an hour and the deployment fails.

What were you expecting to happen?

The Serverless-generated Lambda execution role should automatically include permission for the deployment hook functions to update the deployment status. The deployment with hook functions should work with no extra IAM configuration.

Any logs, error output, etc?

AccessDeniedException: User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/xxxx-api-production-us-west-2-lambdaRole/xxxx-api-production-deployPreTraffic is not authorized to perform: codedeploy:PutLifecycleEventHookExecutionStatus on resource: arn:aws:codedeploy:us-west-2:xxxxxxxxxxxx:deploymentgroup:xxxx-api-production-XXXXapiproductionDeploymentApplication-XXXXXXXXXXXX/xxxx-api-production-HttpLambdaFunctionDeploymentGroup-XXXXXXXXXXXX

Any other comments?

It's possible to work around this issue by manually adding the relevant permissions in serverless.yml:

provider:
  iamRoleStatements:
    - Effect: Allow
      Action:
        - 'codedeploy:PutLifecycleEventHookExecutionStatus'
      Resource:
        Fn::Join:
          - ''
          - - 'arn:aws:codedeploy:'
            - Ref: AWS::Region
            - ':'
            - Ref: AWS::AccountId
            - ':deploymentgroup:'
            - Ref: XXXXapiproductionDeploymentApplication
            - '/*'

What versions of software are you using?

serverless: 1.78.1
serverless-plugin-canary-deployments: 0.4.8

Hello,
I just found out there is an issue when trying to run this package on stage that is not whitelisted.
I have the same configuration in plugin serverless-plugin-aws-alerts and as it turns out, even if alarms are not used (because stage is disabled), CloudFormation still tries to attach them to lambda.
Since alarm is not created in previous plugin based on the same check you have, CloudFormation fails to attach it.

Correct behaviour (for me) would be that if stage is disabled, all functionality of this package is disabled as well.

I can create PR for this but first wanted to get an opinion on this, so I won't do PR that's going to be closed anyway.

Cheers

Hi and thanks for an awesome plugin!

Is there a way to get hold of a reference to the new lambda so it can be called and verified from the pre hook? Like NewVersion is being passed in implementing-safe-aws-lambda-deployments-with-aws-codedeploy

npm pulls the latest tag (by default) but there are some good bug fixes in master

Sorry if this isn't in the appropriate format of a typical issue. I'm considering the use of this plugin to orchestrate canary deployments with my lambda functions. The supported event sources don't list the ALB event source as one of them. Is it accurate to say that this plugin doesn't support this event source yet? If yes, I'm open to contributing a PR that could potentially address this.

Any help would be appreciated 🙏. Thank you

Bug Report

Description

After updating to serverless 1.60.4 (from 1.37.1) deployed stack started to return {"message": "Internal server error"} for all calls.

Tried to run "test" from AWS APIGW "Resources"->"Method" view and got: Thu Dec 26 11:52:32 UTC 2019 : Execution failed due to configuration error: Invalid permissions on Lambda function

Rolled back to 1.37.1 and deploy worked fine.

After some digging in cloud trails found that serveless 1.60.4 didn't give permissions to the "alias" version of our lambda:

Setup info (parts I thought are relevant):

plugins:
  - serverless-plugin-canary-deployments

provider:
  name: aws
  runtime: python3.6

functions:
  - app:
      handler: wsgi_handler.handler
      timeout: 30
      deploymentSettings:
        type: AllAtOnce
        postTrafficHook: tests
        alias: live

If you have a serverless.yml file without a 'custom' section the deployment will fail with the following error message:

Type Error ---------------------------------------------
 
  Cannot read property 'deploymentSettings' of undefined
 
     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
 
  Stack Trace --------------------------------------------
 
TypeError: Cannot read property 'deploymentSettings' of undefined
    at ServerlessCanaryDeployments.getDeploymentSettingsFor (/.../node_modules/serverless-plugin-canary-deployments/serverless-plugin-canary-deployments.js:210:60)
    at ServerlessCanaryDeployments.buildFunctionResources (/.../node_modules/serverless-plugin-canary-deployments/serverless-plugin-canary-deployments.js:55:37)
    at _.flatMap.serverlessFunction (/.../node_modules/serverless-plugin-canary-deployments/serverless-plugin-canary-deployments.js:48:34) 
...

Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           8.10.0
     Serverless Version:     1.26.1

Cheers
Noah

Obviously I'm an idiot, but when I read the docs and attempt a simple linear 10% over 10 minutes release, it just puts the new release live immediately - 100% of traffic is going to the new release and I don't see anything in CodeDeploy to tell me otherwise. Doesn't appear that any traffic balancing is going on at all.

I tested by performing "serverless deploy --stage dev" without the deploymentSettings on the Login function, then modifying my Login function to output some different content, putting in the deploymentSettings YAML, recompiled it and performed another deploy.

Here is my setup:

provider:
  name: aws
  runtime: go1.x
  role: arn:aws:iam::xxxxxxxxxxxxxx:role/ApiHubLambdaRole
  region: eu-west-1
  memorySize: 128
  iamRoleStatements:
    - Effect: Allow
      Action:
        - codedeploy:*
      Resource:
        - "*"

package:
  individually: true
  exclude:
    - ./**

plugins:
  - serverless-plugin-canary-deployments

custom:
  deploymentSettings:
    type: Linear10PercentEvery10Minutes
    alias: Development
    stages:
      - dev
  stage: dev

functions:
  Login:
    handler: bin/login/main
    package:
      include:
        - bin/login/main
    events:
      - http:
          path: /signin
          method: post
    deploymentSettings:
      type: Linear10PercentEvery10Minutes

Thank you for this great plugin!

I've successfully used the plugin to configure canary deployments using the example format in the README.md. This works well but becomes very verbose when configuring multiple alarms for hundreds of lambda functions.

We create standard alarms (Error, Duration, Throttles) using the https://github.com/ACloudGuru/serverless-plugin-aws-alerts plugin.

Currently every function has a deployment block like the following:

...standard function config...
    deploymentSettings:
      alias: live
      type: Linear10PercentEvery1Minute
      alarms:
        - <Function_Name>FunctionErrorsAlarm
        - <Function_Name>FunctionDurationAlarm
        - <Function_Name>FunctionThrottlesAlarm

This is a highly manual and error prone process as the names must match exactly. Is making the alarm binding automatic possible? If not, would you be willing to accept a PR that added this functionality?

Thanks!

So I tried following the readme to implement linear deployments into my existing project, and the result is, well, everything stays the same, serverless deploys everything in one go.
Are there any prerequisites, like do I need to create something manually on AWS console CodeDeploy, ...?

This is what I have in my severless.yml:

service: image-processing

plugins:
  - serverless-plugin-canary-deployments
  - serverless-plugin-scripts
  - serverless-plugin-thundra-lambda-adapters-cw
  - serverless-domain-manager

provider:
  name: aws
  runtime: nodejs8.10
  stage: ${opt:stage, self:custom.defaultStage}
  profile: ${file(.env.${self:provider.stage}.yml):AWS_PROFILE}
  region: ${file(.env.${self:provider.stage}.yml):AWS_REGION}


  # *** IAM ROLE ***
  iamRoleStatements:
      # Code deploy permission
      - Effect: Allow
        Action:
          - codedeploy:*
        Resource:
          - "*"
...
custom:
 ...
  deploymentSettings:
    type: Linear10PercentEvery1Minute
    alias: Live
    stages:

package:
  individually: true
  exclude:
    - ./**  # Exclude all files (needed files are included later for each function)

layers:
  imageProcessing:
    path: image-processing-layer
    package:
      include:
        - "./**"
    compatibleRuntimes:
      - nodejs8.10

functions:
  image-processing:
    handler: image-processing/index.handler
    runtime: nodejs8.10
    timeout: 300
    memorySize: 256
    layers:
      - {Ref: ImageProcessingLambdaLayer}
      - arn:aws:lambda:::awslayer:AmazonLinux1803
    package:
      include:
        - image-processing/**
        - thundra_handlers/**
        - '!node_modules/**'
        - '!image-processing/node_modules/**'
        - '!image-processing/.git/**'
        - '!image-processing/test/**'
        - '!image-processing/img/**'
        - '!image-processing/coverage/**'
        - '!image-processing/.**'
        - '!image-processing/event**'
        - '!image-processing/Dockerfile'
        - '!image-processing/config_**'
        - '!image-processing/tests.js'
    events:
      - sns: ImageProcessing
    deploymentSettings:
      type: Linear10PercentEvery1Minute
      alias: Live

What am I doing wrong or missing?

Seems to be the serverless policy

Currently, a failure in the tests will be shown as a failure to update the lambda alias.
Is it possible to add the test function logs to the deploy log - so it will be easier to see what failed?

I'm mostly using SQS events. using code deploy seems to work by traffic shifting events to new lambdas.

in the readme it seems that this plugin doesn't support it. am I missing something? what should be added in order to support it?

I think this plugin is my best shot at true blue/green deployments using serverless. However, I'd like to use the preTrafficHook before switch to allow an end to end test that uses real HTTP requests, selenium, etc to run, but there is no temporary environment to actually run these tests (an actual endpoint). It's possible to use CodeDeploy to get the version of the lambda that will be deployed and manually invoke test requests against that, but there are a few things specific to API gateway that I would like to test as well. Is this possible?

I was looking at the serverless-domain-manager plugin as well to simply have two seperate stages - a production-current, and production-pending, but I've been unable to find a way to switch the DNS for these environments without having propogation lag/downtime.

Thanks.

(Thanks for reporting an issue! Please, then fill out the blanks below.)

What are the steps to reproduce this issue?

1. install serverless 2.14
2. add serverless-plugin-canary-deployments as a plugin to your serverless.yml
3. make a deployment

What happens?

There is a configuration validation warning

What were you expecting to happen?

No warning for something that is part of the plugin

Any logs, error output, etc?

Serverless: Configuration warning:
Serverless: at 'functions['hosting-api']': unrecognized property 'deploymentSettings'
Serverless: at 'functions['hosting-data-sync-handler']': unrecognized property 'deploymentSettings'
Serverless:
Serverless: Learn more about configuration validation here: http://slss.io/configuration-validation

Any other comments?

Can you have this plugin's attribute additions added to the configuration-validator?

What versions of software are you using?

"serverless": "^2.14.0",
"serverless-plugin-canary-deployments": "^0.4.8"

Within deploymentSettings we can configure a type and have to choose one of the predifined types listed here https://github.com/davidgf/serverless-plugin-canary-deployments#configuration .
At this moment we have a security policy that defines we can only use / modify resources that are prefixed with our team name. Since the type you fill in is prepended with CodeDeployDefault.Lambda hardcoded here https://github.com/davidgf/serverless-plugin-canary-deployments/blob/master/lib/CfTemplateGenerators/CodeDeploy.js#L29 we are blocked.

What are the steps to reproduce this issue?

1. Configure a type that is created yourself, for example

    deploymentSettings:
      type: OurTeam-Linear10PercentEvery1Minute

What happens?

We get an error stating that the deploymentConfig with name CodeDeployDefault.LambdaOurTeam-Linear10PercentEvery1Minute does not exist.

What were you expecting to happen?

It should not prefix with CodeDeployDefault.Lambda by default. The quickfix could be to update the documentation and mention the types with the prefix CodeDeployDefault.Lambda and then it will also support custom DeploymentConfigs.
That change however will not be backwards compatible, so I propose we create a new property DeploymentConfigPrefix which has a default value of CodeDeployDefault.Lambda but can be overridden to an empty value.

What versions of software are you using?

serverless-plugin-canary-deployments : 0.4.8

Last week, I had a working deployment using blue/green with postHook and preHook tests. Deploying the same code since 0.5.0 was released for serverless-plugin-canary-deployments results in a CloudFormation error

What are the steps to reproduce this issue?

1. … Deploy a previously working serverless install using 0.5.0 plugin version

What happens?

Serverless: Validating template...

Error -------------------------------------------------- Error: The CloudFormation template is invalid: Circular dependency between resources: []
at /usr/lib/node_modules/serverless/lib/plugins/aws/deploy/lib/validateTemplate.js:20:13
at runNextTicks (internal/process/task_queues.js:58:5)
at processImmediate (internal/timers.js:434:9)

What were you expecting to happen?

Blue/Green deploy on AWS

Can currently solve by installing the 0.4.8 plugin, but thought it worth flagging here.

Please add ALB events as well to the list of supported events.

Appreciate it!

Any other comments?

In the current list of events supported by serverless-plugin-canary-deployments, ALB events are not supported. With Lambdas being exposed over ALB (instead of API Gateway) is becoming a popular pattern since AWS announced it 18 months ago, it makes a perfect puzzle piece to address.

Looking forward to what you think!

What are the steps to reproduce this issue?

I literally cloned the example app: https://github.com/davidgf/sls-canary-example
Created a new origin and pushed the code to the new origin. I simply changed the runtime version from nodejs6.10 to nodejs12.x. Then ran the deploy:

SLS_DEBUG=1 sls deploy --verbose --stage dev --region us-east-2

What happens?

The stack successfully deployed to CloudFormation. I noticed it created a CodeDeployServiceRole. But it did not create a CodeDeploy Application or Deployment. I expected to see a CodeDeploy deployment so I could see the lambdas deployed and the traffic migrated. Why don't I see a CodeDeploy application and group from this plugin?

What are the steps to reproduce this issue?

1. I've SLS setup to deploy Lambda with deployment type Canary10Percent5Minutes, while monitoring three Alerts.
2. Sometimes I need to deploy ASAP regardless of the status of the Alerts, so I thought that using deployment type AllAtOnce would fit here

What happens?

AllAtOnce doesn't prevent deployment to fail if any of Alerts is triggered.

What were you expecting to happen?

I expected that only the validation hooks would be taken into consideration while deploying with type AllAtOnce:
AllAtOnce: shifts all the traffic to the new version, useful when you only need to execute the validation hooks.

Any logs, error output, etc?

...

Any other comments?

…

What versions of software are you using?

"serverless": "^1.66.0",
"serverless-offline": "^5.12.1",
"serverless-plugin-canary-deployments": "^0.4.8"

Hi

I have successfully enabled the canary deployment plugin in our serverless application. But now I a stuck with how can I add integration test towards the newly deployed lambda version in the preHook handler.
I am using "AllAtOnce" canary configuration. I am looking for a way for executing test agianst the new lambda version.

My preHook handler is as follows. Please suggest...

========================================
const aws = require('aws-sdk');
const codedeploy = new aws.CodeDeploy({apiVersion: '2014-10-06'});

module.exports.pre = (event, context, callback) => {
//Read the DeploymentId from the event payload.
var deploymentId = event.DeploymentId;

//Read the LifecycleEventHookExecutionId from the event payload
var lifecycleEventHookExecutionId = event.LifecycleEventHookExecutionId;

console.log('We are running some integration tests before we start shifting traffic...');
console.log('DeploymentId: ' + deploymentId);
console.log('BUT NOT SURE HOW TO RUN TEST HERE  AGAINST THE NEW VERSION!'');

/*
Enter validation tests here.
*/
// Prepare the validation test results with the deploymentId and
// the lifecycleEventHookExecutionId for AWS CodeDeploy.
var params = {
deploymentId: deploymentId,
lifecycleEventHookExecutionId: lifecycleEventHookExecutionId,
status: 'Succeeded' // status can be 'Succeeded' or 'Failed'
};

// Pass AWS CodeDeploy the prepared validation test results.
codedeploy.putLifecycleEventHookExecutionStatus(params, function(err, data) {
if (err) {
// Validation failed.
callback('Validation test failed');
} else {
// Validation succeeded.
callback(null, 'Validation test succeeded');
}
});
};

Sns subscription is not pointing to alias when the configuration as below

canarySns:
  handler: src/event/lambda.canarySns
  description: (SNS) test canary sns binding
  deploymentSettings: ${self:custom.deployment}
  events:
    - sns:
        topicName: fooTopic
        displayName: fooTopic
        filterPolicy:
          eventType:
            - FooEvent

What are the steps to reproduce this issue?

1. npm install --save [email protected]
2. sls deploy

What happens?

Deployment failure due to AWS deprecating the managed role AWSLambdaFullAccess

What were you expecting to happen?

Successful serverless deployment

Any logs, error output, etc?

Serverless: Packaging service...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service my-app-gateway-serverless-0.0.0.0.0-SNAPSHOT.jar file to S3 (63.32 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
.............................
Serverless: Operation failed!
Serverless: View the full error output: https://ca-central-1.console.aws.amazon.com/cloudformation/home?region=ca-central-1#/stack/detail?stackId=arn%3Aaws%3Acloudformation%3Aca-central-1%3A449576349690%3Astack%2Fmy-app-gateway-serverless-dev-py-2%2Ffdd3ead0-8e6f-11eb-bfd8-0e75e71712c2
 
  Serverless Error ---------------------------------------
 
  An error occurred: CodeDeployServiceRole - Policy arn:aws:iam::aws:policy/AWSLambdaFullAccess does not exist or is not attachable. (Service: AmazonIdentityManagement; Status Code: 404; Error Code: NoSuchEntity; Request ID: ea95d5ce-e472-4a3e-8f17-6627ca55675a; Proxy: null).
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information ---------------------------
     Operating System:          linux
     Node Version:              14.16.0
     Framework Version:         1.83.3 (local)
     Plugin Version:            3.8.4
     SDK Version:               2.3.2
     Components Version:        2.34.9

Any other comments?

Please refer to https://docs.aws.amazon.com/lambda/latest/dg/access-control-identity-based.html

The new role(s) are AWSLambda_FullAccess and AWSLambda_ReadOnlyAccess

What versions of software are you using?

serverless framework 2.31.0 (also tried 1.83.3)
serverless-plugin-canary-deployments 0.5.0 (also tried 0.4.8)

(Thanks for reporting an issue! Please, then fill out the blanks below.)

What are the steps to reproduce this issue?

1. …
2. …
3. …

Operating System: darwin
Node Version: 14.5.0
Framework Version: 2.20.1
Plugin Version: 4.4.2
SDK Version: 2.3.2
Components Version: 3.5.1
What happens?

…Serverless: Configuration warning:
Serverless: at 'functions.e911provHndler': unrecognized property 'alarms'
Serverless: at 'functions.e911provHndler': unrecognized property 'deploymentSettings'
Serverless: at 'provider.iamRoleStatements[0]': unrecognized property 'resource'
Serverless: at 'provider.iamRoleStatements[0]': unrecognized property 'role'

What were you expecting to happen?

…Success

Any logs, error output, etc?

(If it’s long, please paste to https://pastebin.com/ and insert the link here.)

Any other comments?

…

What versions of software are you using?

Serverless 2.20.1

1. I use this YML

service: url

plugins:
  - serverless-offline
  - serverless-pseudo-parameters
  - serverless-plugin-stage-variables
  - serverless-plugin-tracing
  - serverless-plugin-canary-deployments

package:
  # include:
  #   - handlers
  exclude:
    - .gitignore
    - test-data*.json
    - doc
    - scripts
    - aws-sdk
    - .bak
    - .local*
    - config/config.local.json
    # - .serverless  

ApiGatewayStage:
  Type: AWS::ApiGateway::Stage
  Properties:
    RestApiId:
      Ref: ApiGatewayRestApi
    MethodSettings:
    - DataTraceEnabled: true
      HttpMethod: "*"
      LoggingLevel: INFO
      ResourcePath: "/*"
      MetricsEnabled: true

resources:
  Resources:
    urlLogGroup:
      Type: AWS::Logs::LogGroup
      Properties:
        RetentionInDays: "14"

custom: 
  configS3: config-sys-${opt:stage, 'dev'}
  configS3Path: url
  deploymentSettings: //have tried with and without this in custom
    type: Linear10PercentEvery1Minute
    alias: fivehundred
    alarms:
      - name: Canary test alarm
      
provider:
  name: aws
  timeout: 300
  runtime: nodejs8.10
  stage: ${opt:stage, 'dev'}
  region: ap-southeast-2
  tracing: true
  endpointType: regional

  environment:
    region: ${self:provider.region}
    stage: ${opt:stage, 'dev'}
    configBucket: ${self:custom.configS3}
    configPath: ${self:custom.configS3Path}

  deploymentBucket:
    name: name-deploy-${opt:stage, 'dev'}

  iamRoleStatements:
    - Effect: Allow
      Action:
        - codedeploy:*
      Resource:
        - "*"
    - Effect: "Allow" 
      Action:
        - "xray:PutTraceSegments"
        - "xray:PutTelemetryRecords"
      Resource: 
        - "*" 
    - Effect: "Allow"
      Action:
        - sns:publish
      Resource: "arn:aws:sns:${opt:region, self:provider.region}:#{AWS::AccountId}:error-sns-${opt:stage, 'dev'}"
    - Effect: Allow
      Action:
        - "s3:ListBucket"
        - "s3:GetObject"
      Resource: "arn:aws:s3:::${self:custom.configS3}/${self:custom.configS3Path}/*"

functions:
  getUrl:
    handler: handler.getUrl
    timeout: 300
    events:
      - http: 
          path: /url
          method: post
          cors:
            origin: '*'
            headers:
            - Content-Type
            - X-Amz-Date
            - Authorization
            - X-Api-Key
            - X-Amz-Security-Token
            - X-Amz-User-Agent
            - Access-Control-Allow-Origin
            - Access-Control-Allow-Credentials
            allowCredentials: true
          private: true
    deploymentSettings:
      type: Linear10PercentEvery1Minute
      alias: fivehundred
      alarms:
        - name: Canary test alarm
    onError: arn:aws:sns:${opt:region, self:provider.region}:#{AWS::AccountId}:error-sns-${opt:stage, 'dev'}

if I SLS deploy with this in the yml:

deploymentSettings:
  type: Linear10PercentEvery1Minute
  alias: fivehundred

everything works as expected.
But when I specify alarm:

deploymentSettings:
  type: Linear10PercentEvery1Minute
  alias: fivehundred
  alarms:
    - name: Canary test alarm

The deployment happens instantly.

What was I expecting to happen:
I am expecting the lambda to deploy with the traffic shifting gradually over 10 minutes, as it does when i don't specify an alarm.

Any logs, error output, etc?
No errors in console, seems to deploy as normal

Any other comments?
I then deployed the lambda again with all deployment settings commented out and I got an error in console saying that the deployment had been rolled back because the alarm was triggered.

What versions of software are you using?

"serverless-plugin-canary-deployments": "^0.4.8",
"serverless-offline": "^5.8.0",

I have confirmed I am using the right alarm name

some other stuff I noticed:
This only happens the first time you deploy with an alarm specified. If I deploy my lambda with the above yml, it will instantly change to the new version. But if I change the code (not yml) slightly and redeploy it seems to work. however if I make changes to the deployment settings in the yml in between deployments it will deploy instantly both times.

Hi!

I'm trying to test the plugin on serverless.yml containing 2 lambda functions and i can see 2 CodeDeploy deployment groups and 2 deployments created when i run the "serverless deploy" command.

I'm not sure that it will work correctly with my production project which contains 20+ lambda functions. I believe it might hit the resource limit for the CloudFormation template.

Is there any way to start a single CodeDeploy deployment with pre and post hooks, execute some tests from the hooks and rollback all the lambda functions?

HI thanks for this plugin! Its super cool 👍

I'd like to setup notifications from my deploys, and think that being able to optionally pass that through config would be swell.

I think that buildFnDeploymentGroup could be updated to build TriggerConfigurations from serverless.yml like:

deploymentSettings:
  type: Linear10PercentEvery1Minute
  alias: Live
  preTrafficHook: preHook
  postTrafficHook: postHook
  alarms:
    - HelloFooAlarm
  triggerConfigurations:
    - 
      name: rollbacks
      events:
        - DeploymentRollback
      targetArn: !Ref SNSTopic

Does that make sense? Would you be open to a PR for this?

Could I kindly ask for a 0.5.1 release?

I would like to use #113

I'm currently using the master branch of this package in my applications.

It appears that the preTrafficHook does not run when the first version of a stack is deployed. I'm not sure if this a bug or a limitation of CloudFormation.

What are the steps to reproduce this issue?

1. Run sls deploy for a brand new stack/function
2. Observe that the preTrafficHook is not invoked
3. Modify the function and redeploy
4. Observe that the preTrafficHook is invoked this time

What happens?

The preTrafficHook function is not called when stack is first created

What were you expecting to happen?

I expected the preTrafficHook function would be called on every deployment (including the first)

Any logs, error output, etc?

No

Any other comments?

…

What versions of software are you using?

• serverless 1.62.0
• serverless-plugin-canary-deployments 0.4.8

Hi,

Can I just attach CodeDeployServiceRole without creating it?

Because I got this error message:

An error occurred: CodeDeployServiceRole - API: iam:CreateRole User: arn:aws:iam::MY_ACCOUNT_ID:user/MY_USER is not authorized to perform: iam:CreateRole on resource: arn:aws:iam::MY_ACCOUNT_ID:role/canary-deployments-test-CodeDeployServiceRole-.

this is my serverless.yml:

service: canary-deployments-test

provider:
  name: aws
  runtime: nodejs6.10

plugins:
  - serverless-plugin-canary-deployments

functions:
  hello:
    handler: handler.handler
    events:
      - http:
          path: api
          method: get
    deploymentSettings:
      type: Linear10PercentEvery1Minute
      alias: Live

What are the steps to reproduce this issue?

Create a function with an httpApi event and a deploymentConfiguration:

functions:
  http:
    handler: dist/lambda/http.http
    events:
      - httpApi: '*'
    deploymentSettings:
      type: AllAtOnce
      alias: Live
      preTrafficHook: deployPreTraffic

What happens?

The AWS::Lambda::Permission resource for the integration is correctly updated to point to the AWS::Lambda::Alias resource for the function's live alias, but the AWS::ApiGatewayV2::Integration resource still points to the AWS::Lambda::Function resource:

    "HttpApiIntegrationHttp": {
      "Type": "AWS::ApiGatewayV2::Integration",
      "Properties": {
        "ApiId": {
          "Ref": "HttpApi"
        },
        "IntegrationType": "AWS_PROXY",
        "IntegrationUri": {
          "Fn::GetAtt": [
            "HttpLambdaFunction",
            "Arn"
          ]
        },
        "PayloadFormatVersion": "1.0",
        "TimeoutInMillis": 20500
      }
    },

This results in the API Gateway returning 500 errors for that integration with the message "The IAM role configured on the integration or API Gateway doesn't have permissions to call the integration. Check the permissions and try again." because it only has permissions on the live alias, not on the bare function ARN it's using.

What were you expecting to happen?

The AWS::ApiGatewayV2::Integration resource should be overridden to point to the AWS::Lambda::Alias resource for the function's live alias just like the AWS::Lambda::Permission resource is.

    "HttpApiIntegrationHttp": {
      "Type": "AWS::ApiGatewayV2::Integration",
      "Properties": {
        "ApiId": {
          "Ref": "HttpApi"
        },
        "IntegrationType": "AWS_PROXY",
        "IntegrationUri": {
          "Ref": "HttpLambdaFunctionAliasLive"
        },
        "PayloadFormatVersion": "1.0",
        "TimeoutInMillis": 20500
      }
    },

Any logs, error output, etc?

Nope.

Any other comments?

It's possible to work around this issue by overriding the integration URI for all your httpApi events to point to the alias, e.g.

resources:
  extensions:
    HttpApiIntegrationHttp:
      Properties:
        IntegrationUri: {Ref: HttpLambdaFunctionAliasLive}

What versions of software are you using?

serverless: 1.78.1
serverless-plugin-canary-deployments: 0.4.8

(Thanks for reporting an issue! Please, then fill out the blanks below.)

What are the steps to reproduce this issue?

1. npm i
2. cd example
3. sls deploy -s dev

What happens?

Get this error during Stack Update:

Serverless Error ---------------------------------------
 
  An error occurred: SdefaultWebsocketsRoute - Currently, authorization is restricted to the $connect route only (Service: AmazonApiGatewayV2; Status Code: 400; Error Code: BadRequestException; Request ID: XXXX; Proxy: null).

What were you expecting to happen?

Successful deployment as mentioned in the example README.

Any logs, error output, etc?

(If it’s long, please paste to https://pastebin.com/ and insert the link here.)

Any other comments?

…

What versions of software are you using?

From the output of the command:

Operating System: darwin
Node Version: 14.7.0
Framework Version: 1.82.0
Plugin Version: 3.8.3
SDK Version: 2.3.1
Components Version: 2.34.9

Does this lib supports nodejsV8 and AllAtOnce mode for lambda deployment?