CodeDeploy support about autospotting HOT 19 CLOSED

@binarylogic What I did was set the deployment group to use tag groups.

Then added these two scripts to the ami:

• This is pretty self explanatory - it gathers the metadata about the instance and stores them as variables.
  https://gist.github.com/sc-chad/99ba78a7cb1e7b5573ea131cf2015cad

• Then we use this guy to use those variables and check it's deployment group if it has the latest code
  https://gist.github.com/sc-chad/ae0f4acbb5b7283a2dc0b25a3277cf50

We remove the .sh and store in /usr/bin. Once those are added and customized to what you need just add something like:

# Inside /etc/rc.local
/usr/bin/check-codedeploy

Or wherever you want. Can also be added a user-run script but I don't like putting things in there so did it this way.

from autospotting.

I am not using Code Deploy, so I'm not familiar with the hooks, but please try it out and let me know if it works out of the box, or otherwise I'll try to come up with a solution for supporting this.

Do you know how are these CodeDeploy hooks interacting with new instances attached manually to the group after they were running for a while outside the group? That's essentially what autospotting is doing under the hood.

from autospotting.

Hmm I will have to check that our, that will take a few days...

from autospotting.

This is the lifecycle hook of the CodeDeploy integration:

{
    "LifecycleHooks": [
        {
            "GlobalTimeout": 60000, 
            "HeartbeatTimeout": 600, 
            "AutoScalingGroupName": "DEMO-FRONTEND-XXXXXXXXXXXXX", 
            "LifecycleHookName": "CodeDeploy-managed-automatic-launch-deployment-hook-ZZZZZZZZZZZZZZZ", 
            "NotificationMetadata": "692c1da4-940d-492a-ac3f-e81e0620b0f1", 
            "DefaultResult": "ABANDON", 
            "NotificationTargetARN": "arn:aws:sqs:eu-central-1:1111111111111111111:razorbill-eu-central-1-prod-default-autoscaling-lifecycle-hook", 
            "LifecycleTransition": "autoscaling:EC2_INSTANCE_LAUNCHING"
        }
    ]
}

from autospotting.

I read a bit through the CodeDeploy docs and I think it should work, but please test it first on a non-production environment.

• if the CodeDeploy deployment group is based on EC2 tags, the tags are set by AutoSpotting at instance launch, so the agent should be able to perform the deployment as soon as they booted up, even way before they are added to the AutoScaling group(which in turns adds them to the load balancer), so all in all it should work out of the box.

But you may get into trouble in the very unlikely case that...

• you are in the process of replacing all the nodes at once with spot instances(likely immediately after enabling it on your group)...
• and you are deploying a new version at exactly the same time...
• and you use a deployment policy that deploys half the nodes at once.
• and you have a small number of nodes that increase the probability to randomly select all your live nodes together in one of those two deployment halves

There is a chance that all the live nodes would get deployed at the same time and their service is restarted and may cause some downtime. To reduce this risk, I would recommend to use a rolling procedure which updates one node at a time, to block deployments when you first enable AutoSpotting on your existing AutoScaling group, or to increase the group's capacity to reduce the chance to deploy all of the live nodes at once.

• if the CodeDeploy deployment group is based on AutoScaling group membership, the new machines would not be set up before adding them to the group, but they should get configured immediately after being added (later edit: this is assuming the instance attachment lifecycle event is properly handled by CodeDeploy, which seems not to be the case at the moment).

This should also work, except for the not quite so unlikely case in which...

• you use ELB health checks for your AutoScaling group...
• and the deployment is slower than the time it takes for your ELB health checks to get your new instances terminated.

In this case I recommend to do something to speed up your deployments or make your ELB checks take longer to kill the instances than the time you need for the deployment. Another approach may be to switch your AutoScaling group to EC2 health checks.

I would recommend to test both scenarios if you can, but I think that a deployment group based on EC2 tags would be a better idea because it would be more unlikely to get into that 'perfect storm' failure scenario.

Let me know what you think, or if I am missing anything.

from autospotting.

Did a little testing with this today and thought I'd share the results. In my setup our CodeDeploy groups are based on AutoScale groups.

Once the new spot instance was added to the AutoScale group nothing happened. CodeDeploy hooks into the lifecycle process of the AutoScale group, so I expect that Autospotting is missing out a critical piece of the process. If our AMI was pre-baked, this wouldn't be an issue, but then we wouldn't use CodeDeploy either.

from autospotting.

I'm not that familiar with CodeDeploy so correct me if I'm wrong, but according to the introduction's page: http://docs.aws.amazon.com/codedeploy/latest/userguide/welcome.html

It looks like to me, when CodeDeploy manages the instance to create, it also injects meta-data regarding the deployment (version, repo, etc), these being used by the agent to deploy the code.

What you could do is check the presence of the agent and metada on those 2 types of instances (spawned from CodeDeploy and the Spot instances).
I suspect CodeDeploy to spawn instances with the CodeDeploy agent already present and with the correct metada, if that's the case then it's logical that autospotting doesn't work: it might create an instance with the CodeDeploy agent (like CodeDeploy does if base on AMI), but misses the metadata to deploy the code.

I think it's worth checking those elements and provide further feedback.

from autospotting.

As per this document https://aws.amazon.com/blogs/devops/under-the-hood-aws-codedeploy-and-auto-scaling-integration/

CodeDeploy listens only for notifications about instances that have launched and are about to be put in the InService state.

At the time those notifications are sent for your new spot instances they are not yet added to the group so CodeDeploy simply ignores them when configured based on group membership.

@nerk01 try configuring CodeDeploy based on an instance tag set on your group, since it turns out the AutoScaling group membership method is broken due to the way CodeDeploy handles the lifecycle hooks.

If that works for you I would like to have this documented as a known issue.

from autospotting.

CodeDeploy doesn't handle newly launched instances, even with tags. So, having a deployment group with both autoscaling membership and instance tags doesn't solve the issue.

Here is a workaround: Whenever an instance is launched, trigger a deployment (via CloudTrail+SNS) in the same group with option --update-outdated-instances-only.
--update-outdated-instances-only ensures that only new instances get deployed. Without this option, every instance launch would trigger a deployment on all instances.

from autospotting.

Thanks @acsrujan, please create a PR with a documentation update in which to mention this workaround

from autospotting.

@acsrujan can you give a bit more detail on how you accomplished doing this? I understand the context but not sure I understand the implementation.

We are having this exact same issue and would love to have a solution.

from autospotting.

I have no experience with CodeDeploy so I am just shooting in the dark here, but at a first glance I think the simplest way to do it is using the userdata script, all instances may execute a deployment using the aws command line tool and the flag mentioned above by @acsrujan.

from autospotting.

I was able to get this working via a simple bash script that gets my tags of the instance then uses them to run:

aws deploy create-deployment --application-name $PRODUCT --deployment-group $DEPLOYMENTGROUP --update-outdated-instances-only

That will return a deployment-id that checks to see if any instances within that deployment group haven't had any code deployed to them.

This could probably be converted to a simple lambda function that registers the new instances, checks for tags like: deployment_group and application_name and registers a new deployment to it.

I didn't have the time to try and work that out, but I appreciate @acsrujan pointing me in the right direction!

from autospotting.

We're experiencing the same problem and the above command worked as expected. Any ideas around how to automate executing that command when a spot instance is launched?

from autospotting.

@binarylogic, @sc-chad this is really useful information, so let's document it properly.

Please any of you create a PR for this.

from autospotting.

@cristim trade ya - I'll write up a PR to fully document what I've done if you take a look at my Name:tag issue! :)

I'll totally write it up anyway, but figured I'd try!

from autospotting.

@sc-chad I gave you some suggestions for the Name tag, this should be relatively easy to implement, although it may require a bit of refactoring.

Please create a PR and I will do my best to help you out with the implementation.

from autospotting.

I might have some time as well to take a look at issues

from autospotting.

The current version of AutoSpotting trunk implements CodeDeploy support against ASG deployment groups, so the previous workaround using tag-based deployment groups should no longer be necessary.

Please try it out and report back if you notice any issues.

from autospotting.