Instance Refresh for EC2 Auto Scaling about eks-rolling-update HOT 3 CLOSED

@stevehipwell Thanks for sharing I wasn't aware of that. However, although its useful it doesn't have any integration with Kubernetes. You could do something like what Zalando does here https://github.com/zalando-incubator/kubernetes-on-aws/blob/449f8f3bf5c60e0d319be538460ff91266337abc/cluster/userdata-worker.yaml#L92 so before the node shuts down it drains the pods.

However as the tool says: "Each instance is terminated first and then replaced", which doesn't allow you to prescale up with multiple new instances first which mode 2 & 3 of eks-rolling-update allows. Also , with Instance Refresh, when the warmup period is over, there is no guarantee that the node is added to the Kube cluster which eks-rolling-update checks before proceeding in mode 1.

from eks-rolling-update.

It looks interesting, however unless I am missing something, I can't see an obvious way to use this feature to co-ordinate the really important part of the process that eks-rolling-update solves which is particular to Kubernetes clusters - safely cordoning and draining nodes before they are removed from the cluster/terminated.

So I suppose you would still need to layer on top of this some sort of pre-termination lifecycle hook. All of the approaches I have seen to do that seem kind-of messy to me. I do wonder what AWS constructs EKS uses internally with their managed node groups though...

from eks-rolling-update.

Thanks for the reply @crhuber. I also asked a similar question on the aws-node-termination-handler repo but no luck there either.

I've toyed with the idea of creating a K8s operator to manage the rolling update from within the cluster. When the configuration changes it would create a new node from an up to date launch template; once the new node is available it would run the rolling update command on it as a job.

from eks-rolling-update.