Reliable, extensible rolling-upgrades of Autoscaling groups in Kubernetes

RollingUpgrades provides a Kubernetes native mechanism for doing rolling-updates of instances in an AutoScaling group using a CRD and a controller.

• RollingUpgrade is highly inspired by the way kops does rolling-updates.

• It provides similar options for the rolling-updates as kops and more.

• The RollingUpgrade Kubernetes custom resource has the following options in the spec:

  • asgName: Name of the autoscaling group to perform the rolling-update.
  • region: Name of the AWS region in which the ASG exists.
  • preDrain.script: The script to run before draining a node.
  • postDrain.script: The script to run after draining a node. This allows for performing actions such as quiescing network traffic, adding labels, etc.
  • postDrain.waitSeconds: The seconds to wait after a node is drained.
  • postDrain.postWaitScript: The script to run after the node is drained and the waitSeconds have passed. This can be used for ensuring that the drained pods actually were able to start elsewhere.
  • nodeIntervalSeconds: The amount of time in seconds to wait after each node in the ASG is terminated.
  • postTerminate.script: Optional bash script to execute after the node has terminated.
  • strategy.type: This field is optional and currently only random update strategy is supported. Refer to random_update_strategy.yaml for sample custom resource definition.
  • strategy.maxUnavailable: Optional field. The number of nodes that can be unavailable during rolling upgrade, can be specified as number of nodes or the percent of total number of nodes. Default is "1".
  • strategy.drainTimeout: Optional field. Node will be terminated after drain timeout even if kubectl drain has not been completed and value has to be specified in seconds. Default is -1.

• After performing the rolling-update of the nodes in the ASG, RollingUpgrade puts the following data in the "Status" field.

  • currentStatus: Whether the rolling-update completed or errored out.
  • startTime: The RFC3339 timestamp when the rolling-update began. E.g. 2019-01-15T23:51:10Z
  • endTime: The RFC3339 timestamp when the rolling-update completed. E.g. 2019-01-15T00:35:10Z
  • nodesProcessed: The number of ec2 instances that were processed.

For each RollingUprade custom resource that is submitted, the following flowchart shows the sequence of actions taken to perform the rolling-update

• Kubernetes cluster on AWS with nodes in AutoscalingGroups. rolling-upgrades have been tested with Kubernetes clusters v1.12+.
• An IAM role with at least the policy specified below. The upgrade-manager should be run with that IAM role.

For a complete, step by step guide for creating a cluster with kops, editing it and then running rolling-upgrades, please see this

If you already have an existing cluster created using kops, follow the instructions below.

• Ensure that you have a Kubernetes cluster on AWS.

• Install the CRD using: kubectl apply -f https://raw.githubusercontent.com/keikoproj/upgrade-manager/master/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml

• Install the controller using: kubectl create -f https://raw.githubusercontent.com/keikoproj/upgrade-manager/master/deploy/rolling-upgrade-controller-deploy.yaml

• Note that the rolling-upgrade controller requires an IAM role with the following policy

{
    "Effect": "Allow",
    "Action": [
        "ec2:TerminateInstances",
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:TerminateInstanceInAutoScalingGroup"
    ],
    "Resource": [
        "*"
    ]
}

• If the rolling-upgrade controller is directly using the IAM role of the node it runs on, the above policy will have to be added to the IAM role of the node.
• If the rolling-upgrade controller is using it's own role created using KIAM, that role should have the above policy in it.