Add instructions on how to setup a k8s cluster using kops. Base blog: https://aws.amazon.com/blogs/compute/kubernetes-clusters-aws-kops/

Here are different use cases:

• Gossip-based cluster for test/dev (more details at http://blog.arungupta.me/gossip-kubernetes-aws-kops/)
• Single region, multi-AZ master and multi-AZ workers
• With public VPC and private VPC
• --networking with calico and flannel
• --target with direct (default) and terraform

Followed the instructions at https://github.com/arun-gupta/kubernetes-aws-workshop/tree/master/cluster-install#multi-master-multi-node-multi-az-dns-based-cluster-with-private-vpc-hosted-zone

Validating the cluster gives the following error:

$ kops validate cluster --name=cluster.k8s-aws.internal
Validating cluster cluster.k8s-aws.internal


cannot get nodes for "cluster.k8s-aws.internal": Get https://api.cluster.k8s-aws.internal/api/v1/nodes: dial tcp: lookup api.cluster.k8s-aws.internal: no such host

Is eu-central-1 the "blessed region" for this workshop at re:Invent? If so, I'll go through and change all the code snippets to reflect that in a PR; just want to make sure that the latency from LAS is going to be considered acceptable.

Use CodePipeline to create a CI/CD pipeline. Here is a tentative flow:

• git push
• Update Docker image
• Trigger Lambda to do rolling deployment of application

Let's list Kubernetes design patterns here:

• Create a cluster with odd number of masters
• Homogeneous cluster?
• Use Docker images with a specific version instead of latest tag
• For HA, spread masters and workers across multiple AZ in a region
• Never use Pod directly, wrap it in a Deployment (this will create ReplicaSet)
• Instead of federation, create multiple clusters and use CI/CD pipeline to deploy across those multiple clusters

Anti-patterns

• Federated cluster

• Using in-built Graphana + Prometheus, explained at http://blog.arungupta.me/kubernetes-monitoring-heapster-influxdb-grafana/
• EFK (Elasticsearch + Fluentd + Kibana)
• Fluentd -> CloudWatch logs and ElasticSearch
• Statsd
• Which partners should be highlighted?

Scaling a cluster involves at least the following use cases:

• Add new master and worker nodes pro-actively
• Add new worker nodes based upon application alerts
• Add new workers nodes if a certain application can be deployed because of lack of resources

cluster-install already talks about exporting KOPS_STATE_STORE as environment variable. There should be no need to include --state s3://kubernetes-aws-io as part of kops CLI.

Basic developer concepts:

• Run-once
• Daemonset

Use nginx base image?

Configuring a custom ingress controller to use nginx as an alternative load balancing to ALB’s/ELB’s – huge cost savings, vital for small startups

• Download and run minikube: https://github.com/kubernetes/minikube
• Get familiar with kubectl
• Run a simple pod

Possibly use an existing application such as https://github.com/arun-gupta/docker-javaee/

"What is this going to cost me" is probably a question that should be addressed up front. Not everyone's going to have an account in the free tier, and people are scared of AWS billing surprises!

Show rolling update of an application, basic sample at: https://github.com/arun-gupta/kubernetes-java-sample/tree/master/rolling-update

The chosen region for different workshop may be different. Instead of hardcoding the AZ in the command, create a $AVAILABILITY_ZONES and that will be used for creating cluster in cluster-install.

FYI @Cloudymind

Demonstrate using Calico with Kubernetes to introduce security group like functionality.

How would you re-purpose this workshop?

• Account Limits
• Regions to use
• Account Permissions
• Anything else?

Grafana dashboard with Prometheus shows the following screen:

CC: @omarlari

Explain how AWS + CNI works

Suggested by @eswarbala

Add a separate section for ops showing how secrets can be securely stored e.g. in Hashicorp Vault and provided as K8s secrets to a dev

Demonstrate auto-scaling of pods via Horizontal Pod Scaling.

Show how service mesh integrate with k8s on AWS. The couple of possibilities:

• Linkerd
• Istio/Envoy

We should add a volumes section where we describe how to access volumes such as ebs, efs etc

@Cloudymind FYI

Show how cluster can be upgraded from 1.6.x to 1.7.x, The exact version may different based upon when this issue is addressed.

How to use instance groups with kops to add other instance types to your k8s cluster. https://github.com/kubernetes/kops/blob/master/docs/instance_groups.md

Integration with LDAP or other directory services

kubectl expose on the nginx pod fails due to the nginx pod not exposing a port (in this command kubectl expose pod pod-xxx --type=NodePort --name=demo). Running nginx as follows fixes this: kubectl run nginx --image=nginx --port=80

Want to confirm whether this is experienced by others, or something in my setup is causing this.

c4b301d07c5d:kubernetes-aws-workshop edgema$ kubectl run nginx --image=nginx
deployment "nginx" created
c4b301d07c5d:kubernetes-aws-workshop edgema$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP   59m
c4b301d07c5d:kubernetes-aws-workshop edgema$ kubectl get po
NAME                     READY     STATUS    RESTARTS   AGE
nginx-4217019353-d7rcm   1/1       Running   0          20s
c4b301d07c5d:kubernetes-aws-workshop edgema$ kubectl expose  po nginx-4217019353-d7rcm --type=NodePort --name=demo
error: couldn't find port via --port flag or introspection
See 'kubectl expose -h' for help and examples.
c4b301d07c5d:kubernetes-aws-workshop edgema$ kubectl describe po nginx-4217019353-d7rcm
Name:           nginx-4217019353-d7rcm
Namespace:      default
Node:           minikube/192.168.99.100
Start Time:     Tue, 17 Oct 2017 19:07:47 +0800
Labels:         pod-template-hash=4217019353
                run=nginx
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"nginx-4217019353","uid":"6889ab83-b32b-11e7-a837-0800278db302","...
Status:         Running
IP:             172.17.0.4
Created By:     ReplicaSet/nginx-4217019353
Controlled By:  ReplicaSet/nginx-4217019353
Containers:
  nginx:
    Container ID:   docker://c0e046e37fbb1d31261ac58f54e63520a14550519935a472f284bce4905a087a
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:004ac1d5e791e705f12a17c80d7bb1e8f7f01aa7dca7deee6e65a03465392072
    Port:           <none>
    State:          Running
      Started:      Tue, 17 Oct 2017 19:07:53 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7kvjs (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  default-token-7kvjs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-7kvjs
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     <none>
Events:
  Type    Reason                 Age   From               Message
  ----    ------                 ----  ----               -------
  Normal  Scheduled              1m    default-scheduler  Successfully assigned nginx-4217019353-d7rcm to minikube
  Normal  SuccessfulMountVolume  1m    kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-7kvjs"
  Normal  Pulling                1m    kubelet, minikube  pulling image "nginx"
  Normal  Pulled                 1m    kubelet, minikube  Successfully pulled image "nginx"
  Normal  Created                1m    kubelet, minikube  Created container
  Normal  Started                1m    kubelet, minikube  Started container

Following the instructions in deployment-concepts/daemonsets.adoc - logstash fails with OOM. If I increase the memory, it fails for an unknown reason. I haven't had chance to debug it.

kubectl get pods -o wide

NAME                       READY     STATUS             RESTARTS   AGE       IP            NODE
logstash-daemonset-r2775   0/1       CrashLoopBackOff   2          39s       100.96.1.16   ip-172-20-67-137.ec2.internal
logstash-daemonset-xsclg   0/1       CrashLoopBackOff   2          39s       100.96.2.17   ip-172-20-42-76.ec2.internal

kubectl describe po
Name:           logstash-daemonset-xsclg
Namespace:      default
Node:           ip-172-20-42-76.ec2.internal/172.20.42.76
Start Time:     Wed, 18 Oct 2017 21:06:53 +0800
Labels:         app=logstash
                controller-revision-hash=3313225235
                pod-template-generation=1
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"DaemonSet","namespace":"default","name":"logstash-daemonset","uid":"360cd102-b405-11e7-a395-12ff179638ce",...
Status:         Running
IP:             100.96.2.17
Created By:     DaemonSet/logstash-daemonset
Controlled By:  DaemonSet/logstash-daemonset
Containers:
  logstash:
    Container ID:   docker://d4f6ec1b2d560f053605f85df1b2671d10eb9d81772fb030f3bc50b722a3c959
    Image:          logstash:5.5.2
    Image ID:       docker-pullable://logstash@sha256:6d5236d5a2371af15d19300f80be7e742e4fa15a19335c6a1372e685e803bc70
    Port:           <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Wed, 18 Oct 2017 21:07:43 +0800
      Finished:     Wed, 18 Oct 2017 21:07:44 +0800
    Ready:          False
    Restart Count:  3
    Limits:
      memory:  50Mi
    Requests:
      cpu:        50m
      memory:     50Mi
    Environment:  <none>
    Mounts:
      /var/lib/docker/containers from varlibdockercontainers (ro)
      /var/log from varlog (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-bsp6x (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  varlog:
    Type:  HostPath (bare host directory volume)
    Path:  /var/log
  varlibdockercontainers:
    Type:  HostPath (bare host directory volume)
    Path:  /var/lib/docker/containers
  default-token-bsp6x:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-bsp6x
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.alpha.kubernetes.io/notReady:NoExecute
                 node.alpha.kubernetes.io/unreachable:NoExecute
Events:
  Type     Reason                 Age               From                                   Message
  ----     ------                 ----              ----                                   -------
  Normal   SuccessfulMountVolume  1m                kubelet, ip-172-20-42-76.ec2.internal  MountVolume.SetUp succeeded for volume "varlog"
  Normal   SuccessfulMountVolume  1m                kubelet, ip-172-20-42-76.ec2.internal  MountVolume.SetUp succeeded for volume "varlibdockercontainers"
  Normal   SuccessfulMountVolume  1m                kubelet, ip-172-20-42-76.ec2.internal  MountVolume.SetUp succeeded for volume "default-token-bsp6x"
  Normal   Pulled                 24s (x4 over 1m)  kubelet, ip-172-20-42-76.ec2.internal  Container image "logstash:5.5.2" already present on machine
  Normal   Created                24s (x4 over 1m)  kubelet, ip-172-20-42-76.ec2.internal  Created container
  Normal   Started                24s (x4 over 1m)  kubelet, ip-172-20-42-76.ec2.internal  Started container
  Warning  BackOff                9s (x6 over 1m)   kubelet, ip-172-20-42-76.ec2.internal  Back-off restarting failed container
  Warning  FailedSync             9s (x6 over 1m)   kubelet, ip-172-20-42-76.ec2.internal  Error syncing pod

Show how a database can be deployed using StatefulSet with persistence on EBS.

A basic example at https://github.com/arun-gupta/couchbase-kubernetes/tree/master/cluster-statefulset.

https://github.com/coreos/alb-ingress-controller

@omarlari

Show how microservices talk to each other using service discovery

Use https://github.com/arun-gupta/container-service-discovery/blob/master/k8s.adoc as a guidance.

I receive a location constraint error after running the following command:
aws s3api create-bucket --bucket $CONFIG_BUCKET_NAME --region $AWS_DEFAULT_REGION

$CONFIG_BUCKET_NAME=[allowed bucket name]
$AWS_DEFAULT_REGION=eu-central-1

The specific error is:
An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.

Show how secrets can be managed on Kubernetes: https://kubernetes.io/docs/concepts/configuration/secret/

• kubectl create secret
• Using configuration file
• Using in an application
• Using ConfigMap

Other use cases explained at https://kubernetes.io/docs/concepts/configuration/secret/.

assign to @Cloudymind

After #5, show how a federated cluster across multiple regions can be created.

Steps are evolving at: https://github.com/arun-gupta/kubernetes-federated-cluster

We should early on explain the relationship between labels, selectors and annotations as these are important concepts in understanding how kubernetes constructs can relate and reference to each other.

Demonstrate handling security by assigning IAM roles at the pod level (instead of node) via kube2iam.

This needs to be done in cluster-install.

Governance of the internals of Kubernetes (e.g. verify that certain entities, annotations and/or labels exist, privileged containers not deployed, only sanctioned image repos used, enforce the configuration of kube2iam, etc) using k8guard.

I think its a good idea to show how internal components can be exchanged such as dns (kube-dns vs coredns)
I did this very recently and could show how both versions exist next to each other and can be flipped.

The following items are recommended for the microservices section of the documentation:

1. Provide a clearer description of how microservices benefit from service discovery in the introduction? ex: This chapter shows an example of how different microservices within an application can use service discovery to locate each other in the infrastructure rather than via hardcoded IP's.

2. Verify that we really need a cluster this large (3 masters and 5 worker nodes) for the exercise. Each replicaset consists of one replica. Perhaps, we can include a generic "Verify that you have a working cluster from a minikube or cluster-install sections of the lab."

3. Explicitly call out that the webapp service uses environment variables that refer to the service labels of the name and greeter services rather than to pod or host IP addresses.

Basic developer concepts:

• ReplicaSet
• Service
• Deployment

Use nginx base image?

Add instructions on how use an already issued certificate for the cluster instead of using the default one created by API Server.

We should have a section describing how application upgrade such as rolling-update, blue/green are done

Explain how an application consisting of multiple k8s resources can be packaged as Helm Chart.

Helm: https://github.com/kubernetes/helm
Charts: https://github.com/kubernetes/charts

• Show how to setup a Helm chart repository
• Use S3 bucket for storing charts
• Host a index.html page that shows the list of charts
• Write a Lambda function to update the page whenever a new chart is uploaded