This project contains a Vagrantfile and associated Ansible playbook scripts to provisioning a 3 nodes Kubernetes cluster using VirtualBox and Ubuntu 16.04.

You need the following installed to use this playground.

• Vagrant, version 1.9.3 or better. Earlier versions of vagrant do not work with the Vagrant Ubuntu 16.04 box and network configuration.
• VirtualBox, tested with Version 5.1.18 r114002
• Internet access, this playground pulls Vagrant boxes from the Internet as well as installs Ubuntu application packages from the Internet.

To bring up the cluster, clone this repository to a working directory.

git clone http://github.com/davidkbainbridge/k8s-playground

Change into the working directory and vagrant up

cd k8s-playground
vagrant up

Vagrant will start three machines. Each machine will have a NAT-ed network interface, through which it can access the Internet, and a private-network interface in the subnet 172.42.42.0/24. The private network is used for intra-cluster communication.

The machines created are:

As the cluster brought up the cluster master (k8s1) will perform a kubeadm init and the cluster workers will perform a kubeadmin join. This cluster is using a static Kubernetes cluster token.

After the vagrant up is complete, the following command and output should be visible on the cluster master (k8s1).

vagrant ssh k8s1
kubectl -n kube-system get po -o wide

NAME                             READY     STATUS              RESTARTS   AGE       IP            NODE
etcd-k8s1                        1/1       Running             0          10m       172.42.42.11   k8s1
kube-apiserver-k8s1              1/1       Running             1          10m       172.42.42.11   k8s1
kube-controller-manager-k8s1     1/1       Running             0          11m       172.42.42.11   k8s1
kube-discovery-982812725-pv5ib   1/1       Running             0          11m       172.42.42.11   k8s1
kube-dns-2247936740-cucu9        0/3       ContainerCreating   0          10m       <none>        k8s1
kube-proxy-amd64-kt8d6           1/1       Running             0          10m       172.42.42.11   k8s1
kube-proxy-amd64-o73p7           1/1       Running             0          5m        172.42.42.13   k8s3
kube-proxy-amd64-piie9           1/1       Running             0          8m        172.42.42.12   k8s2
kube-scheduler-k8s1              1/1       Running             0          11m       172.42.42.11   k8s1

Stating the clustering networking is NOT automated and must be completed after the vagrant up is complete. A script to start the networking is installed on the cluster master (k8s1) as /usr/local/bin/start-weave.

vagrant ssh k8s1
ubuntu@k8s1:~$ start-weave
clusterrole "weave-net" created
serviceaccount "weave-net" created
clusterrolebinding "weave-net" created
daemonset "weave-net" created

NAMESPACE     NAME                           READY   STATUS              RESTARTS   AGE
kube-system   coredns-86c58d9df4-bqsh5       0/1     ContainerCreating   0          124m
kube-system   coredns-86c58d9df4-ndd26       0/1     ContainerCreating   0          124m
kube-system   etcd-k8s1                      1/1     Running             0          123m
kube-system   kube-apiserver-k8s1            1/1     Running             0          123m
kube-system   kube-controller-manager-k8s1   1/1     Running             0          123m
kube-system   kube-proxy-9ngj5               1/1     Running             0          124m
kube-system   kube-proxy-j2pws               1/1     Running             0          11m
kube-system   kube-proxy-q6vdl               1/1     Running             0          123m
kube-system   kube-scheduler-k8s1            1/1     Running             0          123m
kube-system   weave-net-sz65d                1/2     CrashLoopBackOff    25         119m
kube-system   weave-net-vww4b                2/2     Running             0          119m
kube-system   weave-net-x4fbm                1/2     CrashLoopBackOff    6          11m

After the network is started, assuming weave-net is used, the following command and output should be visible on the master node (k8s1):

vagrant ssh k8s1
$ kubectl -n kube-system get po -o wide
NAME                             READY     STATUS    RESTARTS   AGE       IP            NODE
etcd-k8s1                        1/1       Running   0          14m       172.42.42.11   k8s1
kube-apiserver-k8s1              1/1       Running   1          13m       172.42.42.11   k8s1
kube-controller-manager-k8s1     1/1       Running   0          14m       172.42.42.11   k8s1
kube-discovery-982812725-pv5ib   1/1       Running   0          14m       172.42.42.11   k8s1
kube-dns-2247936740-cucu9        3/3       Running   0          14m       10.40.0.1     k8s1
kube-proxy-amd64-kt8d6           1/1       Running   0          13m       172.42.42.11   k8s1
kube-proxy-amd64-o73p7           1/1       Running   0          8m        172.42.42.13   k8s3
kube-proxy-amd64-piie9           1/1       Running   0          11m       172.42.42.12   k8s2
kube-scheduler-k8s1              1/1       Running   0          14m       172.42.42.11   k8s1
weave-net-33rjx                  2/2       Running   0          3m        172.42.42.12   k8s2
weave-net-3z7jj                  2/2       Running   0          3m        172.42.42.11   k8s1
weave-net-uvv48                  2/2       Running   0          3m        172.42.42.13   k8s3

Included in the git repository is a sample service and deployment specification that work with Kubernetes. These can be found on the master node (k8s1) as /vagrant/service.yml and /vagrant/deployment.yml.

These descriptors will create a hello-service sample service using a simple docker image davidkbainbridge/docker-hello-world. This image is a simple HTTP service that outputs the the hostname and the IP address information on which the request was processed. An example output is:

Hello, "/"
HOST: hello-deployment-2911225940-qhfn2
ADDRESSES:
    127.0.0.1/8
    10.40.0.5/12
    ::1/128
    fe80::dcc9:4ff:fe5c:f793/64

To start the service and deployment you can issue the following command on the master node (k8s1):

kubectl create -f /vagrant/service.yml -f /vagrant/deployment.yml

After issuing the create command you should be able to see the service and deployment using the following commands.

ubuntu@k8s1:~$ kubectl get service
NAME            CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
hello-service   100.76.247.60   <none>        80/TCP    6s
kubernetes      100.64.0.1      <none>        443/TCP   36m

ubuntu@k8s1:~$ kubectl get deployment
NAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
hello-deployment   3         3         3            0           12s

After the sample service container is pulled from dockerhub and started the available count should go to the value 3.

Kubernetes includes a DNS server through which service IPs and ports can be queried. The IP address of the DNS service can be seen using the following command:

kubectl -n kube-system get svc -l k8s-app=kube-dns -o json | jq -r '.items[].spec.clusterIP'

The IP address of the DNS server will be used in some of the following commands, such as dig in for form @a.b.c.d. Please substitute the IP address returned about into this commands.

Kubernetes creates a SRV record in its DNS server for the service that was defined in the service.yml file. Looking at this file you can see that a service named http was defined. This information can be queried from DNS use the following command:

ubuntu@k8s1:~$ dig +short  @a.b.c.d _http._tcp.hello-service.default.svc.cluster.local SRV
10 100 80 hello-service.default.svc.cluster.local.

The information returned form this query is of the form:

<priority> <weight> <port> <target>

Using a query recursively on this information, specifically the target the IP address for the service can be retrieved.

ubuntu@k8s1:~$ dig +short  @a.b.c.d hello-service.default.svc.cluster.local.
100.76.247.60

Thus using DNS a client can dynamically determine the IP address and the port on which to connect to a service. To test the service the following command can be used on any node in the cluster:

curl -sSL http://$(dig +short @a.b.c.d $(dig +short  @a.b.c.d \
     _http._tcp.hello-service.default.svc.cluster.local SRV | cut '-d ' -f4)):\
     $(dig +short @a.b.c.d _http._tcp.hello-service.default.svc.cluster.local SRV | cut '-d ' -f3)

A script has been provided to simplify this tasks and returns the host IP and the port given a DNS server and service. Thus the same results can be acheived in the playground with the following command:

curl -sSL http://$(/vagrant/kube-target _http._tcp.hello-service.default.svc.cluster.local)

The IP address for the service can also be seen via the kubectl get service command.

The IP address returned in every enivornment may be differet.

To test the service you can use the following command on any node in the cluster:

ubuntu@k8s1:~$ curl -sSL http://$(/vagrant/kube-target _http._tcp.hello-service.default.svc.cluster.local)
Hello, "/"
HOST: hello-deployment-2911225940-b3tyn
ADDRESSES:
    127.0.0.1/8
    10.32.0.2/12
    ::1/128
    fe80::e89f:bfff:fec2:b67a/64

To test the scaling of the service, you can open a second terminal and ssh to a node in the cluster (e.g. vagrant up ssh k8s1). In this terminal if you issue the following command it will periodically issue a curl request to the service and display the output, highlighting the difference from the previous request. This demonstates that the request is being handled by different services.

watch -d curl -sSL http://$(/vagrant/kube-target  _http._tcp.hello-service.default.svc.cluster.local)

Currently there should be 3 instances of the service implementation being used. To scale to a single instance, issue the following command:

ubuntu@k8s1:~$ kubectl scale deployment hello-deployment --replicas=1
deployment "hello-deployment" scaled

After scaling to a single instance the watch command from above should show no differences between successive request as all requests are being handled by the same instance.

The following command scales the number of instances to 5 and after issuing this command differences in the watch command should be highlighted.

ubuntu@k8s1:~$ kubectl scale deployment hello-deployment --replicas=5
deployment "hello-deployment" scaled

The test container image used above davidkbainbridge/docker-hello-world:latest is built with a health check capability. The container provides a REST end point that will return 200 Ok by default, but this can be manual set to a different value to test error cases. See the container documentation at https://github.com/davidkbainbridge/docker-hello-world for more information.

To see the health of any given instance of the service implementation, you can ssh to the k8s1 and perform a kubectl get po -o wide. This will show the pods augmented with the number of restarts.

ubuntu@k8s1:~$ kubectl get po -o wide
NAME                                READY     STATUS    RESTARTS   AGE       IP          NODE
hello-deployment-3696513547-fhh2y   1/1       Running   0          12s       10.40.0.1   k8s2
hello-deployment-3696513547-ocgas   1/1       Running   0          12s       10.38.0.2   k8s3
hello-deployment-3696513547-y257u   1/1       Running   0          12s       10.38.0.1   k8s3

To demonstrate the health check capability of the cluster, you can open up a ssh session to k8s1 and run watch -d kubectl get po -o wide. This command will periodically update the screen with information about the pods including the number of restarts.

To cause one of the container instances to start reporting a failed health value you can set a random instance to fail using

curl -XPOST -sSL http://$(/vagrant/kube-target  _http._tcp.hello-service.default.svc.cluster.local)/health -d '{"status":501}'

This will set the health check on a random instance in the cluster to return "501 Internal Server Error". If you want to fail the health check on a specific instance you will nee to make a similar curl request to the specific container instance.

After setting the health check to return a failure value monitor the kubectl get po -o wide command. After about 30 seconds one of the pod restarts counts should be incremented. This represented Kubernetes killing and restarting a pod because of a failed health check.

NOTE: the frequency of health checks is configurable

Included in the repo is an example client written in the Go language that demonstrates how a container can directly look up another service and call that service.

To test this client use the following steps:

The output of the make logs step should display a similar output as the curl statement above.

ubuntu@k8s1:/vagrant/examples/client/go$ make logs
kubectl logs hello-client
Hello, "/"
HOST: hello-deployment-1725651635-u8f65
ADDRESSES:
    127.0.0.1/8
    10.40.0.1/12
    ::1/128
    fe80::bcdc:21ff:fecd:83db/64

The above example forces the deployment of the sample client to the k8s1 node for simplicity. If you would like to have the client deployed to any of the nodes in the cluster you need to ssh to each node in the cluster and

cd /vagrant/examples/client/go
sudo apt-get install -y make
make image

The run the sample client use the following commands

vagrant ssh k8s1
kubectl delete po hello-client
kubectl run hello-client --image=hello-client --image-pull-policy=Never --restart=Never

On each vagrant machine is installed a utility as /usr/local/bin/clean-k8s. executing this script as sudo will reset the servers back to a point where you can execute vagrant provisioning.