bcwaldon / klondike Goto Github PK

It's annoying to have to pull the IP of the bastion manually. We should manage a DNS record for it. We should probably prefix the current A record we manage for the controller ELB with "api." to make some room for this new record.

Occasionally, when trying to update farva manually via the bastion instance I see:

TASK [kubelet : upload cluster CA certificate] *********************************
fatal: [***]: FAILED! => {"failed": true, "msg": "ERROR! failed to resolve remote temporary directory from ansible-tmp-1463680630.15-264393012566816: `( umask 22 && mkdir -p \"$( echo $HOME/.ansible/tmp/ansible-tmp-1463680630.15-264393012566816 )\" && echo \"$( echo $HOME/.ansible/tmp/ansible-tmp-1463680630.15-264393012566816 )\" )` returned empty string"}

Doing these sort of out-of-band updates is generally not the approach you should be using when interacting with the klondike core components, but I'm concerned this might happen during a normal deployment.

Repeating ansible-playbook -e cluster=bulbasaur -l tag_group_gateway site.yml tends to work.

The process of sizing the cluster for load is a complicated and dangerous process:

• Add instructions for tuning elasticsearch and it's relevant logs.
• Add instructions on resizing the workers. Or adding new worker pools.
• Add instructions to aid in volume management.

The project is generally useful, but several important pieces remain:

• manage secrets securely
• disable CoreOS auto-updates using cloud-config
• spin TLS certificates per worker
• enable AWS Kubernetes cloudprovider
• deploy datadog agent via DaemonSet
• reserve host memory for kubelets
• expand README
• mount /dev/xvdb at /var/lib/docker
• create Route53 record for ELB

What are cluster.yml configure.yml and site.yml?

klondike should ensure workers are spread across availability zones. We may have to use multiple worker subnets to achieve this.

Rather than route traffic through the public-facing ELB, the workers should be able to communicate with the controllers without leaving the VPC.

@bcwaldon What do we think about adding prometheus and statsd support out of the box?

klondike currently depends on installing python directly to the CoreOS filesystem to run ansible. This feels janky. Figure out how to do this in such a way that we run python in a container.

The deployer uses an AWS access/secret key pair to create the CloudFormation stack. These credentials need to be able to create quite a few resources. Document the minimum IAM policy needed for these credentials to properly deploy a klondike cluster.

There's a script that will pull down the authorized_keys.d/ folder from the S3 config bucket. This rebuilds /home/ubuntu/.ssh/authorized_keys on demand. This should probably be run automatically (with configuration).

How does one upgrade all of the components of klondike. What does adding new nodes and reaping all nodes in each node type.

Each node currently runs logstash. This is wasteful. We might as well just run logstash on the controllers, and have logstash-forwarder (or something like it) ship logs to this reduced set of logstash instances.