we'll need to make a bootkube.service systemd unit in order to survive reboots.

Bringing the discussion from #3 to a proper issue.

The error with seen with kube-router is:

ubuntu@osh-sh-ci-01:~$ kubectl logs  kube-router-hkf2d -n kube-system
panic: nodes "kubernetes" not found

goroutine 1 [running]:
panic(0x1596120, 0xc42040a450)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/cloudnativelabs/kube-router/app/controllers.NewNetworkPolicyController(0xc420315540, 0xc420314960, 0x0, 0x0, 0x0)
	/home/kube/go/src/github.com/cloudnativelabs/kube-router/app/controllers/network_policy_controller.go:785 +0x413
github.com/cloudnativelabs/kube-router/app.(*KubeRouter).Run(0xc4203c1660, 0xc4203c1660, 0x0)
	/home/kube/go/src/github.com/cloudnativelabs/kube-router/app/server.go:120 +0x710
main.main()
	/home/kube/go/src/github.com/cloudnativelabs/kube-router/kube-router.go:37 +0x13c
ubuntu@osh-sh-ci-01:~$ 

In the openstack-helm-up.sh, there is an export of Sigil but the var is not used.

Expected: when executing curl use the var to version the pulled version of sigil
Actual: the sigil var is not used.

@v1k0d3n , Just wanted to mention these...

1. Not sure which combination of packages. But i've had issues with sudo after installing ZFS then running bootkube (i think it's the docker package that is causing it).... Sudo fails, long story short it's an issue with /dev/null permissions, and some package adding a configuration line to /etc/sudoers to log to /dev/null... manually edited to remove, and it works again.

2. Another issue I was having was with having to manually add kubernetes.default to my /etc/hosts file. After doing that twice, i realized i could have just edited .bootkube.env to be my server's hostname. (User error)
   **This issue was causing my containers to come up without IP addresses... and the api-server to crash constantly.

3. DNSmasq was also causing random crashing issues for the containers, so I manually disabled DNSmasq and fixed /etc/resolv.conf.

4. In addition, I don't think bootkube-clean.sh is putting it back the way it was before bootkube-up.sh...

Anyway, hope these help someone else.

I now have a clean metal install, with a ZFS storage array, just need to figure out how to point etcd to use it.

we should really start converting this bash code into real code (golang preferred), using our other [currently] private project.

for a while it was perfectly fine to have users download bootkube-ci to whatever directory they want, but i think going forward it would make sense to force some directories in order to provide better consistency/reliability for the project.

/opt/bootkube should be used, and docs will need to be updated to reflect this. /etc/bootkube can be used for some of the deployment yaml's (coming soon), which are simply exported variable files at this point. once we move to systemd units, we can create a systemd EnvironmentFile.

found out that the api version for docker 17.06+ will not exactly work with the openstack-helm deployments, and thus i need to fall back to installing via the standard ubuntu packages (docker.io).

i need to fix this tonight.

several updates to the documentation should be made.

Hi! I also like to bundle ceph-common into hyperkube, and I had some issues with PVCs getting stuck trying to be mapped to my nodes. I updated ceph-common to the official Ceph Kraken packages and those issues seemed to go away, although I cannot confirm the root cause of the issue.

Anyways, if you're interested in using ceph tools for a recent Ceph release in hyperkube, here's how I did it in the Dockerfile:

before:

# [...]
RUN DEBIAN_FRONTEND=noninteractive apt-get update -y \
    && DEBIAN_FRONTEND=noninteractive apt-get -yy -q install \
    iptables \
    # [...]

after:

# [...]
RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 08b73419ac32b4e966c1a330e84ac2c0460f3994 \
    && echo "deb http://download.ceph.com/debian-kraken/ jessie main" > /etc/apt/sources.list.d/ceph-kraken.list \
    && DEBIAN_FRONTEND=noninteractive apt-get update -y \
    && DEBIAN_FRONTEND=noninteractive apt-get -yy -q install \
    iptables \
    # [...]

Ran through the setup, but it failed very early on:

./bootkube-ci/bootkube-up.sh: line 199: grep: command not found
./bootkube-ci/bootkube-up.sh: line 199: grep: command not found
^C

Looks like kubectl not found by the script:

rwellum@bootkube-ci:~$ kubectl
kubectl: command not found

What's also odd is this left my ubuntu VM in a very odd state - for example can't find basic commands like 'which' and 'grep' - but they are installed of course. So I think my VM is smoked.