Trying examples/footloose with master results in a cluster where the wks controller doesn't start:

  Normal   Pulling           13s (x2 over 29s)  kubelet, node0     Pulling image "quay.io/wksctl/controller:latest"
  Warning  Failed            12s (x2 over 28s)  kubelet, node0     Failed to pull image "quay.io/wksctl/controller:latest": rpc error: code = Unknown desc = Error response from daemon: manifest for quay.io/wksctl/controller:latest not found
  Warning  Failed            12s (x2 over 28s)  kubelet, node0     Error: ErrImagePull
  Normal   BackOff           1s (x2 over 27s)   kubelet, node0     Back-off pulling image "quay.io/wksctl/controller:latest"
  Warning  Failed            1s (x2 over 27s)   kubelet, node0     Error: ImagePullBackOff

After an install of Firekube, I have these error message in the controller "from time to time":

E0919 13:59:58.919302       1 controller.go:160] Error checking existence of machine instance for machine object worker-0; failed to establish connection to machine worker-0: failed to read SSH key: failed to get WKS' secret: secrets "wks-controller-secrets" not found

But

$ kubectl -n weavek8sops get secrets 
NAME                     TYPE                                  DATA   AGE
default-token-p5qzv      kubernetes.io/service-account-token   3      32s
flux-git-deploy          Opaque                                1      26s
flux-token-vbgzc         kubernetes.io/service-account-token   3      25s
weave-net-token-29nsj    kubernetes.io/service-account-token   3      29s
wks-controller-secrets   Opaque                                4      34s

$ kubectl -n weavek8sops describe secret wks-controller-secrets 
Name:         wks-controller-secrets
Namespace:    weavek8sops
Labels:       <none>
Annotations:  
Type:         Opaque

Data
====
bootstrapTokenID:          6 bytes
certificateKey:            64 bytes
discoveryTokenCaCertHash:  71 bytes
sshKey:                    3243 bytes

I was just able to upgrade my wksctl 1.14.1 cluster to a later version (1.14.6) but it required a bit more care than I think should be required. I am using footloose with the docker driver, and wksctl 0.8.0.

Desired flow:

• I ensure each machine definition has an accurate version locked, matching the cluster (1.14.1) under versions.kubelet and versions.controlPlane
• I update all of the masters to a later version, 1.14.6, by applying a yaml with the later kubelet/controlPlane versions, all at once.
• wks controller figures out the safe upgrade path
• I don't have to mess with any node role labels or cordoned states on my own

Actually what happens:
wks-controller was scheduled on node0 (master-1), and node0 is the first node to get upgraded. This is where things go wrong.

When the node0 is cordoned, wks-controller is evicted, and evidently loses track of where it was in the upgrade process. It starts again on one of the remaining masters, and begins upgrading node1 (before node0 is finished upgrading.) It looks like the node0 upgrade won't proceed because the partially upgraded node is no longer marked as a master role, and only masters should be upgraded at first.

Start over... this time I am careful to upgrade master-2 and master-3 first, but I don't take any extra steps of caution.

I then mark master-1 as 1.14.6 (time to upgrade) and my kubeconfig loses contact with the cluster. This is expected (there is no load-balancer) so I repoint my kubeconfig from port 6443 to port 6444 by hand. Now I'm getting:

$ kubectl -n weavek8sops logs -f wks-controller-c977b89d6-s5zqw
Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy)

It's not clear what's gone wrong. My next step is to try and make sure wks-controller is rescheduled before bringing down node0, so that it never gets interrupted and doesn't lose track. (When I upgrade master-2 and master-3 first, the expected "one at a time" behavior proceeds normally and both nodes return with the new control plane version successfully.)

But the bug report here is, wks-controller seems to forget when the node it was preparing to upgrade was a master after cordon, when it has been interrupted in the most likely point.

It would be good to have a logo image that we could upload to all repositories where wksctl appears.

The first example I have is the linux snap store (#71) - see fluxctl for an example.

For flux to work with private hosted git, we need to add known hosts to the flux daemon. See:
https://docs.fluxcd.io/en/1.17.0/guides/use-private-git-host.html

Currently I am not able to use "wksctl apply --git-url ..." as it fails to clone throwing "no such host" error.

I wonder if we could add an arg to pass something like --known-hosts-file which would then add this as a configmap to flux and it would then be able to clone private repos?

The wksctl init command needs more detailed documentation to make its contract clear to users.

Rather than compiling from source.

When following GCE docs here https://wksctl.readthedocs.io/en/latest/wks-on-gce.html this line is throwing

./generate-machines-manifest.sh
generate-machines-manifest.js:3
const input = std.param.String("instances", "instances.json");
                        ^
TypeError: Cannot read property 'String' of undefined
    at generate-machines-manifest.js:3:25

Here're the changes I used to get past the error: #96

if you download the release binary (on a mac) the version reported from wksclt version is undefined. This causes the controller image to use a tag of undefined which doesn't exist. If you build from source, the version is correctly reported as 0.8.0-beta.2

Is this a problem with goreleaser?

The namespace setting for --memcached-hostname= is refernces the wrong namespace. Instead of memcached.flux.svc.cluster.local it should be memcached.weavek8sops.svc.cluster.local

The chunk of code that tells whether the cluster/machines.yaml files come from a local FS or from a git repo to be cloned is identical in several places in our code. This is not good, because these implementations will diverge and cause extra maintenance burden.

When using these directions https://wksctl.readthedocs.io/en/latest/wks-on-gce.html I get the error below.

bob@starship:~/Downloads/wksctl/examples/gce$ wksctl apply
Error: failed to create SSH client: failed to read private key from "/home/bob/Downloads/wksctl/examples/gce": failed to read private key "/home/bob/Downloads/wksctl/examples/gce": read /home/bob/Downloads/wksctl/examples/gce: is a directory
Usage:
  wksctl apply [flags]

Flags:
      --cluster string              Location of cluster manifest (default "cluster.yaml")
      --config-directory string     Directory containing configuration information for the cluster (default ".")
      --git-branch string           Git branch WKS should use to sync with your cluster (default "master")
      --git-deploy-key string       Path to the Git deploy key
      --git-path string             Relative path to files in Git (default ".")
      --git-url string              Git repo containing your cluster and machine information
  -h, --help                        help for apply
      --machines string             Location of machines manifest (default "machines.yaml")
      --namespace string            namespace override for WKS components (default "weavek8sops")
      --sealed-secret-cert string   Path to a certificate used to encrypt sealed secrets
      --sealed-secret-key string    Path to a key used to decrypt sealed secrets
      --use-manifest-namespace      use namespaces from supplied manifests (overriding any --namespace argument)

Global Flags:
  -v, --verbose   Enable verbose output

failed to create SSH client: failed to read private key from "/home/bob/Downloads/wksctl/examples/gce": failed to read private key "/home/bob/Downloads/wksctl/examples/gce": read /home/bob/Downloads/wksctl/examples/gce: is a directory

Here's the content of directory where wksctl apply is be exeucted"

bob@starship:~/Downloads/wksctl/examples/gce$ tree
.
├── cluster-key
├── cluster-key.pub
├── cluster.yaml
├── create-instances.sh
├── create-network.sh
├── delete-instances.sh
├── delete-network.sh
├── generate-machines-manifest.js
├── generate-machines-manifest.sh
├── instances.json
├── machines.yaml
├── repo-config.yaml
└── ssh_key.list

Thanks
Bob

Currently the integration tests are disabled (despite appropriate jobs showing up in CircleCI).

Checklist

• container-tests - #81
• integration-tests-container - #81
• integration-tests-gcp-centos - #84
• integration-tests-gcp-ubuntu - #84

the apply semantics of most cloud native tools are idempotent, but wksctl is not. This breaks expectations for users who think that apply will not reset their cluster, even though it does.

This could be fixed by either making apply idempotent or renaming the command.

It's easy enough to modify a running flux deployment to accept a known_host config map, but there are no hooks in wksctl to inject the known_hosts for wksctl to work with custom git hosts.

This is a non-trivial limitation that prevents wksctl being used with Github enterprise or custom git hosts while retaining a clear GitOps provisioning process.

The wksctl command is always overriding the default image value instead of using the one from wks-controller.yaml (if present).

I'm removing the default override.

When using wksctl and the --git* args, if you don't specify a git-path, flux is configured with "" instead of ".".

When using the -v flag for the apply, kubeconfig, or view commands, the following error is reported:

Error: unknown shorthand flag: 'v' in -v

The -v flag should work identically to the --verbose flag.

In standalone mode, wksctl builds a static cluster based on the contents of cluster.yaml and machines.yaml …

It'd be good to link to where these files come from and how they should be structured. Is it

• https://release-0-1.cluster-api.sigs.k8s.io/common_code/cluster_controller.html
• https://release-0-1.cluster-api.sigs.k8s.io/common_code/machine_controller.html

This is v1alpha1, right - are we moving to v1alpha2 some time?

In the last couple of days, we've had two PRs (#91 and #92) submitted to this repo for which the CircleCI tests didn't run automatically.

Moreover, there was no easy way for me as a maintainer to trigger the test run so I had to copy the branch and push it to this repo to make CircleCI run the tests.

It would be best if the tests ran seamlessly also for the PRs coming from the forked repos, but a good enough start might be for the maintainers to be able to trigger them manually.

We already have this working in https://github.com/weaveworks/scope, but I'm not sure what needs to be done - maybe @bboreham knows more.

cc @palemtnrider

Currently wksctl operations (such as wksctl apply, wksctl plan) are implemented directly as Cobra subcommands which disables their reuse with Go APIs. This issue is about refactoring these subcommands to make them usable as Go APIs for any project that imports wksctl as a Go module.

To avoid scope creep, no behavior change is planned within the scope of this issue. The Go interfaces created should match the existing CLI flags.

I expect some fixes in propagation of errors to be necessary (so that log.Fatalf-alikes don't kill the API caller).

I built wksctl and thought I'd check how best to use it and saw this:

[daniel@reef wksctl ]$ ./cmd/wksctl/wksctl -h
Weave Enterprise Kubernetes Subscription CLI

Usage:
  wksctl [command]

Available Commands:
  addon                  Manipulate addons
  apply                  Create or update a Kubernetes cluster
  bash-completions       Generate bash completion scripts
  help                   Help about any command
  init                   Update stored kubernetes manifests to match the local cluster environment
  kubeconfig             Generate a kubeconfig file for the cluster
  registry-sync-commands Synchronize container images to an internal registry
  version                Display wksctl version
  zsh-completions        Generate zsh completion scripts

Flags:
  -h, --help      help for wksctl
  -v, --verbose   Enable verbose output

Use "wksctl [command] --help" for more information about a command.
[daniel@reef wksctl ]$ 

Is "Weave Enterprise Kubernetes Subscription CLI" the name we want to use? It doesn't sound like its open source, or like I need to be subscribed to something.

We require a few dependencies, e.g., Docker running on the machine, in order to build wksctl. We need to identify these in the README.md

Is $GOPATH/bin a better location or also not always there?

This is just a couple of ideas when looking at the main README.md.

Current sections are:

# Weave Kubernetes Subscription Control - `wksctl`
## Install wksctl binary 
## Quick start
## Modes of use
### Standalone mode
### GitOps mode
## Development
### Build
#### Upgrading the build image
# Checkpoint
## Contributing
## Getting Help
## License

What I'd like to do is:

• Explain better what wksctl is good for and why people should use it, list features(?)
• Keep install and quick start
• Move modes of use to a separate get started doc
• Add docs directory
• Move development bits to CONTRIBUTING.md - make sure it's sufficiently linked
• move examples README.md files to docs
• Move checkpoint bits to a FAQ doc

User story:

As a Firekube user,
I'd like to have Flux Helm Operator as an add-on,
so that I can be able to deploy HelmRelease resources OOTB

As a profile developer,
I'd like to see wksctl support HelmRelease resources,
so that I can include this kind of resources into my profiles.

/cc @palemtnrider could you ptal for go/no-go

Not sure what happened recently: installing a two node cluster results in:

$ kubectl get nodes
NAME               STATUS   ROLES    AGE     VERSION
5a54f5fde41d3ad3   Ready    master   2m49s   v1.14.1
9aa426eda1f21c4d   Ready    <none>   62s     v1.14.1

As a user
I'd like to enable multiple profiles using one command
so that I can have a powerful multi-functional cluster OOTB

The command should display the stderr of the command that has failed so we can understand why the command failed.

Error log:

INFO[2019-09-17T21:29:42+07:00] State of Resource 'install:device-mapper-persistent-data' is Invalid.
Explanation:
{
 "resource": "install:device-mapper-persistent-data",
 "status": "Invalid",
 "reason": "ApplyError",
 "error": "command exited with 1"
} 

According to this: https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#known-issue-etcd-client-balancer-with-secure-endpoints

We should ship etcd 3.3.14 or later. However, the current version of wksctl will install 3.3.10.

WDYT? cc @palemtnrider

We need users to be able to run wksctl w/o having to build the source. To me this means, tagging the master branch, creating images stored in quay.io with that tag, ensuring wks-controller is using that image tag, and a tar file that a user can download which contains the wksctl binary and the example files.

I believe the first release should be v0.7.0-alpha.1

Maybe goreleaser can help us implement this?

We will also need to update the README.md to instruct users how to do this.

For PRs and master branch we need images built and pushed to quay.io so the wks actuator is able to start in k8s.

It'd be good if we explained who's responsible for this code.

https://github.com/fluxcd/flux/blob/master/MAINTAINERS could serve as an example.

https://goreleaser.com/homebrew/

The code would look something like:

brew:
  github:
    owner: weaveworks
    name: homebrew-tap
  commit_author:
    name: weaveworksbot
    email: [email protected]
  folder: Formula
  homepage: https://github.com/weaveworks/wksctl
  description: Open Source Weaveworks Kubernetes

Acceptance criteria:

• tested on mac
• tested on linux

When using wksctl to start a multi-master Firekube cluster, the cluster ended up like this when the 2nd or the 3rd master started to join the cluster. This failure is deterministic and always reproducible.

time="2019-10-19T10:52:19Z" level=debug msg="running command: sudo -n -- sh -c 'cat /etc/machine-id 2>/dev/null || cat /var/lib/dbus/machine-id 2>/dev/null'"
7a98435980b347bf8935da14e206654f
time="2019-10-19T10:52:19Z" level=debug msg="running command: sudo -n -- sh -c 'cat /sys/class/dmi/id/product_uuid 2>/dev/null || cat /etc/machine-id 2>/dev/null'"
7a98435980b347bf8935da14e206654f
time="2019-10-19T10:52:19Z" level=debug msg="running command: sudo -n -- sh -c 'command -v -- \"selinuxenabled\" >/dev/null 2>&1'"
time="2019-10-19T10:52:19Z" level=debug msg="running command: sudo -n -- sh -c 'selinuxenabled'"
time="2019-10-19T10:52:19Z" level=debug msg="running command: sudo -n -- sh -c 'cat /proc/1/environ'"
time="2019-10-19T10:52:19Z" level=debug msg="the following env-specific configuration will be used" config="&{0 true [] true true true weavek8sops }"
time="2019-10-19T10:52:19Z" level=info msg="cordon node \"ff308594e2a1c34e\""
time="2019-10-19T10:52:21Z" level=warning msg="ignoring DaemonSet-managed Pods: kube-system/kube-proxy-9xbjm, weavek8sops/weave-net-2m8l2"
time="2019-10-19T10:52:24Z" level=debug msg="5 pod(s) to be evicted from ff308594e2a1c34e"
time="2019-10-19T10:52:24Z" level=fatal msg="<nil>"

After that, kubectl could not connect to the API server any more.

Here's the config.yaml:

# This file contains high level configuration parameters. The setup.sh script
# takes this file as input and creates lower level manifests.

# backend defines how the machines underpinning Kubernetes nodes are created.
#  - docker: use containers as "VMs" using footloose:
#            https://github.com/weaveworks/footloose
#  - ignite: use footloose with ignite and firecracker to create real VMs using:
#            the ignite backend only works on linux as it requires KVM.
#            https://github.com/weaveworks/ignite.
backend: ignite

# Number of nodes allocated for the Kubernetes control plane and workers.
controlPlane:
  nodes: 3
workers:
  nodes: 1

https://docs.readthedocs.io/en/stable/webhooks.html

In the footloose docs we should download a prebuilt binary (v0.6.0+) instead of go install-ing it, which requires the user to have Go installed correctly (non-user-friendly)

Need to change the default version of kubernetes installed to a more recent version. 1.16.0 was just released so I believe we should update the install to be 1.15.4

It appears that eksctl only supports stacked etcd nodes. In production its in general better to decouple the two, but it would still be nice to benefit from wksctl's declarative YAML interface.

This is a feature request to add to the Machine CRD support for declaring etcd nodes that would be provisioned via etcdadm or by kubeadm for now. For example, a simple 3 node cluster (master, etcd, and worker node) could be defined as:

apiVersion: v1
kind: List
items:
- apiVersion: cluster.k8s.io/v1alpha1
  kind: Machine
  metadata:
    name: master
    labels:
      set: master
  spec:
    providerSpec:
      value:
        apiVersion: baremetalproviderspec/v1alpha1
        kind: BareMetalMachineProviderSpec
        public:
          address: 127.0.0.1
          port: 2222
        private:
          address: 172.17.0.2
          port: 22
- apiVersion: cluster.k8s.io/v1alpha1
  kind: Machine
  metadata:
    name: etcd
    labels:
      set: etcd
  spec:
    providerSpec:
      value:
        apiVersion: baremetalproviderspec/v1alpha1
        kind: BareMetalMachineProviderSpec
        public:
          address: 127.0.0.1
          port: 2223
        private:
          address: 172.17.0.3
          port: 22
- apiVersion: cluster.k8s.io/v1alpha1
  kind: Machine
  metadata:
    name: worker-1
    labels:
      set: worker
  spec:
    providerSpec:
      value:
        apiVersion: baremetalproviderspec/v1alpha1
        kind: BareMetalMachineProviderSpec
        public:
          address: 127.0.0.1
          port: 2225
        private:
          address: 172.17.0.5
          port: 22

Thoughts? There could perhaps be some other configuration that maps to etcdadm cli flags

We need to refresh the version of Docker installed by default. 19.03.1 seems like a good candidate and has been out for a few months.

As a user,

I would like to reuse profiles from eksctl and apply to my local cluster.

So that I can have the similar cluster functionalities and help me validate the behavior of my EKS cluster.

Local eksctl-like experience: we could use Firekube to replicate eksctl + AWS behavior by having the same set of eksctl profiles running on top of Firekube. This will be enabled by implementing --profile option for wksctl.

Here's the comment from @mflendrich on UX:

In this proposal enable becomes a command group that has one subcommand: profile. disable becomes another command group with one subcommand profile. This solution has an issue that enable/disable don't wrap a bounded context in the problem domain, so it might be better to proceed with wksctl profile as a command group with subcommands enable/disable: that would mean wksctl profile enable, wksctl profile disable

There are quite a few setup things that happen asynchronously when setting up a cluster with wksctl. It would be useful to have a wksctl status command to inform the user what's happening behind the scenes.

When running wksctl on a cluster more than once, wksctl destroys the cluster and re-applies configuration.

This would be a nice feature that it detects any changes in the configuration and only applies whats needed, a bit like how kubectl would work.

A long with this have an argument to destroy the cluster.

https://github.com/weaveworks/wksctl/releases/tag/0.8.0-beta.2 was not tagged as a pre-release as all the others were (e.g. 0.8.0-beta.1, 0.8.0-alpha.2, 0.8.0-alpha.1)

If this was done in error, we should investigate whether we can stop this kind of error from happening again in the future with a pre-commit hook.

@palemtnrider from what I can tell you were the tagger in this instance:

$ git show 0.8.0-beta.2
tag 0.8.0-beta.2
Tagger: Mark Emeis <[email protected]>
Date:   Thu Sep 19 18:31:59 2019 -0600

0.8.0-beta.2

commit cab2ce29e590fda896a7f70993d778247cb9f8dd (tag: 0.8.0-beta.2)
Author: Mark Emeis <[email protected]>
Date:   Thu Sep 19 18:26:26 2019 -0600

If so, please let me know if there was a reason why this tag was not a pre-release.

In the context of an open source project this is a non-trivial error (again, if it was an error) because a user's automated tooling may be watching for releases. Since the exact purpose of the pre-release option is to point out to users that this release is not ready for production, this is important to the user that we get right.

After 0.8.0 is released, we will need to merge the changes from the release/0.8.0 branch into master.

Inspired by #73. Affects wksctl apply and wksctl plan.

It's currently impossible to reliably inject the controller image to use, and the logic that determines the image tag is complex and has several inputs (the default value, the command line value, a manifest in the repo).

This situation leads to unexpected interactions. A good solution would be to either validate inputs (and fail loudly if more than 1 competing value is provided to wksctl, or refactor to use only one source of truth about the container image tag that will be run.

When using wksctl/centos, it is required to have 2 configmaps for docker and kubernetes installs.

If you change the names of the configmap files, for example "docker-config,yaml" to "test-docker-config.yaml"

When running "wksctl apply...." it will fail as it can't find "docker-config,yaml".

I can't see anywhere where this requirement is hardcoded. Can this be updated so it just applies the configmaps in the current directory and not care about file names?

Need to migrate to using cluster-api v1alpha2.

We need to install:

• wksctl
• jk
• footloose
• ignite