Provide a generic mechanism for flag customizations about bootkube HOT 5 CLOSED

Configuring manifest flags with another config map seems to add a decent amount of complexity for what is ultimately template rendering.

A big reason I want to go this route is that upstream is moving toward all components being configured via configMaps. If we follow this pattern we stand to get a lot of functionality out-of-box (e.g. we can validate initial configuration, then push that configuration directly into the api as configMaps + our components will natively use those objects already).

Rather than coming up with an intermediary config (where config complexity adds up quickly -- see kube-up, kube-aws, etc.), we just use the native end-state -- and provide early validation.

I think we're moving to simply use bootkube start only and create our own manifests going forward

I think this will be a much more common route as people's cluster configuration becomes more standardized for their own environments / is committed after initial rendering. This ultimately is the preferred end-state as we're never going to be able to provide "default configuration" that is everything to everyone.

However, I think this still lends really well to standardizing on configMaps + validation, because it is the standard way of expressing configuration of components -- so even if you don't use our tools to generate manifests at all - as long as you're using configMaps we can at least do some validation (rather than trying to parse out flags from an exec line).

from bootkube.

Another item brought up is that some flags are going to be common (but cannot be assumed as defaults). Such as --cloud-provider=foo (#105 (comment))

I'm wondering about the option of layering multiple configMaps, such that we could say something like:

bootkube init --profiles="default,cloudprovider-aws"

This would then use the set of flags present in the default profile (which is just a set of configMap objects), but then also adds (or overwrites existing) flags that are present in the cloudprovider-aws. The output would be the combination of both profiles.

For example, the default profile configMap for the apiserver might look like:

apiVersion: v1
kind: ConfigMap
data:
  bind-address: 0.0.0.0
  allow-privileged: true

And the cloudprovider-aws profile configMap would be:

apiVersion: v1
kind: ConfigMap
data:
  cloud-provider: aws

And the resulting apiserver configMap would be:

apiVersion: v1
kind: ConfigMap
data:
  bind-address: 0.0.0.0
  allow-privileged: true
  cloud-provider: aws

Internally in the bootkube binary we could support a few common profiles - but letting other's self-provide a profile could just mean pointing to a particular directory which contains configMaps.

/cc @dghubble @sym3tri @derekparker @pbx0

from bootkube.

I think it would be nice to still provide a couple of the more commonly-used options as command-line flags to bootkube -- first and foremost, cloud-provider and things cloud-provider needs to work (e.g. taking into account things like kubernetes/kubernetes#26897). People understand flags and there aren't as many syntax worries as if I have to edit YAML or JSON.

But yes, it will quickly get unwieldy if bootkube has to support every flag for every Kubernetes component ever and so I think it's entirely valid to draw a line and say "beyond this point you have to do X by creating a Y" (whether Y is a configMap, some sort of YAM or JSON manifest, or even just a plaintext file with the desired flags per component).

from bootkube.

Configuring manifest flags with another config map seems to add a decent amount of complexity for what is ultimately template rendering. On my side, we're already rendering lots of templates and generating TLS assets so I think we're moving to simply use bootkube start only and create our own manifests going forward. Perhaps the validate could be useful if it could be called programmatically.

from bootkube.

Now that #168 is implemented - the needed flexibility is essentially allowing an admin to modify anything about the rendered manifests. This proposal wouldn't really add much unless we wanted to further abstract higher-level concepts (which we likely do not).

I still think we probably want to move to a versioned config file rather than flags - but opened: #565 to track that

from bootkube.