Add support for custom ipxe boot about cluster-api-provider-packet HOT 6 CLOSED

Hey @gianarb, I'm gonna chime in and hopefully add a bit more detail about what @andrewrynhard and I are up to. We've been using Packet for a long time as our base for our CI nodes with the Talos project, as well as doing some e2e testing with it as well ( <3 you all for the sponsorship ). TLDR on Talos is that it's an OS that's tailor-made for Kubernetes and provides an API instead of things like a shell and SSH.

We've always installed the OS via PXE because we're not a supported OS out there (yet!). So we've got lots of tooling around PXE booting as well as own Talos bootstrap provider for CAPI. I think we could totally get away with what you're saying by just exposing the PXE URL to hit. It can be our responsibility to make sure that the metadata that gets passed to the machine during and after installation is valid.

In regards to the stuff you mention in #118, we don't really need to worry about that either. We don't have to use clusterctl to generate cluster manifests for us; we're happy to bring our own. But maybe this is a good segue to a conversation with the larger CAPI team to figure out how we might all support clusterctl and cluster-template.yaml being more dynamic.

Feel free to ping me on the K8s slack channel if you want to discuss further!

from cluster-api-provider-packet.

There is not, you are right and this is because we rely on cloud-init and on a specific package management apt right now to make the cluster-api to work.

There is an open issue about how to support multiple operating systems #118 .

I think we can take some middle steps here in order to enable some use cases, but it will help me if you can explain to me your use case in practice and where you are blocked so we can try to figure out how to make it to work.

For example, it is easy to expose the "ipxe_script_url": "string" if you need that, but I am not sure about how the cloud-init script will take effect, plus you have to install an os that as apt, that is not necessary if you will have your own cluster specification.

from cluster-api-provider-packet.

This one is interesting. We definitely want to get less opinionated about what OS you are running under CAPP, but at the same time, you do need the bootstrap provider to support it. A relatively easy solution may be to provide a Talos bootstrap provider (and maybe you already have one?) that can be used in the cluster template. Unfortunately, cluster template is not quite so easy to do things like swapping out whole blocks; I wish it were.

We still would need to inject a different OS into the PacketMachineTemplate, but that shouldn't be too hard.

from cluster-api-provider-packet.

Yeah, so the Talos bootstrap and control plane providers already exist, you can find them here and here if you have any interest.

We use these with a bunch of other infra providers already (our own metal provider, azure, gcp, aws). I think it's a pretty common pattern, and something we're totally cool with, to just keep your cluster-template.yaml the way it is. It works for the 90% use case and I'm not overly interested in using the clusterctl config commands anyways. If you're doing anything out of the ordinary on any of the infra providers, you pretty much have to make edits to the cluster template that gets spit out anyways. We'll bring our own kustomize for the packet clusters. I do have longer term aspirations to try and make clusterctl better though so you can plug in different bootstrap providers.

WRT the PacketMachineTemplate, I think that's all we really want right now. I think it should totally be possible to have the pxe url field exposed and, if it's set, we could override the OS to be "custom" or whatever is needed in the backend. I just haven't looked at packngo in a while so it's not fresh on my brain. I've got some old provider code that did this before CAPI v1alpha2 came around.

@deitch and @gianarb is it cool with you if I prototype this out and throw up a PR for what I have in mind?

from cluster-api-provider-packet.

Now that I better understand the use case I think it is safe to edit the PacketMachine spec to expose the ipxe URL.

Sure feel free to open a PR! I am happy to review it

from cluster-api-provider-packet.

What @gianarb said ^^^^

from cluster-api-provider-packet.