[BUG] - Nodes remain uninitialized --> CCM complains about "providerID cannot be an empty string" about docker-machine-driver-vultr HOT 7 CLOSED

Hello @kgeipel-retail7

I believe there is some misunderstandings from my comment. I'm fully aware of how the ProviderID should be constructed and the statement that it's just UUID is correct as the "vultr://" is more of an indicator for the provider(us) parsing it so we ensure the correct value is present.

Additionally, when I stated usage without the providerid that was meant for initial lookup in the CCM when the instance joins the cluster. When a node joins it attempts to register with the CCM and when the lookup is completed successfully the taint is removed. At this time we set the providerID in the kubelet config for VKE so there was not a reason to lookup by name or set the providerID in the metadata. This is no longer the case due to clusters such as Rancher so it is something we would want to extend the capability for.

from docker-machine-driver-vultr.

Hello @kgeipel-retail7

Thank you for reporting this! The Vultr-CCM is primarily used for VKE and requires the providerID which is essentially just the Instance UUID of the machine so that it can quickly locate it. Unfortunately, docker-machine is not just for rancher/k8s so it does not have the ability to add labels to nodes.

We can review adding the ability of the CCM to allow for usage without providerID labels, however, this would mean that it would have to operate off the node name so it would be logical to add a UID into the name of the machine to ensure duplicate labels do not cause issues with the CCM.

from docker-machine-driver-vultr.

Hi @happytreees thanks for the fast reply

EDIT:

the providerID which is essentially just the Instance UUID of the machine

And it has to have a prefix containing the provider name, in case of vultr it should be something like vultr://UUID

The CCM should in fact be enhanced to be used in other k8s environments, since the node driver is promoted to be used in Rancher as well

We can review adding the ability of the CCM to allow for usage without providerID labels

This is not the way to go i guess, the CCM should add this metadata to the provisioned nodes, like they do it here and is described here

otherwise the nodes will keep this "NoSchedule" taint forever

from docker-machine-driver-vultr.

@happytreees thanks for the detailed explanation. Can you say how long it would take to implement such an extension? We really would like to have Vultr on our supported hoster list.

from docker-machine-driver-vultr.

@kgeipel-retail7

I don't have an estimated timeline but it also doesn't look difficult to do either. I will work on it this week.

I am going to close this issue, however, if you want you can open an issue on our CCM repo and I can update that once the PR is in to add that.

https://github.com/vultr/vultr-cloud-controller-manager/tree/master

from docker-machine-driver-vultr.

@kgeipel-retail7 Can you share which version of the CCM you are using?

I reviewed the code and saw that I inadvertently added this ability to the CCM awhile back for a different issue, however, the functionality is there. You can see in this function that if the providerID is blank then it looks it up using the name and it's also set in the metadata:
https://github.com/vultr/vultr-cloud-controller-manager/blob/master/vultr/instancesv2.go#L123

I went ahead and deployed an RKE1 cluster using this driver and added the CCM as an external cloud provider. Once the nodes were up I ran into a small hiccup where the CCM required the instances to be attached to a VPC for the internal IP, however, once attached the CCM added the labels and providerID without any issues. I confirmed by reviewing the node itself and the providerID and vultr labels were present.

As long as you use the newest version of the CCM v0.10.0 and ensure the instances get attached to a VPC(you can specify the VPC within the node template in rancher) then everything should come up without issue.

In the new year I'm going to be adding the addons yaml to this directory to allow for the CCM, CSI to be added simply, but it is not present at this time.

from docker-machine-driver-vultr.

Hi @happytreees as written above we are using the latest version Vultr CCM v0.10.0 attaching a VPC to the nodes did the trick, now everything works like expected.

Only one small issue still exists with the CCM, in CCM deployment file there is a node scheduling rule which doesn't match any node in the cluster, since RKE clusters have no node annotation/label or taint "node-role.kubernetes.io/master"

So as a workaround, i simply cloned the workload as of type deployment and removed the scheduling rule.

from docker-machine-driver-vultr.