Support OOT credential provider for all templates about cluster-api-provider-azure HOT 12 OPEN

/reopen

@lzhecheng #4169 added OOT credential provider to all the k8s CI artifacts templates, however we need a more user friendly way to install it (e.g. Helm chart) before we can add it to all templates.

from cluster-api-provider-azure.

@CecileRobertMichon I never update templates in capz repo. Can you show how/example for me or you can help take this issue? I have included all necessary changes in issue description. Thank you.

from cluster-api-provider-azure.

Should the credential provider yaml be downloaded at runtime? Or can we put it into the reference OS image?

from cluster-api-provider-azure.

Should the credential provider yaml be downloaded at runtime? Or can we put it into the reference OS image?

Yes I think it can be put into the image.

from cluster-api-provider-azure.

cc @mboersma

@lzhecheng Thanks! Does OOT credital provider work with versions of k8s < 1.29? If so can we backport these files into each cloud-provider-azure release branch:

https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/examples/out-of-tree/credential-provider-config.yaml

As far as I know CAPZ builds OS images for each release, so we'd want to update those CI to pull the release-appropriate yaml file to put into /var/lib/kubelet.

from cluster-api-provider-azure.

@jackfrancis yes, I can do it. Notice, windows config file is different from linux one so its config file is called credential-provider-config-win.yaml and should be renamed when put into /var/lib/kubelet.

from cluster-api-provider-azure.

Another option: we could generate the file in our existing azurejson controller and then add it to the VM in the template the same way we add azure.json

Pros:

• it's more consistent with how we do out of tree cloud provider
• changing the config doesn't require building a new image
• we can still test in tree creds provider if needed

Cons:

• it requires every template to have the file added under KubeadmConfig "files" (but looks like we have to do that for kubeletExtraArgs in any case)

from cluster-api-provider-azure.

curl --retry 10 --retry-delay 5 -Lo /var/lib/kubelet/credential-provider/acr-credential-provider "https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/${JOB_NAME}/${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/azure-acr-credential-provider"

Looks like the template above is building the OOT credential provider binary for the purpose of testing it, how does it work when we don't want to build it ourselves but just use a release? Is that where it would need to be put on the VM OS image? or is there a helm chart for it to install it on k8s clusters? @lzhecheng

from cluster-api-provider-azure.

curl --retry 10 --retry-delay 5 -Lo /var/lib/kubelet/credential-provider/acr-credential-provider "https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/${JOB_NAME}/${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/azure-acr-credential-provider"

Looks like the template above is building the OOT credential provider binary for the purpose of testing it, how does it work when we don't want to build it ourselves but just use a release? Is that where it would need to be put on the VM OS image? or is there a helm chart for it to install it on k8s clusters? @lzhecheng

My recent change to ci-entrypoint.sh doesn't support it. I think it should be done the same way as CCM.
So far no helm chart.

from cluster-api-provider-azure.

/assign
/milestone v1.12

from cluster-api-provider-azure.

@CecileRobertMichon: Reopened this issue.

from cluster-api-provider-azure.

@lzhecheng it seems that the credential provider doesn't work with kubelet 1.25? The tests are failing for k8s 1.25 since this merged, kubelet is crashing with

Nov 04 20:58:31.931758 capz-conf-wmjvfg-control-plane-s64vm kubelet[2684]: E1104 20:58:31.931732    2684 kuberuntime_manager.go:261] "Failed to register CRI auth plugins" err="error decoding config /var/lib/kubelet/credential-provider-config.yaml: no kind \"CredentialProviderConfig\" is registered for version \"kubelet.config.k8s.io/v1\" in scheme \"pkg/credentialprovider/plugin/plugin.go:56\""

https://storage.googleapis.com/kubernetes-jenkins/logs/capz-conformance-dual-stack-1-25/1720896299273818112/artifacts/clusters/capz-conf-wmjvfg/machines/capz-conf-wmjvfg-control-plane-b8zps/kubelet.log
https://testgrid.k8s.io/provider-azure-1.25-signal#capz-conformance-dual-stack

from cluster-api-provider-azure.