EKS Cluster detach does not delete k8s resources from cluster about terraform-provider-tanzu-mission-control HOT 6 CLOSED

@jorgemoralespou I believe the stated bug is not emanating from TMC terraform provider, as its also seen when you do delete operation from TMC UI. I see an internal VMware bug report (OLYMP-33128) on EKS/Auth team to ensure they cleanup the said EC2 ELB resource when delete call is made.

from terraform-provider-tanzu-mission-control.

I'm no engineer, but looking at the code, it seems that on creation, the cluster is attached and the required k8s manifests installed on the cluster (https://github.com/vmware/terraform-provider-tanzu-mission-control/blob/main/internal/resources/cluster/resource_cluster.go#L343-L362) but on deletion, no similar approach is taken, but rather delegating to TMC deletion (https://github.com/vmware/terraform-provider-tanzu-mission-control/blob/main/internal/resources/cluster/resource_cluster.go#L463) which seems to call the TMC API (https://github.com/vmware/terraform-provider-tanzu-mission-control/blob/main/internal/client/cluster/cluster_resource.go#L74-L92). If the operation fails, the message is concerning (https://github.com/vmware/terraform-provider-tanzu-mission-control/blob/main/internal/resources/cluster/resource_cluster.go#L468-L469) as it points to https://docs.vmware.com/en/VMware-Tanzu-Mission-Control/services/tanzumc-using/GUID-3061A796-CA3D-4354-A0B7-19F50F2617CE.html where it's not specified how to delete the pinniped svc that creates the ELB.

from terraform-provider-tanzu-mission-control.

@jorgemoralespou not sure I completely understand what steps you took- did you detatch the cluster from TMC and then delete it? If it had been detached, do you mean that you deleted it manually in EKS, not using terraform?

from terraform-provider-tanzu-mission-control.

No, terraform does create with VPC and EKS module, then tanzu-mission-control provider does the attach. On destroy, TMC provider does the de-attach, and TF EKS module does the cluster deletion and then VPC deletion. At this point, VPC can not be deleted as there's referenced resources that had not been cleared up.
We pivoted from using TMC provider creating EKS cluster to using it to Attach, as we expected detaching would remove the k8s resources that create ELB (pinniped). This has not been the case.

We're considering now not using TMC at all for our cluster management, since it gives us more problems in automation than expected and desired.

from terraform-provider-tanzu-mission-control.

The TMC engineering team recently updated the Cloud Formation template which creates permissions used by TMC to lifecycle manage EKS clusters. This change added loadbalancing permissions including DescribeLoadBalancers, DescribeTags and DeleteLoadBalancer. This change was done in order to fix an issue where cleaning up EKS clusters could leave behind loadbalancers in AWS accounts. You may notice that credentials become INVALID, and will want you to update your permission template, which will return your credentials to a VALID state. While credentials are invalid, it will prevent you from making new lifecycle operations on your EKS clusters, but has no effect on the state of your clusters.
Please follow these steps to update your credential permissions : https://docs.vmware.com/en/VMware-Tanzu-Mission-Control/services/tanzumc-using/GUID-E64C94C6-447B-47E3-BB8A-8D300F4A6512.html

@jorgemoralespou : EKS team has fixed a bug for "Delete pinniped created load balancer when deleting EKS cluster" - OLYMP-39798. So can we confirm the issue report again, as the fix (OLYMP-39798) is live and can you please try with said changes?

from terraform-provider-tanzu-mission-control.

Closing issue - as we believe it was fixed by TMC EKS and users are requested to ensure credential permissions are as outlined in earlier comment.

from terraform-provider-tanzu-mission-control.