Cluster Security Group Change Issue about terraform-aws-eks HOT 7 OPEN

I've been seeing this error a lot. All the other issues that I've seen with this are closed without solution. Theres the workaround of changing the IP ranges to force a change but thats not pratical. Another hacky way I found is to create a variable and append it to the key of the security group rules. For instance, in your case would be something like:

variable "security_group_update_trigger" {
  description = "Trigger updates without getting duplicate security group rule error"
  type        = bool
  default     = false
}

  cluster_security_group_additional_rules = merge(
    {
      # Default Private Control Plane Access
      "vpc_${var.security_group_update_trigger}" = {
        description = "Allow VPC Network Access"
        protocol    = "tcp"
        from_port   = 443
        to_port     = 443
        cidr_blocks = data.aws_vpc.vpc.cidr_block_associations[*].cidr_block
        type        = "ingress"
      },
      "private_network_access_${var.security_group_update_trigger}" = {
        description = "Allow Private Network Access"
        protocol    = "tcp"
        from_port   = 443
        to_port     = 443
        cidr_blocks = ["10.0.0.0/8"]
        type        = "ingress"
      }
  }, var.cluster_security_group_additional_rules)

But if you have a huge list of CIDRs like in my case, its bad since you dont have the easy tracking of changes when adding/removing a single cidr, because it will show as adding/removing all.

I hope we can have a proper fix for this issue.

from terraform-aws-eks.

I don't see how these relate to the module - this looks like you could easily reproduce this on just simple security group rule resources.

from terraform-aws-eks.

I guess I had opened this because it was not clear to me why the aws_security_group_rule was triggering a replace as opposed to just modifying it. It seemed like it was a side effect of however its been implemented in the module.

from terraform-aws-eks.

its being replaced due to changes that it has identified here:

~ cidr_blocks = [ # forces replacement
      # (2 unchanged elements hidden)
      "10.60.7.0/24",
    + "10.40.0.0/16",
  ]

And I suspect the duplicate rules is being thrown because both 10.60.7.0/24 and 10.40.0.0/16 are already captured within the rule

private_network_access = {
    description = "Allow Private Network Access"
    protocol    = "tcp"
    from_port   = 443
    to_port     = 443
    cidr_blocks = ["10.0.0.0/8"]
    type        = "ingress"
  }

from terraform-aws-eks.

I can confirm the second isn't an issue, we had multiple overlapping rules already in place within the rule that do overlap with 10.0.0.0/8 and should have failed if that was an EC2 SG limitation.

I was also able to manually add 10.40.0.0/16 within the EC2 UI without error, its just Terraform failing because it is attempting to create_before_destroy the aws_security_group_rule and the rule it attempts to create is already on the SG.

To get around this issue, since I already had that overlapping 10.0.0.0/8 rule defined in Terraform I actually just deleted all the other more specific rules Terraform was trying to append manually, removed them from state, then re-applied Terraform successfully.

I'd suspected something about how the variables are getting passed into the aws_security_group_rule resource was triggering the create_before_destroy: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/main.tf#L296-L299

But wasn't sure how to investigate further. Also wonder if this even newer resource might avoid this as well, especially since within the EC2 console, each CIDR block we're adding has its own unique security group rule ID already: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_security_group_ingress_rule
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_security_group_egress_rule

from terraform-aws-eks.

the newer individual rules (cannot pass a list of CIDRs, etc.) are recommended but, that comes at the cost of disruption to users. we have started to use those where possible, but we have to identify the right time to introduce those since it will be disruptive for users

from terraform-aws-eks.

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

from terraform-aws-eks.