aws / aws-application-networking-k8s Goto Github PK

A Kubernetes controller for Amazon VPC Lattice

Home Page: https://www.gateway-api-controller.eks.aws.dev/

License: Apache License 2.0

Dockerfile 0.08% Makefile 0.54% Go 96.63% Shell 2.23% Smarty 0.52%

aws-application-networking-k8s's Introduction

AWS Gateway API Controller for VPC Lattice

AWS Application Networking is an implementation of the Kubernetes Gateway API. This project is designed to run in a Kubernetes cluster and orchestrates AWS VPC Lattice resources using Kubernetes Custom Resource Definitions like Gateway and HTTPRoute.

Documentation

Website

The API specification and detailed documentation is available on the project website: https://www.gateway-api-controller.eks.aws.dev/.

Concepts

To get started, please read through API concepts. These documents give the necessary background to understand the API and the use-cases it targets.

Getting started

Once you have a good understanding of the API at a higher-level, check out getting started to install your first Gateway controller and try out one of the guides.

References

A complete API reference, please refer to:

Contributing

Developer guide can be found on the developer guide page. Our Kubernetes Slack channel is #aws-gateway-api-controller.

Code of conduct

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.

aws-application-networking-k8s's People

Stargazers

Watchers

aws-application-networking-k8s's Issues

Gabadge globally instead of for each resource (e.g. HTTPRoute)

Today, we reconcile every 10 mins for each resource, e.g. HTTPRoute, this will not scale

We need to change it to global reconcile at longer interval, e.g. 1 hour, or 2 hours

Upgrade to use gateway beta APIs

Kubernetes Gateway API has already graduated to Beta. The current controller still uses v1alpha2 version. It needs to be updated to beta.

Controller `Upgrade` procedure needs to be improved

The eksctl command to setup IRSA is missing the --role-name argument. It is not a bug but it makes the tool create a IAM role with a random name.
The manifest to deploy CRDs and the Gateway Controller has a ServiceAccount definition for gateway-api-controller though we are creating the latter using eksctl. Again, this is not an issue until it is. If you unveil a new version, say, deploy-v0.0.2.yaml, and a user deletes the old manifest with kubectl delete -f and then does a kubectl apply -f with the new manifest, user will end up with a ServiceAccount definition without the IRSA annotation and it will break things.

Failed to create listener after adding a backendRef in httproute

After adding a backendRef in httproute, gateway api controller fails to create a listener.

httproute config:

  - backendRefs:
    - group: ""
      kind: Service
      name: front-end-primary
      port: 80
      weight: 100
    - group: ""
      kind: Service
      name: front-end
      port: 80
      weight: 0
    matches:
    - path:
        type: PathPrefix
        value: /

Error messages in gateway api controller logs:

2023-02-19T03:04:08.709Z	INFO	controller.httproute	Adding/Updating	{"reconciler group": "gateway.networking.k8s.io", "reconciler kind": "HTTPRoute", "name": "front-end", "namespace": "default"}
I0219 03:04:08.709589       1 model_build_lattice_service.go:135] No customter-domain-name for httproute :front-end-default
2023-02-19T03:04:08.709Z	INFO	controller.httproute	Successfully built model:	{"reconciler group": "gateway.networking.k8s.io", "reconciler kind": "HTTPRoute", "name": "front-end", "namespace": "default", "{\"id\":\"default/front-end\",\"resources\":{\"AWS::VPCServiceNetwork::Listener\":{\"front-end-default-0-HTTP\":{\"spec\":{\"name\":\"front-end\",\"namespace\":\"default\",\"port\":0,\"protocol\":\"HTTP\",\"defaultaction\":{\"isimport\":false,\"backendservicename\":\"front-end-primary\",\"backendservicenamespace\":\"default\"}}}},\"AWS::VPCServiceNetwork::Rule\":{\"rule-1\":{\"spec\":{\"name\":\"front-end\",\"namespace\":\"default\",\"port\":0,\"protocol\":\"HTTP\",\"ruletype\":\"HTTPRouteMatch\",\"value\":\"/\",\"id\":\"rule-1\",\"action\":{\"ruletarget\":[{\"name\":\"front-end-primary\",\"namespace\":\"default\",\"isServiceImport\":false,\"weight\":100},{\"name\":\"front-end\",\"namespace\":\"default\",\"isServiceImport\":false,\"weight\":0}]},\"time\":\"2023-02-19T03:04:08.709658968Z\"}}},\"AWS::VPCServiceNetwork::Service\":{\"front-end-default\":{\"spec\":{\"name\":\"front-end\",\"namespace\":\"default\",\"protocols\":[\"HTTP\"],\"servicenetworkhname\":\"sock-shop\",\"customerdomainname\":\"\",\"customercertarn\":\"\",\"IsDeleted\":false}}},\"AWS:VPCServiceNetwork::TargetGroup\":{\"k8s-front-end-default\":{\"spec\":{\"Name\":\"k8s-front-end-default\",\"config\":{\"port\":8079,\"protocol\":\"HTTP\",\"protocolversion\":\"HTTP1\",\"vpcid\":\"vpc-07075f78eb0d17c18\",\"eksclustername\":\"\",\"serviceimport\":false,\"serviceexport\":false,\"k8sservice\":\"front-end\",\"k8sservicenamespace\":\"default\",\"k8shttproutename\":\"front-end\",\"k8shttproutenamespace\":\"default\"},\"Type\":\"IP\",\"IsDeleted\":false,\"LatticeID\":\"\"}},\"k8s-front-end-primary-default\":{\"spec\":{\"Name\":\"k8s-front-end-primary-default\",\"config\":{\"port\":8079,\"protocol\":\"HTTP\",\"protocolversion\":\"HTTP1\",\"vpcid\":\"vpc-07075f78eb0d17c18\",\"eksclustername\":\"\",\"serviceimport\":false,\"serviceexport\":false,\"k8sservice\":\"front-end-primary\",\"k8sservicenamespace\":\"default\",\"k8shttproutename\":\"front-end\",\"k8shttproutenamespace\":\"default\"},\"Type\":\"IP\",\"IsDeleted\":false,\"LatticeID\":\"\"}}},\"AWS:VPCServiceNetwork::Targets\":{\"k8s-front-end-default\":{\"spec\":{\"name\":\"front-end\",\"namespace\":\"default\",\"targetgroupID\":\"\",\"targetIPlist\":[{\"targetID\":\"192.168.31.160\",\"port\":8079}]}},\"k8s-front-end-primary-default\":{\"spec\":{\"name\":\"front-end-primary\",\"namespace\":\"default\",\"targetgroupID\":\"\",\"targetIPlist\":[{\"targetID\":\"192.168.31.160\",\"port\":8079}]}}}}}": ""}
2023-02-19T03:04:08.709Z	DEBUG	events	Normal	{"object": {"kind":"HTTPRoute","namespace":"default","name":"front-end","uid":"8d487b60-9377-4c96-9a43-0db7860d9b21","apiVersion":"gateway.networking.k8s.io/v1alpha2","resourceVersion":"17100571"}, "reason": "Reconcile", "message": "Adding/Updating Reconcile"}
E0219 03:04:10.500588       1 request.go:539] Failed request: VpcLattice/CreateListener, Payload: {  DefaultAction: {    Forward: {      TargetGroups: [{          TargetGroupIdentifier: "tg-0ece3379e72118857",          Weight: 1        }]    }  },  Name: "front-end-default-0-http",  Port: 0,  Protocol: "HTTP",  ServiceIdentifier: "svc-044e0973b656236c3"}, Error: InvalidParameter: 1 validation error(s) found.
- minimum field value of 1, CreateListenerInput.Port.
I0219 03:04:10.500611       1 listener_manager.go:100] ############req creating listner ###########
I0219 03:04:10.500619       1 listener_manager.go:101] {
  DefaultAction: {
    Forward: {
      TargetGroups: [{
          TargetGroupIdentifier: "tg-0ece3379e72118857",
          Weight: 1
        }]
    }
  },
  Name: "front-end-default-0-http",
  Port: 0,
  Protocol: "HTTP",
  ServiceIdentifier: "svc-044e0973b656236c3"
}
I0219 03:04:10.500862       1 listener_manager.go:102] ############resp creating listner ###########
I0219 03:04:10.500869       1 listener_manager.go:103] create listener err :InvalidParameter: 1 validation error(s) found.
- minimum field value of 1, CreateListenerInput.Port.

I0219 03:04:10.500881       1 listener_manager.go:104] {

}
E0219 03:04:10.550989       1 request.go:539] Failed request: VpcLattice/ListRules, Payload: {  ListenerIdentifier: "",  ServiceIdentifier: "svc-044e0973b656236c3"}, Error: InvalidParameter: 1 validation error(s) found.
- minimum field size of 20, ListRulesInput.ListenerIdentifier.
E0219 03:04:10.551087       1 request.go:539] Failed request: VpcLattice/CreateRule, Payload: {  Action: {    Forward: {      TargetGroups: [{          TargetGroupIdentifier: "tg-0ece3379e72118857",          Weight: 100        },{          TargetGroupIdentifier: "tg-042eec348e3608dfd",          Weight: 0        }]    }  },  ListenerIdentifier: "",  Match: {    HttpMatch: {      PathMatch: {        Match: {          Prefix: "/"        }      }    }  },  Name: "k8s-1676775848-rule-1",  Priority: 0,  ServiceIdentifier: "svc-044e0973b656236c3"}, Error: InvalidParameter: 2 validation error(s) found.
- minimum field size of 20, CreateRuleInput.ListenerIdentifier.
- minimum field value of 1, CreateRuleInput.Priority.
I0219 03:04:10.551105       1 rule_manager.go:270] ############req creating rule ###########
I0219 03:04:10.551114       1 rule_manager.go:271] {
  Action: {
    Forward: {
      TargetGroups: [{
          TargetGroupIdentifier: "tg-0ece3379e72118857",
          Weight: 100
        },{
          TargetGroupIdentifier: "tg-042eec348e3608dfd",
          Weight: 0
        }]
    }
  },
  ListenerIdentifier: "",
  Match: {
    HttpMatch: {
      PathMatch: {
        Match: {
          Prefix: "/"
        }
      }
    }
  },
  Name: "k8s-1676775848-rule-1",
  Priority: 0,
  ServiceIdentifier: "svc-044e0973b656236c3"
}
I0219 03:04:10.551170       1 rule_manager.go:272] ############resp creating rule ###########, err: InvalidParameter: 2 validation error(s) found.
- minimum field size of 20, CreateRuleInput.ListenerIdentifier.
- minimum field value of 1, CreateRuleInput.Priority.
 
I0219 03:04:10.551187       1 rule_manager.go:273] {

}
2023-02-19T03:04:10.551Z	DEBUG	events	Normal	{"object": {"kind":"HTTPRoute","namespace":"default","name":"front-end","uid":"8d487b60-9377-4c96-9a43-0db7860d9b21","apiVersion":"gateway.networking.k8s.io/v1alpha2","resourceVersion":"17100571"}, "reason": "Retry-Reconcile", "message": "retry reconcile..."}

Audit error handling throughout the codebase

Followup from #60 (comment).

There are a number of cases where only the NOT_FOUND error case is the only one that is handled. We should go through the code base and make sure that the logical flows handle transient errors, not found errors, and other cases. This can result in resource leakage and code correctness issues.

Vanity DNS support

Today, after HTTPRoute CREATE reconcile, it gets an long lattice DNS name, This PR allows K8S user to specify a vanity name for the service. e.g.

the HTTPRoute name is the vanity name
or the "hostname" value of HTTPRoute is the vanity name

 kubectl get httproute -o yaml
apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1alpha2
  kind: HTTPRoute
...
status:
    parents:
    - conditions:
      - lastTransitionTime: "2022-10-31T02:12:15Z"
        message: 'DNS Name: oct30-inventory-default-065bc76ad6f66c05e.7d67968.lattice-svcs-prod.us-west-2.on.aws'
        reason: Reconciled
        status: "True"
        type: httproute

Update controller replicas to 2

c. Kubernetes best practices recommends running minimum of 2 replicas with leader election implemented. I do see Application networking controller already implementing leader election, https://github.com/aws/aws-application-networking-k8s/search?q=leader. Hence we should be able to update replicas to 2 (Must do).

Fix fleaky unit test "Test_RuleModelBuild ()"

This following test failed randomly from time to time

FAIL: Test_RuleModelBuild (0.00s)
    model_build_rule_test.go:274: 
        	Error Trace:	model_build_rule_test.go:274
        	Error:      	Not equal: 
        	            	expected: "rule-2"
        	            	actual  : "rule-1"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-rule-2
        	            	+rule-1
        	Test:       	Test_RuleModelBuild
    model_build_rule_test.go:283: 
        	Error Trace:	model_build_rule_test.go:283
        	Error:      	Not equal: 
        	            	expected: "targetgroup2"
        	            	actual  : "targetgroup1"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1,2 @@
        	            	-(v1alpha2.ObjectName) (len=12) "targetgroup2"
        	            	+(v1alpha2.ObjectName) (len=12) "targetgroup1"
        	            	 
        	Test:       	Test_RuleModelBuild
    model_build_rule_test.go:291: 
        	Error Trace:	model_build_rule_test.go:291
        	Error:      	Not equal: 
        	            	expected: true
        	            	actual  : false
        	Test:       	Test_RuleModelBuild
    model_build_rule_test.go:274: 
        	Error Trace:	model_build_rule_test.go:274
        	Error:      	Not equal: 
        	            	expected: "rule-1"
        	            	actual  : "rule-2"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-rule-1
        	            	+rule-2
        	Test:       	Test_RuleModelBuild
    model_build_rule_test.go:283: 
        	Error Trace:	model_build_rule_test.go:283
        	Error:      	Not equal: 
        	            	expected: "targetgroup1"
        	            	actual  : "targetgroup2"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1,2 @@
        	            	-(v1alpha2.ObjectName) (len=12) "targetgroup1"
        	            	+(v1alpha2.ObjectName) (len=12) "targetgroup2"
        	            	 
        	Test:       	Test_RuleModelBuild
    model_build_rule_test.go:289: 
        	Error Trace:	model_build_rule_test.go:289
        	Error:      	Not equal: 
        	            	expected: false
        	            	actual  : true
        	Test:       	Test_RuleModelBuild
stack &{{ns1 export1} map[{0x13b0be0 k8s-export1-ns1}:0xc00034ea80 {0x13b0c60 k8s-export1-ns1}:0xc0009f0e80] 0xc0004edd20} tg &{{0xc0004a0000 AWS:VPCServiceNetwork::TargetGroup k8s-export1-ns1} {k8s-export1-ns1 {0 HTTP HTTP1 vpc-xxxx  false} IP false } <nil>} err <nil>
stack &{{ns1 export2} map[] 0xc000a21c40} tg <nil> err proto: integer overflow
stack &{{ns1 export3} map[] 0xc000ab46e0} tg <nil> err proto: integer overflow
stack &{{ns1 export4} map[{0x13b0be0 k8s-export4-ns1}:0xc00034e000] 0xc000b28280} tg &{{0xc000afa0c0 AWS:VPCServiceNetwork::TargetGroup k8s-export4-ns1} {k8s-export4-ns1 {0 HTTP HTTP1 vpc-xxxx  false} IP true } <nil>} err <nil>
create K8S service {{ } {service1-tg1  ns11    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] []  []} {[] map[]  []  []   []   0 false nil [] <nil> <nil> <nil> <nil>} {{[]} []}}
httpBacndendRef Service
--dsTG {{k8s-service1-tg1-ns11 false}   []  false true}
create K8S service {{ } {service2-tg1  ns21    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] []  []} {[] map[]  []  []   []   0 false nil [] <nil> <nil> <nil> <nil>} {{[]} []}}
httpBacndendRef Service
--task.tgByResID[tgName] &{{0xc00079d180 AWS:VPCServiceNetwork::TargetGroup k8s-service2-tg1-ns21} {k8s-service2-tg1-ns21 {0 HTTP HTTP1 vpc-xxxx  false} IP true } <nil>} 
err <nil>
httpBacndendRef ServiceImport
dsTG {{k8s-service1-tg2-tg1-ns1 true}   []  false false}
err <nil>
httpBacndendRef ServiceImport
dsTG {{k8s-service1-tg2-tg1-ns1 true}   []  false false}
t.latticeTargets &{{0xc0003bbd40 AWS:VPCServiceNetwork::Targets k8s-export1-ns1} {export1 ns1  [{10.10.1.1 8675} {10.10.1.1 309} {10.10.2.2 8675} {10.10.2.2 309}]}} 
t.latticeTargets &{{0xc000c97f40 AWS:VPCServiceNetwork::Targets k8s-export5-ns1} {export5 ns1  [{10.10.1.1 8675} {10.10.1.1 309} {10.10.2.2 8675} {10.10.2.2 309}]}} 
FAIL
coverage: 75.3% of statements
FAIL	github.com/aws/aws-application-networking-k8s/pkg/gateway	0.043s

Enhance controller to allow a single HTTPbackend ref to have targets across multiple clusters

Use Case

Requirement

Customer want to using one single HTTPbackend ref and dynamically scale out targets across multiple cluster(s)

Why

Customer want to automatically scale-out across multiple cluster(s) and want lattice to load balancing them.

Proposal

Define one backendRef and cluster splits traffic across multiple service names.

all cluster(s) define same K8S service name e.g. foo
all cluster(s) define same K8S serviceexport name foo
HTTPbackend ref just point to serviceimport name foo

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: foo
spec:
  parentRefs:
  - name: my-hotel
    sectionName: http 
  rules:
  - backendRefs:  
    - name: foo
      kind: serviceimport
      port: 80

each cluster will have

apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport
metadata:
  name: foo
  annotations:
          multicluster.x-k8s.io/federation: "amazon-vpc-lattice"

Change k8's namespace used Lattice controller

Change namespace name from "system" to "aws-lattice-system" for Lattice controller.

Enhance DNS name discovery

Today, the DNS name is in message

kubectl get httproute -o yaml
apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1alpha2
  kind: HTTPRoute
...

  status:
    parents:
    - conditions:
      - lastTransitionTime: "2022-11-07T21:32:40Z"
        message: 'DNS Name: oct25-parking-default-022170a62d6205d58.7d67968.vpc-service-network-svcs.us-west-2.amazonaws.com'
        reason: Reconciled
        status: "True"
        type: httproute
      controllerName: application-networking.k8s.aws/gateway-api-controller
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: oct25-my-hotel
...

This should be enhanced

use annotation
or part of "HOSTNAMES"

kubectl get httproute
NAME            HOSTNAMES   AGE
oct25-parking               4m43s

Need to support named `targetPort`

The lattice controller does not allow you to use a named (i.e. string) targetPort in your kubernetes service resource, and have to use pod port number (i.e. int) directly - we use named ports a lot as they give a level of abstraction, so would like to see this supported:

works:

ports:

name: http
protocol: TCP
port: 80
targetPort: 80

does not work:

ports:

name: http
protocol: TCP
port: 80
targetPort: http

Controller is not following `HTTPRouteRule` spec, if no rule is matched a 404 must be returned

When deploying the examples/rate-route-path.yaml, two rules are specified:

  rules:
  - backendRefs:  
    - name: parking
      kind: Service
      port: 8090
    matches:
    - path:
        type: PathPrefix
        value: /parking
  - backendRefs:
    - name: review
      kind: Service
      port: 8090
    matches:
    - path:
        type: PathPrefix
        value: /review

however performing a request to / results in a route to the parking service.

According to the HTTPRouteRule spec:

When no rules matching a request have been successfully attached to the parent a request is coming from, a HTTP 404 status code MUST be returned.

Looking at the Lattice Config it's creating a default action to send to the first service in the rule list which is not compliant with the spec and not what's expected based upon the HTTPRoute rule definition above.

Issues when using `non` default namespace for HTTPRoute

cross-namespace routing seems not working. Also seems can not deploy HttpRoute to any namespace other than default

[bug] controller are not reconcile updated targets if it referenced by serviceexport

Today, controller creates a lattice targetgroup for each serviceexport . And it registers endpoints of the k8s service which have same name as serviceexport to lattice targetgroup.

We have seen a bug that when scaling up and down the matching k8s service endpoints (the replicas), the controller is not updating the latest endpoint to lattice target group

To workaround, restart controller and controller will register the up-to-date targets (k8s endpoints) to lattiice target group

`iam:CreateServiceLinkedRole` not needed but included in recommended-inline-policy

The gateway controller never calls iam:CreateServiceLinkedRole:

https://github.com/aws/aws-application-networking-k8s/search?q=ServiceLink

and yet iam:CreateServiceLinkedRole is listed in the recommended-inline-policy.json file.

This line should be removed from the recommended-inline-policy.json file if the controller is not actually calling that API.

[code cleanup] Need to use kube style

for example

use

kube style for this is typically to use v1alpha1 or mcsv1alpha1` if there's a conflict. for example, this code

EKS and Lattice configuration reconciliation

Create Kubernetes service in EKS, add the service to Lattice using gateway and httproute which only have http listener configured
When Lattice service is operational for http configure an additional listener via the AWS console for https
Lattice service will now be available via both http and https, be it for a limited time
EKS configuration will overwrite console configuration and the https listener will disappear, somewhere between 10-30 minutes

Question: should configurations which belong in EKS be greyed out and prohibited from being changed in the AWS console?

Document how to block pod-to-pod direct communication except traffic to and from lattice fleet

We need to add document on how to prevent following if customer desires to block them

"
I can bypass any Service level IAM auth policies within a cluster by calling the service "direct" using the kube dns host (app.namespace) rather than the app’s lattice host, as the request does not go via lattice so is not subject to its rules. Are there any thoughts here? As mentioned in the other queries it would be great if kube users just use kube dns styles hosts everywhere and the platform transparently routes them through lattice
"

Lattice listener not listing rules for all the paths listed in the HTTPRoute resource

I used the HTTPRoute below:

---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: frontend
  namespace: default
spec:
  parentRefs:
  - name: eks-store-network
    sectionName: http 
    namespace: default
  rules:
  - backendRefs:  
    - name: frontend-svc
      kind: Service
      port: 3000
      namespace: default
    matches:
    - path:
        type: PathPrefix
        value: /summary    
    - path:
        type: PathPrefix
        value: /popular

I expected the corresponding Listener created under VPC Lattice to have separate rules for "/summary" and "/popular". But, I see the listener has a default rule and just one other non-default rule that matched "/summary". There no rule specific to the path "/popular"

aws lattice --endpoint $ENDPOINT list-rules --listener-identifier $LISTENER_ID --service-identifier $SERVICE_ID     
{
    "items": [
        {
            "arn": "arn:aws:vpc-lattice:us-west-2:937351930975:service/svc-0f521522304b8db18/listener/listener-0a69653897eba06e7/rule/rule-0565e9f746a9fd13a",
            "createdAt": "2022-12-20T00:51:12.734000+00:00",
            "id": "rule-0565e9f746a9fd13a",
            "isDefault": true,
            "lastUpdatedAt": "2022-12-20T00:51:12.734000+00:00",
            "name": "default"
        },
        {
            "arn": "arn:aws:vpc-lattice:us-west-2:937351930975:service/svc-0f521522304b8db18/listener/listener-0a69653897eba06e7/rule/rule-0ebabd8dba993eaa4",
            "createdAt": "2022-12-20T03:01:32.023000+00:00",
            "id": "rule-0ebabd8dba993eaa4",
            "isDefault": false,
            "lastUpdatedAt": "2022-12-20T03:01:33.689000+00:00",
            "name": "k8s-1671505290-rule-1"
        }
    ]
}

RULE_ID="rule-0ebabd8dba993eaa4"         
aws lattice --endpoint $ENDPOINT get-rule --rule-identifier $RULE_ID --listener-identifier $LISTENER_ID --service-identifier $SERVICE_ID

{
    "action": {
        "forward": {
            "targetGroups": [
                {
                    "targetGroupIdentifier": "tg-07ee612f7751f2293",
                    "weight": 1
                }
            ]
        }
    },
    "arn": "arn:aws:vpc-lattice:us-west-2:937351930975:service/svc-0f521522304b8db18/listener/listener-0a69653897eba06e7/rule/rule-0ebabd8dba993eaa4",
    "createdAt": "2022-12-20T03:01:32.023000+00:00",
    "id": "rule-0ebabd8dba993eaa4",
    "lastUpdatedAt": "2022-12-20T03:01:33.689000+00:00",
    "match": {
        "httpMatch": {
            "headerMatches": [],
            "pathMatch": {
                "caseSensitive": true,
                "match": {
                    "prefix": "/summary"
                }
            }
        }
    },
    "name": "k8s-1671505290-rule-1",
    "priority": 1
}

Gateway API conformance test

We need to run against upstream gateway API conformance test and fix the gaps if any.

Need End-to-End integration Tests

Here is manual end-to-end test steps

Need to write go test program to configure these lattice resources.

Upgrade to go v1.19

Simple task to warm up with gh workflows.

[documentation] on K8S Gateway name

The association between the K8s Gateway name and the Lattice Service Network name should be explicitly mentioned somewhere in the documentation. It will not be uncommon to deploy the Gateway to a VPC that is already part of a Service Network.

[Spike] Investigate blocking VPC association when adding a shared service to a service network

By default the controller adds the VPC when the gateway is created. We would need some sort of flag to disable the auto association and just register the service instead. Can you please investigate and determine the effort and impact of this change.

Note: this is from customer feedback

Enhance gateway-api-controller to allow a HTTPRoute to associated to multiple gateway(s)

Today, lattice allow a single lattice service to be registered to multiple lattice service networks. And K8S gateway APIs also allows a single HTTPRoute to have multiple parentRef(gateways).

Here are the enhancements

allow K8S user to define additional gateway(s) and their associated VPC(s)
allow httpRoute to associate to multiple gateway(s)

controller support to allow k8S user to attach IAM policy to HTTPRoute

Add support in gateway-api-controller to allow user to attach IAM policy CRD to gateway HTTPRoute object.

Support gateway GRPCroute object

Add support for gateway API gRPC routing

Lattice target groups are not removed upon deletion of HTTPRoute

When I delete the HTTPRoute resources, I noticed that the associated Lattice Target Groups are not removed. They stick around with zombie IP addresses of targets that have been deleted. They do, get reused when the same HTTPRoute resource is redeployed. Is this done for optimization reasons? The Lattice Service, OTOH, does get deleted when you delete the HTTPRoute resource

Add environment variablel to specify lattice CP endpoint

Today, the endpoint is hardcode pointing to pdx prod, We need add environment variable which allow customer to specify lattice endpoint

Make docker-build faild

Not able to run make docker-build

make docker-build
go test ./... -coverprofile coverage.out
?   	github.com/aws/aws-application-networking-k8s	[no test files]
ok  	github.com/aws/aws-application-networking-k8s/controllers	6.767s	coverage: 0.0% of statements
?   	github.com/aws/aws-application-networking-k8s/controllers/eventhandlers	[no test files]
?   	github.com/aws/aws-application-networking-k8s/mocks/controller-runtime/client	[no test files]
?   	github.com/aws/aws-application-networking-k8s/pkg/aws	[no test files]
ok  	github.com/aws/aws-application-networking-k8s/pkg/aws/services	0.008s	coverage: 3.1% of statements
?   	github.com/aws/aws-application-networking-k8s/pkg/config	[no test files]
?   	github.com/aws/aws-application-networking-k8s/pkg/deploy	[no test files]
ok  	github.com/aws/aws-application-networking-k8s/pkg/deploy/lattice	0.039s	coverage: 86.7% of statements
ok  	github.com/aws/aws-application-networking-k8s/pkg/gateway	0.039s	coverage: 75.3% of statements
?   	github.com/aws/aws-application-networking-k8s/pkg/k8s	[no test files]
ok  	github.com/aws/aws-application-networking-k8s/pkg/latticestore	0.004s	coverage: 75.3% of statements
ok  	github.com/aws/aws-application-networking-k8s/pkg/model/core	0.004s	coverage: 40.9% of statements
ok  	github.com/aws/aws-application-networking-k8s/pkg/model/core/graph	0.003s	coverage: 17.2% of statements
?   	github.com/aws/aws-application-networking-k8s/pkg/model/lattice	[no test files]
?   	github.com/aws/aws-application-networking-k8s/pkg/runtime	[no test files]
?   	github.com/aws/aws-application-networking-k8s/pkg/utils	[no test files]
ok  	github.com/aws/aws-application-networking-k8s/pkg/utils/log	0.003s	coverage: 0.0% of statements [no tests to run]
?   	github.com/aws/aws-application-networking-k8s/pkg/utils/retry	[no test files]
?   	github.com/aws/aws-application-networking-k8s/pkg/utils/ttime	[no test files]
sudo docker build -t controller:latest .
Sending build context to Docker daemon  397.5MB
Step 1/17 : FROM golang:1.16 as builder
 ---> 972d8c0bc0fc
Step 2/17 : WORKDIR /workspace
 ---> Using cache
 ---> de9c45840855
Step 3/17 : COPY go.mod go.mod
 ---> Using cache
 ---> c8a741a309de
Step 4/17 : COPY go.sum go.sum
 ---> Using cache
 ---> 40fe0a93bd9a
Step 5/17 : COPY scripts/aws_sdk_model_override/aws-sdk-go/go.mod scripts/aws_sdk_model_override/aws-sdk-go/go.mod
 ---> Using cache
 ---> c6e62c1e59cc
Step 6/17 : COPY scripts/aws_sdk_model_override/aws-sdk-go/go.sum scripts/aws_sdk_model_override/aws-sdk-go/go.sum
 ---> Using cache
 ---> 066d15f54b9c
Step 7/17 : RUN    GOPROXY=direct go mod download
 ---> Using cache
 ---> adac7472f4a5
Step 8/17 : COPY main.go main.go
 ---> Using cache
 ---> b1f15516e470
Step 9/17 : COPY pkg/ pkg/
 ---> Using cache
 ---> b6d9b60fb91f
Step 10/17 : COPY controllers/ controllers/
 ---> Using cache
 ---> 1b0dcbef08f1
Step 11/17 : COPY scripts scripts
 ---> 6b3ecb1b6c15
Step 12/17 : RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go
 ---> Running in 805cfa369dc4
# golang.org/x/sys/unix
/go/pkg/mod/golang.org/x/[email protected]/unix/syscall.go:83:16: undefined: unsafe.Slice
/go/pkg/mod/golang.org/x/[email protected]/unix/syscall_linux.go:2255:9: undefined: unsafe.Slice
/go/pkg/mod/golang.org/x/[email protected]/unix/syscall_unix.go:118:7: undefined: unsafe.Slice
/go/pkg/mod/golang.org/x/[email protected]/unix/sysvshm_unix.go:33:7: undefined: unsafe.Slice
note: module requires Go 1.17
The command '/bin/sh -c CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go' returned a non-zero code: 2
make: *** [docker-build] Error 2

Use lower case for crd/* instead of uppercase CRD/*

CR comments from #27

I'd recommend lower casing CRDs -- not all operating systems respect folder capitalizaiton.

Update Gateway IP address field

To be gateway API compliant, we need to provide an IP address to Gateway Object.

We can add 1st HTTPRoute IP to gateway.

Controller patch error

2022-11-15T00:11:14.674Z	DEBUG	events	Normal	{"object": {"kind":"HTTPRoute","namespace":"default","name":"nov14-parking","uid":"b1ab2716-9609-43c6-ac6a-b96f6cc8bf62","apiVersion":"gateway.networking.k8s.io/v1alpha2","resourceVersion":"33667"}, "reason": "Retry-Reconcile", "message": "retry reconcile..."}
E1115 00:11:14.676502       1 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"nov14-parking.17279990d5e9caa6", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"HTTPRoute", Namespace:"default", Name:"nov14-parking", UID:"b1ab2716-9609-43c6-ac6a-b96f6cc8bf62", APIVersion:"gateway.networking.k8s.io/v1alpha2", ResourceVersion:"33667", FieldPath:""}, Reason:"Retry-Reconcile", Message:"retry reconcile...", Source:v1.EventSource{Component:"httproute", Host:""}, FirstTimestamp:time.Date(2022, time.November, 15, 0, 10, 34, 307267238, time.Local), LastTimestamp:time.Date(2022, time.November, 15, 0, 11, 14, 674453890, time.Local), Count:4, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "nov14-parking.17279990d5e9caa6" is forbidden: User "system:serviceaccount:system:gateway-api-controller" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)

Garbage collect lattice target groups when HTTProute is deleted

Today, when Controller reconcile HTTPRoute CREATE, it automatically creates lattice target group references by HTTPRoute backend Ref.

But when controller reconcile a HTTPRoute DELETE, it left those target group around.

Create serviceexport fails if already present

I have not tried this to determine whether serviceexport and serviceimport are idompotent at lattice data plane level (as these are cluster resources). Can the same service simultaneously be imported or exported in two distinct clusters? What is the behavior if so.

The controller creates backendRef (Target groups) without creating HttpRoute (Service)

The controller doesn't fail safe and allows creation of TargetGroups but fails creating service when gateway doesn't exist yet. But the delete of route fails with not able to find service and hence TargetGroups are left behind. The reconciliation will fail eventually as TargetGroups already present.

Unable to delete HTTPRoute object

I've a failed HTTPRoute object (crystal) in default namespace due to "Failed build model due to RETRY". When I tried to delete the object, kubectl command just hangs forever. Lattice is unable to release the finalizer and see the following messages in the controller logs.

Retrying Reconcile after 20 seconds ...
2023-01-24T19:21:11.095Z INFO controller.httproute HTTPRouteReconciler {"reconciler group": "gateway.networking.k8s.io", "reconciler kind": "HTTPRoute", "name": "crystal", "namespace": "default"}
2023-01-24T19:21:11.095Z INFO controller.httproute Deleting {"reconciler group": "gateway.networking.k8s.io", "reconciler kind": "HTTPRoute", "name": "crystal", "namespace": "default"}
2023-01-24T19:21:11.096Z DEBUG events Normal {"object": {"kind":"HTTPRoute","namespace":"default","name":"crystal","uid":"275832ce-830e-469b-84f0-777027e351d9","apiVersion":"gateway.networking.k8s.io/v1alpha2","resourceVersion":"7605439"}, "reason": "Reconcile", "message": "Deleting Reconcile"}
2023-01-24T19:21:11.096Z DEBUG events Warning {"object": {"kind":"HTTPRoute","namespace":"default","name":"crystal","uid":"275832ce-830e-469b-84f0-777027e351d9","apiVersion":"gateway.networking.k8s.io/v1alpha2","resourceVersion":"7605439"}, "reason": "FailedBuildModel", "message": "Failed build model due to RETRY"}
2023-01-24T19:21:31.096Z INFO controller.httproute HTTPRouteReconciler {"reconciler group": "gateway.networking.k8s.io", "reconciler kind": "HTTPRoute", "name": "crystal", "namespace": "default"}
2023-01-24T19:21:31.097Z INFO controller.httproute Deleting {"reconciler group": "gateway.networking.k8s.io", "reconciler kind": "HTTPRoute", "name": "crystal", "namespace": "default"}

Events:
Type Reason Age From Message

Warning FailedBuildModel 53m (x12 over 57m) httproute Failed build model due to RETRY
Normal Reconcile 12m (x137 over 57m) httproute Adding/Updating Reconcile
Normal Reconcile 2m18s (x24 over 9m49s) httproute Deleting Reconcile

Delete of gateway stuck and fails with API throttle issues finally

Deleting a gateway created under a non default namespace fails and the kubectl delete command is stuck in delete mode. And after sometime, we do see throttle errors in the gateway controller log.

E0222 03:54:25.069782 1 request.go:539] Failed request: VpcLattice/DeleteServiceNetwork, Payload: { ServiceNetworkIdentifier: "sn-06f18144e6970102b"}, Error: InternalServerException: The service has encountered an internal error. We apologize for the inconvenience.
{
RespMetadata: {
StatusCode: 500,
RequestID: "17fa20a6-a97e-46ad-99af-4f7c8fdf904f"
},
Message_: "The service has encountered an internal error. We apologize for the inconvenience."
}
I0222 03:54:25.069863 1 gateway_controller.go:165] Failed to cleanup gw &{{Gateway gateway.networking.k8s.io/v1alpha2} {my-hotel my-hotel 699c3d01-ac69-4abc-9712-91f6bc37ec40 152207 2 2023-02-22 02:50:50 +0000 UTC 2023-02-22 03:53:41 +0000 UTC 0xc000cbc080 map[] map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"Gateway","metadata":{"annotations":{},"name":"my-hotel","namespace":"my-hotel"},"spec":{"gatewayClassName":"amazon-vpc-lattice","listeners":[{"name":"http","port":80,"protocol":"HTTP"}]}}
] [] [gateway.k8s.aws/resources] [{kubectl-client-side-apply Update gateway.networking.k8s.io/v1alpha2 2023-02-22 02:50:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:gatewayClassName":{},"f:listeners":{".":{},"k:{"name":"http"}":{".":{},"f:allowedRoutes":{".":{},"f:namespaces":{".":{},"f:from":{}}},"f:name":{},"f:port":{},"f:protocol":{}}}}} } {manager Update gateway.networking.k8s.io/v1alpha2 2023-02-22 03:14:34 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:"gateway.k8s.aws/resources"":{}}}} } {manager Update gateway.networking.k8s.io/v1alpha2 2023-02-22 03:14:55 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{"type":"Scheduled"}":{"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status}]} {amazon-vpc-lattice [{http 80 HTTP 0xc0008ee480}] []} {[] [{Scheduled True 0 2023-02-22 03:14:56 +0000 UTC Reconciled aws-gateway-arn: arn:aws:vpc-lattice:us-west-2:615847537300:servicenetwork/sn-06f18144e6970102b}] []}}, err LATTICE_RETRY

Retrying Reconcile after 20 seconds ...
2023-02-22T03:54:45.070Z INFO controller.gateway GatewayReconciler {"reconciler group": "gateway.networking.k8s.io", "reconciler kind": "Gateway", "name": "my-hotel", "namespace": "my-hotel"}
2023-02-22T03:54:45.070Z INFO controller.gateway Successfully built model {"reconciler group": "gateway.networking.k8s.io", "reconciler kind": "Gateway", "name": "my-hotel", "namespace": "my-hotel", "{"id":"my-hotel/my-hotel","resources":{"AWS::VPCServiceNetwork::ServiceNetwork":{"ServiceNetwork":{"spec":{"name":"my-hotel","account":"615847537300","IsDeleted":true}}}}}": ""}
E0222 03:54:46.473407 1 request.go:539] Failed request: VpcLattice/DeleteServiceNetwork, Payload: { ServiceNetworkIdentifier: "sn-06f18144e6970102b"}, Error: InternalServerException: The service has encountered an internal error. We apologize for the inconvenience.
{
RespMetadata: {
StatusCode: 500,
RequestID: "827d9329-24fd-4b81-8a30-5a55a695db9d"
},
Message_: "The service has encountered an internal error. We apologize for the inconvenience."
}
Retrying Reconcile after 20 seconds ...
I0222 03:54:46.473478 1 gateway_controller.go:165] Failed to cleanup gw &{{Gateway gateway.networking.k8s.io/v1alpha2} {my-hotel my-hotel 699c3d01-ac69-4abc-9712-91f6bc37ec40 152207 2 2023-02-22 02:50:50 +0000 UTC 2023-02-22 03:53:41 +0000 UTC 0xc000cbe968 map[] map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"Gateway","metadata":{"annotations":{},"name":"my-hotel","namespace":"my-hotel"},"spec":{"gatewayClassName":"amazon-vpc-lattice","listeners":[{"name":"http","port":80,"protocol":"HTTP"}]}}
] [] [gateway.k8s.aws/resources] [{kubectl-client-side-apply Update gateway.networking.k8s.io/v1alpha2 2023-02-22 02:50:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:gatewayClassName":{},"f:listeners":{".":{},"k:{"name":"http"}":{".":{},"f:allowedRoutes":{".":{},"f:namespaces":{".":{},"f:from":{}}},"f:name":{},"f:port":{},"f:protocol":{}}}}} } {manager Update gateway.networking.k8s.io/v1alpha2 2023-02-22 03:14:34 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:"gateway.k8s.aws/resources"":{}}}} } {manager Update gateway.networking.k8s.io/v1alpha2 2023-02-22 03:14:55 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{"type":"Scheduled"}":{"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status}]} {amazon-vpc-lattice [{http 80 HTTP 0xc0006027c0}] []} {[] [{Scheduled True 0 2023-02-22 03:14:56 +0000 UTC Reconciled aws-gateway-arn: arn:aws:vpc-lattice:us-west-2:615847537300:servicenetwork/sn-06f18144e6970102b}] []}}, err LATTICE_RETRY
2023-02-22T03:55:06.474Z INFO controller.gateway GatewayReconciler {"reconciler group": "gateway.networking.k8s.io", "reconciler kind": "Gateway", "name": "my-hotel", "namespace": "my-hotel"}
2023-02-22T03:55:06.474Z INFO controller.gateway Successfully built model {"reconciler group": "gateway.networking.k8s.io", "reconciler kind": "Gateway", "name": "my-hotel", "namespace": "my-hotel", "{"id":"my-hotel/my-hotel","resources":{"AWS::VPCServiceNetwork::ServiceNetwork":{"ServiceNetwork":{"spec":{"name":"my-hotel","account":"615847537300","IsDeleted":true}}}}}": ""}
E0222 03:55:08.081835 1 request.go:539] Failed request: VpcLattice/DeleteServiceNetwork, Payload: { ServiceNetworkIdentifier: "sn-06f18144e6970102b"}, Error: InternalServerException: The service has encountered an internal error. We apologize for the inconvenience.
{
RespMetadata: {
StatusCode: 500,
RequestID: "c090ca16-69e6-46a7-9dad-0d77b52ea72f"
},
Message_: "The service has encountered an internal error. We apologize for the inconvenience."
}
] [] [gateway.k8s.aws/resources] [{kubectl-client-side-apply Update gateway.networking.k8s.io/v1alpha2 2023-02-22 02:50:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:gatewayClassName":{},"f:listeners":{".":{},"k:{"name":"http"}":{".":{},"f:allowedRoutes":{".":{},"f:namespaces":{".":{},"f:from":{}}},"f:name":{},"f:port":{},"f:protocol":{}}}}} } {manager Update gateway.networking.k8s.io/v1alpha2 2023-02-22 03:14:34 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:"gateway.k8s.aws/resources"":{}}}} } {manager Update gateway.networking.k8s.io/v1alpha2 2023-02-22 03:14:55 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{"type":"Scheduled"}":{"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status}]} {amazon-vpc-lattice [{http 80 HTTP 0xc000c3c020}] []} {[] [{Scheduled True 0 2023-02-22 03:14:56 +0000 UTC Reconciled aws-gateway-arn: arn:aws:vpc-lattice:us-west-2:615847537300:servicenetwork/sn-06f18144e6970102b}] []}}, err LATTICE_RETRY
Retrying Reconcile after 20 seconds ...
2023-02-22T04:02:00.689Z INFO controller.gateway GatewayReconciler {"reconciler group": "gateway.networking.k8s.io", "reconciler kind": "Gateway", "name": "my-hotel", "namespace": "my-hotel"}
2023-02-22T04:02:00.689Z INFO controller.gateway Successfully built model {"reconciler group": "gateway.networking.k8s.io", "reconciler kind": "Gateway", "name": "my-hotel", "namespace": "my-hotel", "{"id":"my-hotel/my-hotel","resources":{"AWS::VPCServiceNetwork::ServiceNetwork":{"ServiceNetwork":{"spec":{"name":"my-hotel","account":"615847537300","IsDeleted":true}}}}}": ""}
E0222 04:02:02.194057 1 request.go:539] Failed request: VpcLattice/DeleteServiceNetwork, Payload: { ServiceNetworkIdentifier: "sn-06f18144e6970102b"}, Error: ThrottlingException: Too Many Requests
{
Retrying Reconcile after 20 seconds ...
RespMetadata: {
StatusCode: 429,
RequestID: "03639a82-4c14-41da-9932-a78632af8987"
},
Message_: "Too Many Requests"] [] [gateway.k8s.aws/resources] [{kubectl-client-side-apply Update gateway.networking.k8s.io/v1alpha2 2023-02-22 02:50:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:gatewayClassName":{},"f:listeners":{".":{},"k:{"name":"http"}":{".":{},"f:allowedRoutes":{".":{},"f:namespaces":{".":{},"f:from":{}}},"f:name":{},"f:port":{},"f:protocol":{}}}}} } {manager Update gateway.networking.k8s.io/v1alpha2 2023-02-22 03:14:34 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:"gateway.k8s.aws/resources"":{}}}} } {manager Update gateway.networking.k8s.io/v1alpha2 2023-02-22 03:14:55 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{"type":"Scheduled"}":{"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status}]} {amazon-vpc-lattice [{http 80 HTTP 0xc000c3c020}] []} {[] [{Scheduled True 0 2023-02-22 03:14:56 +0000 UTC Reconciled aws-gateway-arn: arn:aws:vpc-lattice:us-west-2:615847537300:servicenetwork/sn-06f18144e6970102b}] []}}, err LATTICE_RETRY
Retrying Reconcile after 20 seconds ...
2023-02-22T04:02:00.689Z INFO controller.gateway GatewayReconciler {"reconciler group": "gateway.networking.k8s.io", "reconciler kind": "Gateway", "name": "my-hotel", "namespace": "my-hotel"}
2023-02-22T04:02:00.689Z INFO controller.gateway Successfully built model {"reconciler group": "gateway.networking.k8s.io", "reconciler kind": "Gateway", "name": "my-hotel", "namespace": "my-hotel", "{"id":"my-hotel/my-hotel","resources":{"AWS::VPCServiceNetwork::ServiceNetwork":{"ServiceNetwork":{"spec":{"name":"my-hotel","account":"615847537300","IsDeleted":true}}}}}": ""}
E0222 04:02:02.194057 1 request.go:539] Failed request: VpcLattice/DeleteServiceNetwork, Payload: { ServiceNetworkIdentifier: "sn-06f18144e6970102b"}, Error: ThrottlingException: Too Many Requests
{
Retrying Reconcile after 20 seconds ...
RespMetadata: {
StatusCode: 429,
RequestID: "03639a82-4c14-41da-9932-a78632af8987"
},
Message_: "Too Many Requests"

Update error msgs when controller retry reconciling

As the route creation takes time, it is not very clear from the httproute status about the creation of the VPC resources and mercury data plane components. The gateway-controller logs keeps showing reconciling error. This is confusing to customers. Catch the error within controller code and show progress messages instead.

Need to update Gateway, HTTPRoute status.

Need to Update Gateway, HTTPRoute status when config is accepted

Today, controller only update HTTPRoute status after it finishes reconciling HTTPRoute to lattice control plane. It can takes multiple minutes before controller finishes reconciling. The controller needs to update status, so that customer knows controller has accept the HTTPRoute and is in the process of reconciling

Need to Populate the VPC lattice error return msg to corresponding K8S object, e.g. Gateway, HTTPRoute

for example,

can not delete gatway, since there are still httproute referencing this gateway
the rules of HTTPRoute is rejected by VPC lattice , maybe due to mis-configuration, or not supported by VPC-lattice

Need to handle 404 for all K8S API calls (even for local cache)

Set it to low priority, since we have not seen this during our testing. We will raise the priority if it happens.

Automate associating custom domain name with service

Today, customer needs manually configure route53, add a CNAME record to map custom domain name to a VPC lattice generated DNS name

Ideally, lattice controller can automate this process .

Canary traffic shifting does not work when using 1+ rules in HTTPRoute

This is related to #75 but wanted create this separately to call out the implications of not handling 1+ rules.
Canary shifting of traffic breaks due to the fact that Lattice does not pick up all the path-based rules.

The following is the HTTPRoute I used.
Here, the /summary endpoint returns different results for v1 and v2.

---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: backend-canary
  namespace: default
spec:
  parentRefs:
  - name: eks-store-network
    sectionName: http 
    namespace: default
  rules:
  - backendRefs:  
    - name: backend-v1-svc
      kind: Service
      port: 3000
      namespace: default
      weight: 25
    - name: backend-v2-svc
      kind: Service
      port: 3000
      namespace: default      
      weight: 75
    matches:
    - path:
        type: PathPrefix
        value: /popular
    - path:
        type: PathPrefix
        value: /summary

When I look at the listener using the following CLI

aws lattice --endpoint $ENDPOINT list-rules --listener-identifier $LISTENER_ID --service-identifier $SERVICE_ID

gives the following output

{
    "items": [
        {
            "arn": "arn:aws:vpc-lattice:us-west-2:937351930975:service/svc-070486ef14a0e8348/listener/listener-09d05a6365bd3e3c1/rule/rule-0913b01117b721751",
            "createdAt": "2023-02-22T05:02:49.110000+00:00",
            "id": "rule-0913b01117b721751",
            "isDefault": true,
            "lastUpdatedAt": "2023-02-22T05:02:49.110000+00:00",
            "name": "default"
        },
        {
            "arn": "arn:aws:vpc-lattice:us-west-2:937351930975:service/svc-070486ef14a0e8348/listener/listener-09d05a6365bd3e3c1/rule/rule-01de45c95a0bb8f8a",
            "createdAt": "2023-02-22T05:02:49.377000+00:00",
            "id": "rule-01de45c95a0bb8f8a",
            "isDefault": false,
            "lastUpdatedAt": "2023-02-22T05:02:51.621000+00:00",
            "name": "k8s-1677042167-rule-1"
        }
    ]
}

The first rule is the default that sends 100% the responses to the target group associated with v1 of the service.

RULE_ID=rule-0913b01117b721751 
aws lattice --endpoint $ENDPOINT get-rule --rule-identifier $RULE_ID --listener-identifier $LISTENER_ID --service-identifier $SERVICE_ID

{
    "action": {
        "forward": {
            "targetGroups": [
                {
                    "targetGroupIdentifier": "tg-07c009c6ed1ff42a3",
                    "weight": 1
                }
            ]
        }
    },
    "arn": "arn:aws:vpc-lattice:us-west-2:937351930975:service/svc-070486ef14a0e8348/listener/listener-09d05a6365bd3e3c1/rule/rule-0913b01117b721751",
    "createdAt": "2023-02-22T05:02:49.110000+00:00",
    "id": "rule-0913b01117b721751",
    "isDefault": true,
    "lastUpdatedAt": "2023-02-22T05:02:49.110000+00:00",
    "name": "default"
}

The second rule is the weighted one but it takes effect only if the request path is /popular.
As a result, requests to the endpoint /summary are all routed to v1 of the service.

RULE_ID=rule-01de45c95a0bb8f8a
aws lattice --endpoint $ENDPOINT get-rule --rule-identifier $RULE_ID --listener-identifier $LISTENER_ID --service-identifier $SERVICE_ID

{
    "action": {
        "forward": {
            "targetGroups": [
                {
                    "targetGroupIdentifier": "tg-07c009c6ed1ff42a3",
                    "weight": 25
                },
                {
                    "targetGroupIdentifier": "tg-07792a965fe50fda2",
                    "weight": 75
                }
            ]
        }
    },
    "arn": "arn:aws:vpc-lattice:us-west-2:937351930975:service/svc-070486ef14a0e8348/listener/listener-09d05a6365bd3e3c1/rule/rule-01de45c95a0bb8f8a",
    "createdAt": "2023-02-22T05:02:49.377000+00:00",
    "id": "rule-01de45c95a0bb8f8a",
    "lastUpdatedAt": "2023-02-22T05:02:51.621000+00:00",
    "match": {
        "httpMatch": {
            "headerMatches": [],
            "pathMatch": {
                "caseSensitive": true,
                "match": {
                    "prefix": "/popular"
                }
            }
        }
    },
    "name": "k8s-1677042167-rule-1",
    "priority": 1
}

Controller need to configure target group health check

Today, lattice controller does not configure any health check for target group. So, target group uses lattice default configuration. This will be problem and Pod stay in Unhealthy if. Pod does NOT respond to "/" HTTP GET

aws vpc-lattice get-target-group --target-group-identifier tg-028db3d23f0c48565 --endpoint-url=https://vpc-lattice.us-west-2.amazonaws.com
{
    "arn": "arn:aws:vpc-lattice:us-west-2:694065802095:targetgroup/tg-028db3d23f0c48565",
    "config": {
        "healthCheck": {
            "enabled": true,
            "healthCheckIntervalSeconds": 30,
            "healthCheckTimeoutSeconds": 5,
            "healthyThresholdCount": 5,
            "matcher": {
                "httpCode": "200"
            },
            "path": "/",
            "protocol": "HTTP",
            "unhealthyThresholdCount": 2
        },
        "port": 8090,
        "protocol": "HTTP",
        "protocolVersion": "HTTP1",
        "vpcIdentifier": "vpc-053e8d083fabe4780"
    },
    "createdAt": "2022-12-01T14:40:44.808000+00:00",
    "id": "tg-028db3d23f0c48565",
    "lastUpdatedAt": "2022-12-01T14:40:44.808000+00:00",
    "name": "k8s-nov28-parking-ver1-default",
    "serviceArns": [
        "arn:aws:vpc-lattice:us-west-2:694065802095:service/svc-086ea32d9c319060b"
    ],
    "status": "ACTIVE",
    "type": "IP"
}

Controller doesn't create a service when using namespace other than default in HTTPRoute

I used this manifest to create a HTTPRoute.

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: backend
  namespace: store
spec:
  parentRefs:
  - name: eks-store-network
    sectionName: http 
    namespace: store
  rules:
  - backendRefs:  
    - name: backend-svc
      kind: Service
      port: 3000
      namespace: store
    matches:
    - path:
        type: PathPrefix
        value: /popular
    - path:
        type: PathPrefix
        value: /summary

This does not lead to the creation of a Lattice Service. The logs from the controller are shown below. If the change the namespace to default, it works.

2023-02-22T02:31:03.477Z	INFO	controller.httproute	HTTPRouteReconciler	{"reconciler group": "gateway.networking.k8s.io", "reconciler kind": "HTTPRoute", "name": "backend", "namespace": "store"}
2023-02-22T02:31:03.477Z	INFO	controller.service	ServiceReconciler	{"reconciler group": "", "reconciler kind": "Service", "name": "backend-svc", "namespace": "store"}

There are no further logs that a indicate a service was created.

Add tags to all lattice resource created by controller

gateway --> lattice service networks: list of VPCs associated with this gateway
HTTPRoute --> lattice service: tag k8s httproute name, namespace
K8S service --> lattice target group, tag K8S service name, namespace and it is regular K8S service type and the HTTProute name that use it as backend
K8S serviceexport --> lattice target group, tag K8S serviceexport
K8S serviceimport --> reference a lattice target group, and tag K8S serviceimport, and HTTPRoute name that use it as backend

Update to v1beta1 of gateway-api

The external-dns supports v1beta1 version of gateway-api. We can’t test external-dns route53 integration until the controller code is updated to use v1beta1 of gateway-api.

Lattice fails to create 1+ HTTPRoutes that reference the same backend K8s service

Unable to deploy 1+ HTTPRoute resources that reference the same backend K8s service. This is a practical use case where one HTTPRoute will be associated with v1 of a backend service and another could be created with weighted target groups for B/G testing, referencing v1 & v2 of the same backend service. When I do this, Lattice attempts to reuse the Target Group that was previously created for the v1 service and fails with the message TargetGroup tg-XXX has been associated with other services

Here is the first HTTPRoute I used:

---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: frontend
  namespace: default
spec:
  parentRefs:
  - name: eks-store-network
    sectionName: http 
    namespace: default
  rules:
  - backendRefs:  
    - name: frontend-svc
      kind: Service
      port: 3000
      namespace: default
    matches:
    - path:
        type: PathPrefix
        value: /summary    
    - path:
        type: PathPrefix
        value: /popular

This create the following Target Group.

{
    "items": [
        {
            "arn": "arn:aws:vpc-lattice:us-west-2:937351930975:targetgroup/tg-07ee612f7751f2293",
            "createdAt": "2022-12-20T00:50:10.321000+00:00",
            "id": "tg-07ee612f7751f2293",
            "lastUpdatedAt": "2022-12-20T00:53:32.702000+00:00",
            "name": "k8s-frontend-svc-default",
            "port": 3000,
            "protocol": "HTTP",
            "serviceArns": [
                "arn:aws:vpc-lattice:us-west-2:937351930975:service/svc-0af88a98b19436982"
            ],
            "status": "ACTIVE",
            "type": "IP",
            "vpcIdentifier": "vpc-03e33c3a543ddcaee"
        }
    ]
}

Then, I try to deploy another HTTPRoute as follows:

---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: frontend-bg
  namespace: default
spec:
  parentRefs:
  - name: eks-store-network
    sectionName: http 
    namespace: default
  rules:
  - backendRefs:  
    - name: frontend-svc
      kind: Service
      port: 3000
      namespace: default
      weight: 25
    - name: frontend-v2-svc
      kind: Service
      port: 3000
      namespace: default      
      weight: 75
    matches:
    - path:
        type: PathPrefix
        value: /popular
    - path:
        type: PathPrefix
        value: /summary

Now, Lattice tries to use the same Target Group tg-07ee612f7751f2293 for the K8s service frontend-svc and fails with the message "TargetGroup tg-07ee612f7751f2293 has been associated with other services"