Summary
Add support for AWS Cloud Map service-discovery with App Mesh virtual-nodes when using aws-app-mesh-controller-for-k8s.
Motivation
AWS Cloud Map is a cloud resource discovery service. With Cloud Map, you can define custom names for your application resources, and it maintains the updated location of these dynamically changing resources. This increases your application availability because your web service always discovers the most up-to-date locations of its resources. AWS App Mesh is a service mesh that provides application-level networking to make it easy for your services to communicate with each other across multiple types of compute infrastructure.
AWS App Mesh recently announced the support for AWS Cloud Map as a service-discovery option for virtual-nodes. In App Mesh *client *virtual-nodes add backends to define their service dependencies. These backends point to virtual-service that is backed by virtual-router with routes to each of the corresponding service virtual-nodes. Now for client to communicate with service, it will use the DNS specified on virtual-nodes. So prior to Cloud Map users need to add DNS records for each of the virtual-nodes. With Cloud Map users can create one DNS service for virtual-service and use attributes to subset the endpoints behind that DNS to target specific virtual-nodes. Additionally, App Mesh provides Envoy EDS that can be used by Envoy to discover endpoints for upstream cluster with minimal propagation latencies.
Currently the way to integrate Kubernetes cluster with Cloud Map is to use external-dns. However, external-dns lacks functionality to propagate attributes for endpoints (i.e. pod labels) to Cloud Map. To support App Mesh with Cloud Map aws-app-mesh-controller need to provide native support for Cloud Map.
User Stories
- User should be able to use AWS Cloud Map as service-discovery for pods associated with virtual-node.
- User should be able to use connect applications in Kubernetes cluster in AWS with services running outside of Kubernetes cluster.
Design
UX
Below we will discuss on how customers will use AWS Cloud Map in the context of aws-app-mesh-controller-for-k8s.
- Lets assume customer has the following K8s namespace
apiVersion: v1
kind: Namespace
metadata:
labels:
appmesh.k8s.aws/sidecarInjectorWebhook: enabled
name: color
- Customer creates mesh using the following spec
apiVersion: appmesh.k8s.aws/v1beta1
kind: Mesh
metadata:
name: eks-mesh
- Customer creates a virtual-node using the following spec
apiVersion: appmesh.k8s.aws/v1beta1
kind: VirtualNode
metadata:
name: colorteller-red
namespace: color
spec:
meshName: eks-mesh
listeners:
- portMapping:
port: 9080
protocol: http
serviceDiscovery:
cloudMap:
serviceName: colorteller
namespaceName: prod.svc.aws.local
- Customer creates a virtual-service using the following spec (No Change)
apiVersion: appmesh.k8s.aws/v1beta1
kind: VirtualService
metadata:
name: colorteller.prod.svc.aws.local
namespace: color
spec:
meshName: eks-mesh
virtualRouter:
name: colorteller-router
listeners:
- portMapping:
port: 9080
protocol: http
routes:
- name: color-route
http:
match:
prefix: /
action:
weightedTargets:
- virtualNodeName: colorteller-red
weight: 1
- Customer then can create K8s deployment (No change)
apiVersion: apps/v1
kind: Deployment
metadata:
name: colorteller-red
namespace: color
spec:
replicas: 1
selector:
matchLabels:
app: colorteller
version: red
template:
metadata:
annotations:
appmesh.k8s.aws/mesh: eks-mesh
labels:
app: colorteller
version: red
spec:
containers:
- name: colorteller
image: ${COLOR_TELLER_IMAGE}
ports:
- containerPort: 9080
env:
- name: "SERVER_PORT"
value: "9080"
- name: "COLOR"
value: "red"
Controller
Behind the scenes aws-app-mesh-controller will perform the following actions
- Watches for Mesh CRD
- Creates mesh using aws.appmesh.CreateMesh
- Watches for VirtualNode CRD
- Creates virtual-node using aws.appmesh.CreateVirtualNode
- Creates cloudmap service using *aws.servicediscovery.CreateService *using the service-name and cloudmap-namespace-name.
- Watches for pods (corresponding to K8s deployment)
- Determine the virtual-node and mesh corresponding to the pod.
- (Pod Added) If virtual-node uses CloudMap then register pod endpoint with corresponding cloudmap service using *aws.servicediscovery.RegisterInstances *api with attributes “appmesh.k8s.aws/virtualNode" and "appmesh.k8s.aws/mesh" set appropriately.
- (Pod Deleted) If virtual-node uses CloudMap then deregister pod endpoint by calling aws.servicediscovery.DeregisterInstances.
- Reconciles K8s pods and CloudMap instances periodically as updates to virtual-node take place.
Following App Mesh resources are created for above flow
$ aws appmesh describe-mesh \
--mesh-name eks-mesh
{
"mesh": {
"meshName": "eks-mesh",
"metadata": {
"arn": "arn:aws:appmesh:us-west-2:1234567890:mesh/eks-mesh",
"createdAt": 1560962735.095,
"lastUpdatedAt": 1560962735.095,
"uid": "7a31dd56-ef1e-44d5-b7f3-db98d109315c",
"version": 1
},
"spec": {},
"status": {
"status": "ACTIVE"
}
}
}
$ aws appmesh describe-virtual-node \
--mesh eks-mesh \
--virtual-node colorteller-red-color
{
"virtualNode": {
"meshName": "eks-mesh",
"metadata": {
"arn": "arn:aws:appmesh:us-west-2:1234567890:mesh/eks-mesh/virtualNode/colorteller-red-color",
"createdAt": 1560962757.57,
"lastUpdatedAt": 1564612052.233,
"uid": "4e1e5288-3585-4abe-809b-5c9d9018cdf9",
"version": 284
},
"spec": {
"backends": [],
"listeners": [
{
"portMapping": {
"port": 9080,
"protocol": "http"
}
}
],
"serviceDiscovery": {
"awsCloudMap": {
"attributes": [
{
"key": "appmesh.k8s.aws/mesh",
"value": "eks-mesh"
},
{
"key": "appmesh.k8s.aws/virtualNode",
"value": "colorteller-red-color"
}
],
"namespaceName": "prod.svc.aws.local",
"roleArn": "arn:aws:iam::1234567890:role/aws-service-role/appmesh.amazonaws.com/AWSServiceRoleForAppMesh",
"serviceName": "colorteller"
}
}
},
"status": {
"status": "ACTIVE"
}
}
}
$ aws appmesh describe-virtual-router \
--mesh eks-mesh \
--virtual-router colorteller-router-color
{
"virtualRouter": {
"meshName": "eks-mesh",
"metadata": {
"arn": "arn:aws:appmesh:us-west-2:1234567890:mesh/eks-mesh/virtualRouter/colorteller-router-color",
"createdAt": 1561148137.763,
"lastUpdatedAt": 1561148137.763,
"uid": "5c34c86a-2dac-4b45-a99c-18389f2ea994",
"version": 1
},
"spec": {
"listeners": [
{
"portMapping": {
"port": 9080,
"protocol": "http"
}
}
]
},
"status": {
"status": "ACTIVE"
},
"virtualRouterName": "colorteller-router-color"
}
}
$ aws appmesh describe-route \
--mesh $MESH_NAME \
--virtual-router colorteller-router-color \
--route color-rout
{
"route": {
"meshName": "eks-mesh",
"metadata": {
"arn": "arn:aws:appmesh:us-west-2:1234567890.:mesh/eks-mesh/virtualRouter/colorteller-router-color/route/color-route",
"createdAt": 1561148137.814,
"lastUpdatedAt": 1564613583.153,
"uid": "a6d43813-793f-4096-900b-e5b2fb717788",
"version": 14
},
"routeName": "color-route",
"spec": {
"httpRoute": {
"action": {
"weightedTargets": [
{
"virtualNode": "colorteller-red-color",
"weight": 1
}
]
},
"match": {
"prefix": "/"
}
}
},
"status": {
"status": "ACTIVE"
},
"virtualRouterName": "colorteller-router-color"
}
}
$ aws appmesh describe-virtual-service \
--mesh eks-mesh \
--virtual-service colorteller.prod.svc.aws.local
{
"virtualService": {
"meshName": "eks-mesh",
"metadata": {
"arn": "arn:aws:appmesh:us-west-2:1234567890:mesh/eks-mesh/virtualService/colorteller.prod.svc.aws.local",
"createdAt": 1561148137.624,
"lastUpdatedAt": 1561148137.911,
"uid": "f63e9694-6b0d-4e96-80bf-996064aca29f",
"version": 2
},
"spec": {
"provider": {
"virtualRouter": {
"virtualRouterName": "colorteller-router-color"
}
}
},
"status": {
"status": "ACTIVE"
},
"virtualServiceName": "colorteller.prod.svc.aws.local"
}
}
Following Cloud Map resources will be created by controller
$ aws servicediscovery get-service \
--id srv-sxvjmn5we2fz6nqm
{
"Service": {
"Id": "srv-sxvjmn5we2fz6nqm",
"Arn": "arn:aws:servicediscovery:us-west-2:1234567890:service/srv-sxvjmn5we2fz6nqm",
"Name": "colorteller",
"NamespaceId": "ns-omcos67xvs7tat4z",
"DnsConfig": {
"NamespaceId": "ns-omcos67xvs7tat4z",
"RoutingPolicy": "MULTIVALUE",
"DnsRecords": [
{
"Type": "A",
"TTL": 300
}
]
},
"CreateDate": 1564671683.11,
"CreatorRequestId": "app-mesh-controller"
}
}
$ aws servicediscovery discover-instances \
--namespace prod.svc.aws.local \
--service colorteller
{
"Instances": [
{
"InstanceId": "192.168.122.110",
"NamespaceName": "prod.svc.aws.local",
"ServiceName": "colorteller",
"HealthStatus": "UNKNOWN",
"Attributes": {
"AWS_INSTANCE_IPV4": "192.168.122.110",
"app": "colorteller",
"appmesh.k8s.aws/mesh": "eks-mesh",
"appmesh.k8s.aws/virtualNode": "colorteller-red-color",
"k8s.io/namespace": "color",
"k8s.io/pod": "colorteller-red-8c745484-sphc6",
"pod-template-hash": "8c745484",
"version": "red"
}
}
]
}
CRD Changes
type CloudMapServiceDiscovery struct {
- CloudMapServiceName string `json:"cloudMapServiceName"`
+ ServiceName string `json:"serviceName"`
+ NamespaceName string `json:"namespaceName"`
+ Attributes map[string]string `json:"attributes,omitempty"`
}
type VirtualNodeStatus struct {
// VirtualNodeArn is the AppMesh VirtualNode object's Amazon Resource Name
// +optional
VirtualNodeArn *string `json:"virtualNodeArn,omitempty"`
- // CloudMapServiceArn is a CloudMap Service object's Amazon Resource Name
+ Conditions []VirtualNodeCondition `json:"conditions"`
+ // CloudMapService is AWS CloudMap Service object's info
// +optional
- CloudMapServiceArn *string `json:"cloudMapServiceArn,omitempty"`
+ CloudMapService *CloudMapServiceStatus `json:"cloudmapService,omitempty"`
+}
+
+// CloudMapServiceStatus is AWS CloudMap Service object's info
+type CloudMapServiceStatus struct {
+ // ServiceID is AWS CloudMap Service object's Id
// +optional
- QueryParameters map[string]string `json:"queryParameters,omitempty"`
- Conditions []VirtualNodeCondition `json:"conditions"`
+ ServiceID *string `json:"serviceId,omitempty"`
+ // NamespaceID is AWS CloudMap Service object's namespace Id
+ // +optional
+ NamespaceID *string `json:"namespaceId,omitempty"`
}
Future Work
- Automatic cleanup of Cloud Map services
- Automatic creation and management of Cloud Map namespaces
- Automatic registration of pod endpoints outside the scope of App Mesh
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent