The current AC controller implementation doesn't look up to a trait's TraitDefinition CR when it applies a trait to a workload. We need to throw an error if the workload is not part of the 'appliesTo' field.

The health scope currently only works with containerizedWorkload.

• We should make it work with any workload
• Add more unit test to bring the code coverage to at least 90%

ApplicationConfig controller should only fills the workload GVK to a trait CR if its CRD has a "spec.workloadRef" field conforms to the OAM definition like below

    workloadRef:
         properties:
           apiVersion:
             type: string
           kind:
             type: string
           name:
             type: string
         required:
         - apiVersion
         - kind
         - name
         type: object

Currently, ApplicationConfig just blindly adds a key-value pair to a trait as an unstructured object. This approach doesn't hurt when the CRD of the trait doesn't have the spec.workloadRef field. However, it will result in rejection from K8s api-server if the trait CRD already has the spec.workloadRef field but has a different format. It will also cause failure if the CRD doesn't have a spec field. We need a way to check ahead instead of blindly apply

gcr.io is blocked in some area, so we'd better use images from docker hub instead.

https://github.com/crossplane/oam-kubernetes-runtime/blob/master/charts/oam-core-runtime/templates/oam-controller.yaml#L114

Per discussion on previous community meeting and offline discussion in slack, we will merge OAM related code and repos into one and give it an independent release. It would be much easier to maintain after this re-org.

Tracking steps:

1. Merge https://github.com/crossplane/addon-oam-kubernetes-local into oam-kubernetes-runtime
2. Give oam-kubernetes-runtime a release (runnable controller with independent release version), add documentation for it
3. Refactor the existing documentation of crossplane.io
4. Deprecate https://github.com/oam-dev/crossplane-oam-sample in favor of this merge
5. Deprecate https://github.com/crossplane/addon-oam-kubernetes-remote in favor of deprecation of KubernetesApplication

/cc @ryanzhang-oss @wonderflow @hongchaodeng @zzxwill @negz

Currently, all OAM applications are intact even after the appConfig controller exists.

current struct:

// A ContainerizedWorkloadSpec defines the desired state of a
// ContainerizedWorkload.
type ContainerizedWorkloadSpec struct {
	// OperatingSystem required by this workload.
	// +kubebuilder:validation:Enum=linux;windows
	// +optional
	OperatingSystem *OperatingSystem `json:"osType,omitempty"`

	// CPUArchitecture required by this workload.
	// +kubebuilder:validation:Enum=i386;amd64;arm;arm64
	// +optional
	CPUArchitecture *CPUArchitecture `json:"arch,omitempty"`

	// Containers of which this workload consists.
	Containers []Container `json:"containers"`
}

add a new field WorkloadSubType

// A ContainerizedWorkloadSpec defines the desired state of a
// ContainerizedWorkload.
type ContainerizedWorkloadSpec struct {
	// OperatingSystem required by this workload.
	// +kubebuilder:validation:Enum=linux;windows
	// +optional
	OperatingSystem *OperatingSystem `json:"osType,omitempty"`

	// CPUArchitecture required by this workload.
	// +kubebuilder:validation:Enum=i386;amd64;arm;arm64
	// +optional
	CPUArchitecture *CPUArchitecture `json:"arch,omitempty"`
    
         // WorkLoadSubType required by this workload.
	// +kubebuilder:validation:Enum=server;task;crontask;statefulset;daemonset
	// +optional
	WorkloadSubType string `json:"subType,omitempty"`         

	// Containers of which this workload consists.
	Containers []Container `json:"containers"`
}

background

• In most cases, we hope to reuse kubernetes native ability,deployment,statefulset,job,cronjob,daemonset, so far these controllers covers all of our requirements
• trait and workload are independent controllers, trait can update its associated workload, but it's hard to go from deployment to statefulset
• If so, we can decide which type of native controller to start based on the workloadsubtype, there are also default behaviors, WorkloadSubType is optional filed
• If so, In most cases, we just need to maintain only one CRD for workload, unless they had special case need to develop a new workload CRD
• It's helpful us to construct an official workload that covered more cases, more user friendly and easy to use it without developing

Are we going to add a scope to change the namespace of a kubernetes object?

observedGeneration field of status in K8s object is widely used in K8s built-in resource. It reflects the generation of the most recently observed Object.

With the help of observedGeneration field, user can recognize whether a CR is reconciled by controller.

A workflow will like below:

apiversion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: c1
  generation: 1
spec:
  workload:
    image: nginx:v1
    ...

After the controller reconciled, the observedGeneration will appear and align with generation in metadata:

apiversion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: c1
  generation: 1
spec:
  workload:
    image: nginx:v1
    ...
status:
  observedGeneration: 1

Later, we have another change, it will be:

apiversion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: c1
  generation: 2
spec:
  workload:
    image: nginx:v2
    ...
status:
  observedGeneration: 1

And after reconciled, it will be:

apiversion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: c1
  generation: 2
spec:
  workload:
    image: nginx:v2
    ...
status:
  observedGeneration: 2

Also, CR of AppConfig also need to add this field.

Problem

Traits and workloads are two major types of resources in OAM. Traits usually affect how a workload operate either directly (through spec change) or indirectly (add ingress or sidecar). However, the current OAM implementation does not contain a generic mechanism for traits to locate and apply the desired changes to the resources. Therefore, we propose to address this problem in the OAM Kubernetes runtime.

Design principles

Before we lay out the detailed design. Let's first state the overall design objections.

1. We want to design an extensible mechanism that allows a trait to apply to any eligible workload. This means that this mechanism allows a trait developer to write the controller code once and the controller will work for any new workloads that this trait can apply to in the future.

2. The mechanism cannot put any limit on the trait or workload CRDs. This means that we cannot pre-define any common CRD fields beyond Kubernetes conventions. This is because we want to accommodate any existing CRD to be imported as workload or trait.

Mechanism explained in detail

Here is the step by step workflow

1. Add a childResources field in the WorkloadDefinition.
   Currently, a workloadDefinition is nothing but a shim of a real workload CRD. We propose to add an optional field called childResources to the spec of the workloadDefinition. We encourage workload owners to fill in this field when they register their controllers to the OAM system. In this way, they can declare the types of the Kubernetes resources the workload controller actually generates. For example,

apiVersion: core.oam.dev/v1alpha2
kind: WorkloadDefinition
metadata:
   name: schema.example.oam.dev
spec:
  definitionRef:
     name: schema.example.oam.dev
  childResources:
     - groupVersion: apps/v1
       kind: deployment
     - groupVersion: core/v1
       kind: service      

2. ApplicationConfig controller add labels containing the workload information to the emitted trait CR. For example,

apiVersion: core.oam.dev/v1alpha2
kind: ManualScalerTrait
metadata:
  labels:
    core.oam.dev/workloadGroupVersion: core.oam.dev/v1alpha2
    core.oam.dev/workloadKind:  ContainerizedWorkload
    core.oam.dev/workloadName: foo

3. The trait controller can use the following stps in the Reconcile logic to find out all the resources that it should try to modify.

• Fetch the corresponding workload instance by combining the label fields following Kubernetes convention (gvk)
• Fetch the corresponding workloadDefinition following an OAM convention that the name of the workloadDefinition CR is the name of the workload CRD it refers to. For example, the name of the workloadDefinition CR that refers to a containerizedworkloads.core.oam.dev CRD is exactly containerizedworkloads.core.oam.dev as well.
• Fetch all the childResources from the corresponding workloadDefinition CR.
• List each child resource by its type and filter by owner reference. Here, we assume that all the child resources that the workload generates have an owner reference field pointing back to the workload instance.
• Apply the trait-specific logic to the eligible child resources

4. The OAM Kubernetes runtime will provide a helper function to facilitate this process.

5. The mechanism is entirely optional to accommodate traits that have their own specific logic.

What problem are you facing?

OAM's HealthScope is not supported in Crossplane.

How could Crossplane help solve your problem?

Support HealthScope, similar to Rudr:

https://github.com/oam-dev/rudr/tree/master/healthscope

Parent: crossplane/crossplane#1480

Context

Parent Issue: crossplane/crossplane#1480

HealthScope aggregates the health status of multiple components via a web API.

In Rudr, it was implemented as described here: https://github.com/oam-dev/rudr/tree/master/healthscope

Proposal

Re-implement Rudr's approach but for the new v1alpha2 spec.

1. Deploy OAM's health scope controller:

helm install healthscope ./charts/healthscope

export POD_NAME=$(kubectl get pods -l "app.kubernetes.io/name=healthscope,app.kubernetes.io/instance=healthscope" -o jsonpath="{.items[0].metadata.name}")
kubectl port-forward $POD_NAME 8080:80

The health scope controller (like in Rudr) does 2 things:
1. Periodically check health status of components and update the HealthScope resource status.
2. serve as a http server, to output aggregated health information.

Optionally, it can be extended to offer /metrics API so health status can be exported to Prometheus.

2. Instantiate HealthScope as per the v1alpha2 spec:

apiVersion: core.oam.dev/v1alpha2
kind: HealthScope
metadata:
  name: example-health-scope
spec:
  probe-method: GET
  probe-endpoint: /health

$ kubectl get healthscope
NAME              AGE
example-health-scope   31s

HealthScope controller keeps track of the healthscopes created.

3. Declare one ApplicationConfiguration, using the health-scope:

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: first-app
spec:
  components:
    - componentName: helloworld-python-v1
      parameterValues:
        - name: target
          value: Rudr
        - name: port
          value: "9999"
      scopes:
        - apiVersion: core.oam.dev/v1alpha2
           kind: HealthScope
           name: example-health-scope

Services generated as part of the ApplicationConfig have the following additional label:

healthscope.core.oam.dev/example-health-scope: true

Example:

$ k describe service helloworld-python-v1
Name:                     helloworld-python-v1
Namespace:                default
Labels:                   workload.oam.crossplane.io=2df4035f-8019-4600-9317-ffbf2ac409b0
                            healthscope.core.oam.dev/example-health-scope: true
Annotations:            <none>
Selector:                 containerizedworkload.oam.crossplane.io=2df4035f-8019-4600-9317-ffbf2ac409b0
Type:                     LoadBalancer
IP:                       10.97.248.231
Port:                     helloworld-python-v1  3003/TCP
TargetPort:               3003/TCP
NodePort:                 helloworld-python-v1  31919/TCP
Endpoints:                172.17.0.12:3003,172.17.0.13:3003
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

HealthScope controller can now query for all services in a health scope by selecting them based on label.

Finally, the health of the scope is reported in:

http://localhost:8080/scopes/example-health-scope

The spec for HealthScope might need to change to incorporate other components, like a database. That work is out of scope (ta dan) for this issue.

Background

This is an extension from oam-dev/spec#326. Create the issue here to acquire more interest. We also have an implementation in our fork so we would like to merge them together.

Let me start with three use cases here.

1. Web app depends on database

An applicationConfiguration consists of two components: web and db. Web component should not be created until db component is created and ready, including database account and public endpoint created and ready.

2. ML app workflow

An ML applicationConfiguration has three components:

• preprocess
• training
• serving

They have strict ordering as follows:

  preprocess -> training -> serving

Without the previous stage finished, the next could not start due to lack of data.
Today, ML framework including Tensorflow would not handle such dependency health. They rely on external platform to handle it.

3. Service binding trait

More background info on service binding can be found here: slides.
A service binding trait needs to be applied before creating the component.
Moreover, this trait ordering use case is quite general and applies to many other traits we have.

Internally, there are scenarios where we need to get corresponding Annotation information in the Controller of the Workload and trait, so we can do some business processing based on the annotations, such as namespace isolation
Workload and trait only processing those workload and traits where the information in the Annotation meets the conditions

If app-config can pass this information, it's very convenient, You can write generic annotations to Workload and traits without having to write them intentionally, In particular, writing workload annotations in app-config is a little more complicated

Since a Trait CR 1:1 maps to Workload CR, maybe we can consider using label/annotation to replace trait.workloadRef?

e.g.

apiVersion: core.oam.dev/v1alpha2
kind: ManualScalerTrait
metadata:
  labels:
    core.oam.dev/workload: core.oam.dev/v1alpha2.ContainerizedWorkload
    core.oam.dev/workloadName: foo

The benefit is as Trait could also be brought by user, we don't want to require user modify CRD before using Crossplane.

(Optional) The link can also be Component if we agree on Component won't be shared in oam-dev/spec#350.

/cc @negz @ryanzhang-oss

If trying to apply Knative routes to workload configuration, I hit the issue.

$ k describe applicationconfiguration.core.oam.dev/knative-appconfig
Status:
  Conditions:
    Last Transition Time:  2020-06-10T07:49:04Z
    Message:               cannot apply components: cannot apply trait "blue-green-demo": cannot create object: admission webhook "webhook.serving.knative.dev" denied the request: mutation failed: cannot decode incoming new object: json: unknown field "workloadRef"
    Reason:                Encountered an error during resource reconciliation
    Status:                False
    Type:                  Synced
  Workloads:
    Component Name:  knative-configuration
    Workload Ref:
      API Version:  serving.knative.dev/v1
      Kind:         Configuration
      Name:         blue-green-demo
Events:
  Type     Reason                 Age                  From                                       Message
  ----     ------                 ----                 ----                                       -------
  Warning  CannotApplyComponents  47m                  oam/applicationconfiguration.core.oam.dev  cannot apply trait "blue-green-demo": cannot create object: admission webhook "webhook.serving.knative.dev" denied the request: mutation failed: cannot decode incoming new object: json: unknown field "workloadRef"

"spec.workloadRef" can be patched to manualscalertraits, but it might now be applicable to other traits.

What problem are you facing?

We didn't find concrete use case for Scope concept on K8s before so we decide to skip it temporarily in Crossplane. Though we now found Scope is actually a strong need for traffic mgmt in app. For example the Istio BookInfo sample should be able to be modeled easily with Crossplane's application model.

The mapping would be:

• Component - Deployment
• Trait - Service/Ingress per Deployment
• Scope - Service/Ingress/APIGateway across Deployments
• ApplicationConfiguration - a YAML file to grouping Component/Trait/Scope together

Mirror issue: oam-dev/rudr#558 (comment)

How could Crossplane help solve your problem?

Implement Scope feature in Crossplane's application model.

/cc @omkensey @ryanzhang-oss

Background

We need to have a common component template like helm charts to publish of common components across namespaces and even across clusters.
So far, the solution I can think of is to publish component resources in the form of helm charts.

Question

Does the community have any plans or suggestions for save component template?

xref: oam-dev/spec#331 (comment)

I am thinking maybe we can give them a short name like "trait" and "workload"?

I feel my fingers are trembling after CRUD traitdefinitions and workloaddefinitions for several times, they are just too long and verbose to type in even with best keyboard.

/cc @negz

Listed by priority from high to low:

1. Ingress
2. Autoscaler
3. Rollout (Flagger based)

Feel free to propose more.

I know specify a name can allow traits exist more than once for a component. And I also see some cases (e.g. Volume trait. Service trait) that may need more than once. But trait is designed as attribute of component, an attribute should exist only once.

Before we have a way to match both of the cases, I suggest we support both of them.

• If trait name wasn't specified, the name of trait can be automatically generated.
• If trait name already exist, we don't override.

I saw the core types, like ManualScalerTrait, it is too specific, and we can not reuse things from it, and we can not ensure user written traits works like we assume.
So I think it would be great if we can define a Generic Trait Interface, which define basic attribute and method, such as WorkloadRed, Apply.
Would this be possible?

Currently, we use the component name as the trait name in the setTraitProperties function. It will be good for the name to indicate that it belongs to a trait

Currently, an applictionConfig controller does not do many validations. It is less user friendly for the controller to fail to reconcile than we reject any invalid OAM applications. An admission controller is exactly for that purpose.

In our flight-tracker example, ingress is still installed as a standalone yaml file, it should be trait.

For example:

$ kubectl oam traits
NAME   DEFINITION                       APPLIES TO
route     route.core.oam.dev           containerizedworkload,apps.k8s.io

This means the trait route can be applied to any workloads under apps.k8s.io group.

/cc @ryanzhang-oss @wonderflow

• Revamp the way the containerizedWorkload controller do child resource garbage collection
• Add more unit test to bring up the test coverage

Follow similar steps as issue #40

Prerequisites

Install Knative and OAM runtime.

Deploy Knative Configuration by ApplicationConfiguration.

Step 1: deploy Knative Configuration workload with Component

✗ cat configuration-component-blue.yaml
apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: knative-configuration
spec:
  workload:
    apiVersion: serving.knative.dev/v1
    kind: Configuration
    metadata:
      name: blue-green-demo
      namespace: default
    spec:
      template:
        spec:
          containers:
            - image: gcr.io/knative-samples/knative-route-demo:blue # The URL to the sample app docker image
              env:
                - name: T_VERSION
                  value: "blue"
              ports:
                - containerPort: 80
                  name: wordpress


✗ k apply -f configuration-component-blue.yaml
component.core.oam.dev/knative-configuration created

Step 2: Apply ApplicationConfiguration

✗ cat applicationconfiguration-1-deploy-knative-configuration.yaml
apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: knative-appconfig
spec:
  components:
    - componentName: knative-configuration


✗ k apply -f applicationconfiguration-1-deploy-knative-configuration.yaml
applicationconfiguration.core.oam.dev/knative-appconfig created

✗ k describe applicationconfiguration.core.oam.dev/knative-appconfig
Name:         knative-appconfig
Namespace:    default
Labels:       <none>
Annotations:  API Version:  core.oam.dev/v1alpha2
Kind:         ApplicationConfiguration
Metadata:
  Creation Timestamp:  2020-06-17T15:27:35Z
  Generation:          1
  Resource Version:    1150849944
  Self Link:           /apis/core.oam.dev/v1alpha2/namespaces/default/applicationconfigurations/knative-appconfig
  UID:                 3b128c22-e1ba-402c-bfa7-81e3f267077a
Spec:
  Components:
    Component Name:  knative-configuration
Status:
  Conditions:
    Last Transition Time:  2020-06-17T15:27:35Z
    Message:               cannot apply components: cannot apply workload "blue-green-demo": cannot create object: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation failed: Port name wordpress is not allowed: spec.template.spec.containers[0].ports
Name must be empty, or one of: 'h2c', 'http1'
    Reason:  Encountered an error during resource reconciliation
    Status:  False
    Type:    Synced
Events:
  Type     Reason                 Age                From                                       Message
  ----     ------                 ----               ----                                       -------
  Normal   RenderedComponents     18s (x3 over 24s)  oam/applicationconfiguration.core.oam.dev  Successfully rendered components
  Warning  CannotApplyComponents  18s (x3 over 24s)  oam/applicationconfiguration.core.oam.dev  cannot apply workload "blue-green-demo": cannot create object: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation failed: Port name wordpress is not allowed: spec.template.spec.containers[0].ports
Name must be empty, or one of: 'h2c', 'http1'
  Normal   RenderedComponents     12s (x2 over 14s)  oam/applicationconfiguration.core.oam.dev  Successfully rendered components
  Warning  CannotApplyComponents  12s (x2 over 13s)  oam/applicationconfiguration.core.oam.dev  cannot apply workload "blue-green-demo": cannot create object: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation failed: Port name wordpress is not allowed: spec.template.spec.containers[0].ports
Name must be empty, or one of: 'h2c', 'http1'

If trying to deploy Configuration workload, it will hit a similar issue as above.

cannot apply workload "wordpress": cannot create object: admission webhook "webhook.serving.knative.dev" denied the request: mutation failed: cannot decode incoming new object: json: unknown field "containers"


✗ cat component-knative-configuration-wordpress.yaml
apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: wordpress
spec:
  workload:
    apiVersion: serving.knative.dev/v1
    kind: Configuration
    metadata:
      name: wordpress
      namespace: default
    spec:
      template:
        spec:
          containers:
            - image: wordpress:php7.4-apache
              name: wordpress
              env:
                - name: WORDPRESS_DB_HOST
                  valueFrom:
                    secretKeyRef:
                      name: db-secret
                      key: host
                - name: WORDPRESS_DB_USER
                  valueFrom:
                    secretKeyRef:
                      name: db-secret
                      key: user
                - name: WORDPRESS_DB_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: db-secret
                      key: password
              ports:
                - containerPort: 80
                  name: wordpress
  parameters:
    - name: db-secret
      description: Secret from which to read PostgreSQL connection details.
      required: true
      fieldPaths:
        - spec.containers[0].env[0].valueFrom.secretKeyRef.name
        - spec.containers[0].env[1].valueFrom.secretKeyRef.name
        - spec.containers[0].env[2].valueFrom.secretKeyRef.name

What will happen if I barely deploy Knative configuration and routes

Pure Knative Configuration deployment will also hit the same issue.

➜  /Users/zhouzhengxi/Programming/golang/src/github.com/zzxwill/oam-demos/knative/official git:(master) ✗ k apply -f blue-green-demo-config.yaml
Error from server (BadRequest): error when creating "blue-green-demo-config.yaml": admission webhook "validation.webhook.serving.knative.dev" denied the request: validation failed: Port name wordpress is not allowed: spec.template.spec.containers[0].ports
Name must be empty, or one of: 'h2c', 'http1'
➜  /Users/zhouzhengxi/Programming/golang/src/github.com/zzxwill/oam-demos/knative/official git:(master) ✗ cat blue-green-demo-config.yaml
apiVersion: serving.knative.dev/v1
kind: Configuration
metadata:
  name: blue-green-demo
  namespace: default
spec:
  template:
    spec:
      containers:
        - image: gcr.io/knative-samples/knative-route-demo:blue # The URL to the sample app docker image
          env:
            - name: T_VERSION
              value: "blue"
          ports:
            - containerPort: 80
              name: wordpress

Originally posted by @zzxwill in #40 (comment)

We used to discuss it in Rudr oam-dev/rudr#40
I think this is really a common trait which can be core/standard trait candidate.

This is an issue from crossplane/addon-oam-kubernetes-local#13.

We'll continue discuss here.

Now oam-k8s-runtime will check traitDefinition(for revisionAware) andworkloadDefinition(for workloadRef), but if these definition file not exist, application deploy will fail. However, the trait or workload can work without these information, so we shouldn't require definition files MUST exist.

In fact, we can automatically generate a basic definition file for those workload/trait, or we can regard there's one. Then the deploy won't fail.

For example, for ingress trait as below:

...
      traits:
        - trait:
            apiVersion: extensions/v1beta1
            kind: Ingress
            metadata:
              name: tracker-ingress
            spec:
                backend:
                  serviceName: web-ui
                  servicePort: 8080 

If we don't have a traitdefinition, we can assume it's a basic one like below:

apiVersion: core.oam.dev/v1alpha2
kind: TraitDefinition
metadata:
  name: ingresses.extensions
spec:
  definitionRef:
    name: ingresses.extensions 

All the information can be automatically generated. The main idea is to not block a normal deploy.

This has been raised up in the community meeting when we discussed #24.

Since spec is relatively stable, we want to validate ideas before making changes to spec. Usually we model some runtime functionalities as traits at first stage and then gradually move them into the spec. As a common practice, we need to agree on a standard experimental APIGroup for these kind of system/runtime traits.

My proposal is experimental.core.oam.dev/v1alpha2, where core.oam.dev/v1alpha2 matches the spec version and we add experimental prefix.

Now the oam-k8s-runtime need to specify instance-name parameter and pass it to workload CR.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: example-component
spec:
  workload:
    apiVersion: core.oam.dev/v1alpha2
    kind: ContainerizedWorkload
    spec:
      containers:
        - name: wordpress
          image: wordpress:4.6.1-apache
          # TODO(negz): Allow environment variables to be set from a secret? OAM
          # doesn't have a concept of Kubernetes secrets, so we will likely need
          # to abstract this somehow.
          ports:
            - containerPort: 80
              name: wordpress
  parameters:
    - name: instance-name
      required: true
      fieldPaths:
        - metadata.name

If not give instance-name as parameter, the workload CR will fail to create.

2020-06-01T19:50:56.978+0800	DEBUG	oam kubernetes runtime example	Cannot apply components	{"controller": "oam/applicationconfiguration.core.oam.dev", "request": "default/example-appconfig", "uid": "19c9bb8f-6889-4c06-853a-7dfd0d16e57c", "version": "6200267", "error": "cannot apply workload \"\": cannot get object: resource name may not be empty",

I think we can have a default name.

Some of the examples in the repo does not have README yet. For example, examples/flight-tracker does not have one. It would be good to add it

Workload will be applied to trait CR to its path spec.workloadRef in proposal trait-workload-interaction-mechanism. If the path mechanism support user defined list/slic type spec.gateways[0] as below?

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: bookinfo
spec:
  components:
    - componentName: bookinfo-gateway
      traits:
        - trait:
            apiVersion: networking.istio.io/v1alpha3
            kind: VirtualService
            metadata:
              name: bookinfo
            spec:
              hosts:
              - "*"
              gateways:
              - bookinfo-gateway

For more detailed info, please refer to kubevela/catalog#37

Currently, because the OAM Runtime only watches the component and Appconfig, it does not go to the Watch Workload CRD.

So workload changes are not notified to the OAM Runtime

This scenario might consider designing an elegant solution to the problem

There no mechanism that can do cleaning controllerrevisions

Including:

0. Prerequisite, which version of oam-runtime should we use
1. Workload and trait interact mechanism.
2. Decomposition operational properties from existing workload to be trait.
3. How trait CR will run by oam-k8s-runtime.
   ...

Since for some powerful trait like istio trait, it has the VirtualService, DefinitionRule, SideCar CR. For each CR, we may create a trait.

But they have relationship with each other, need to configure them all together make traffic works. In that case, we need to group them or allow multiple definitionRef of one trait.

why we are using Deployment? Can we use other kinds of kubernetes workload type? such as StatefulSet.

The trait system is an important part of OAM. The OAM community has given us feedbacks and one common concern is that applying a Trait may cause the containers in the workload to restart. This is not a desired behavior in many production environments.

We propose to add a trait's side effect to our traitDefinition. This allows our OAM platform to optimize our trait behavior so that we can minimize the negative impact on the workload.

We will follow up with a design proposal

Now we have Liveness and Readiness Probes in OAM containerized workload, but that's not enough.

In k8s pod, they have one more called startup probe.

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-startup-probes

Why we need it?

For some old apps, we can't expect how much time they need to start. Startup probe can be used to adopt liveness checks on slow starting containers, avoiding them getting killed by the kubelet before they are up and running

The kubelet uses startup probes to know when a container application has started. If such a probe is configured, it disables liveness and readiness checks until it succeeds, making sure those probes don't interfere with the application startup. This can be used to adopt liveness checks on slow starting containers, avoiding them getting killed by the kubelet before they are up and running.

Is it appropriate if ManualScalerTrait can update the container resource ?

// A ManualScalerTraitSpec defines the desired state of a ManualScalerTrait.
type ManualScalerTraitSpec struct {
	// ReplicaCount of the workload this trait applies to.
	ReplicaCount int32 `json:"replicaCount"`
        // container resource request
        ResourceRequest ContainerResources `json:"resourceRequest"` 
        // container resource limit
        ResourceLimit ContainerResources `json:"resourceLimit"`
	// WorkloadReference to the workload this trait applies to.
	WorkloadReference runtimev1alpha1.TypedReference `json:"workloadRef"`
}

1. scaling itself is related to resources
2. In some cases, the user want more replicas, but the resource of the instance can be lowered
3. only when the user passes in a valid value will we update the corresponding value
4. It's easier to maintain one than two

• ManualScalar trait currently is not watching any resources as it is using duck typing. We need to set a proper resync time to make sure that it can correct any changes that happen to the underlying resources. For example, if the deployment is scaled beyond the ManualScalar's back. Another way to do it maybe is to dynamically watch resources until they are removed. For example, only start to watch deployment after a deployment is scaled and remove the watch after the workload is removed.
• Add more unit test to bring up the test coverage

The dependency like below, when one component input parameter( ex. db-host) ready , then apiserver2 will been created, but another parameter(monitor-host) not been paved.

 - componentName: apiserver2
      dataInputs:
        - valueFrom:
            dataOutputName: monitor-host
          toFieldPaths:
            - "spec.containers[0].env[0].value"
        - valueFrom:
            dataOutputName: db-host
          toFieldPaths:
            - "spec.containers[0].env[1].value"

AppConfig now hardcode the workloadRef field on a scope CR. We need to use the same mechanism as trait to know the workloads it is associated with.

Background

In some cases workload will need to mark flags, for example to declare different environments, different tenants, different categories etc.

In OAM world, Application Scopes are very proper for doing so, we can use one scope to indicate some kind of environment. Then component which in this scope will be marked in this environment.

Before OAM exist, most of these demands are implemented by annotation or label of K8s Object. So I propose we implement category scope to uniform control label/annotation mark.

Proposal

Implement Category Scope to control label/annotation mark, the controller need be implemented using K8s webhook. So it can mutate workload CR before they are active in K8s cluster.

Assuming we already have this category scope implemented, the workflow would be like below.

1. Create CategoryScope, we can write it directly or reference a key from Secret/Configmap/ObjectField.

apiVersion: core.oam.dev/v1alpha2
kind: CategoryScope
metadata:
  name: scope1
spec:
  annotation:
    - key: tenant.uid
       value: "123"
    - key: alibabacloud.vpc
       value: "vpc-2ze"
  labels:
     - key: run.environment
        value: production
     - key: category.key
        valueFrom:
           secretKey:
              name: my-category
              key: categoryname       

2. Create AppConfig using this scope

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: cmp1
spec:
  workload:
    apiVersion: apps/v1
    kind: Deployment
    spec:
       containers: 
         - image: nginx:v1
---
apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: app1
spec:
  components:
     - componentName: cmp1
       scopes:
          - scope:
               apiVersion: core.oam.dev/v1alpha2
               kind: CategoryScope
               name: scope1

3. Category Scope CR updated and Workload CR created

apiVersion: core.oam.dev/v1alpha2
kind: CategoryScope
metadata:
  name: scope1
spec:
  workloadRef:
    - apiVersion: apps/v1
       kind: Deployment
       name: cmp1
  annotation:
    - key: tenant.uid
       value: "123"
    ... 

Before workload real created, webhook controller of CategoryScope should mutate and add annotation into it.

    apiVersion: apps/v1
    kind: Deployment
    metadata:
       name: cmp1
# ---------> data injected by webhook start <------------ #
       annotation:
          tenant.uid: "123"
          alibabacloud.vpc: "vpc-2ze"   
       labels:
         run.environment: "production"
         category.key: "ABC"
# ---------> data injected by webhook end  <------------ #
    spec:
       containers: 
         - image: nginx:v1

Problems

In current oam-runtime, the workload created and scope updated at the same time, so we can't ensure webhook will work before workload CR created by oam-runtime.

Solution: Add flag in ScopeDefinition

To ensure webhook controller will definitely work before workload CR created, oam-runtime should have a mechanism to hold creation of workload CR until scope CR marked completed.

apiVersion: core.oam.dev/v1alpha2
kind: ScopeDefinition
metadata:
  name: categoryscopes.core.oam.dev
spec:
  allowComponentOverlap: true
  definitionRef:
    name: categoryscopes.core.oam.dev
  completeCondition:
     - op: eq
        value: running
        fieldPath: "status.state"
     - op: observed
        fieldPath: "status.observedGeneration"

1. observed is compare fieldPath status.observedGeneration with metadata.generation, they should be equal if update was observed. This is a normal K8s way to indicate object is received by controller.
2. eq is compare value with fieldPath status.state.
3. Different conditions are AND-ed together.

When I apply all the crds with the following commands follow the instruction in the README:

$ kubectl apply -f crds/

customresourcedefinition.apiextensions.k8s.io/applicationconfigurations.core.oam.dev created
customresourcedefinition.apiextensions.k8s.io/containerizedworkloads.core.oam.dev created
customresourcedefinition.apiextensions.k8s.io/manualscalertraits.core.oam.dev created
customresourcedefinition.apiextensions.k8s.io/scopedefinitions.core.oam.dev created
customresourcedefinition.apiextensions.k8s.io/traitdefinitions.core.oam.dev created
customresourcedefinition.apiextensions.k8s.io/workloaddefinitions.core.oam.dev created
error: error validating "crds/core.oam.dev_components.yaml": error validating data: [ValidationError(CustomResourceDefinition.spec.validation.openAPIV3Schema.properties.spec.properties.workload): unknown field "x-kubernetes-embedded-resource" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaProps, ValidationError(CustomResourceDefinition.spec.validation.openAPIV3Schema.properties.spec.properties.workload): unknown field "x-kubernetes-preserve-unknown-fields" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaProps]; if you choose to ignore these errors, turn validation off with --validate=false

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.8-86+2a179bc0e65e24", GitCommit:"2a179bc0e65e2448bf00e6d5c2f4cc6159adf81a", GitTreeState:"clean", BuildDate:"2020-01-06T06:59:25Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

K8s has strategic merge patch, this is useful for update. If OAM workload/trait is k8s built-in resource, for example(deloyment, ingress)，then we should use strategy merge patch.

Write this issue for @CoderPoet

• We need to add more unit test
• We also need to have more E2E tests which makes sure that the code works on a Kubernetes cluster.