The Kabanero operator works well in the kabanero namespace but we also need it to watch all namespaces.

Release the operator binary into a publicly available Docker registry. Since we would like to release this operator on OperatorHub, the registry might need to be quay.io

Building upon #10, when we create a new release in github, the Docker image should be built and published into the public Docker registry.

A Docker image can be pushed using the Travis deploy 'script' task. There are two cases to be considered:

Release

For a release build, this is identified as being a 'tag' build. See the documentation here: https://docs.travis-ci.com/user/deployment/releases/#deploying-only-on-tagged-builds

This would something like this:

deploy:
  provider: script
  ...
  draft: false
  on:
    tags: true

For a tagged build, the docker image tag should be kabanero-operator:<tag>

Draft Release

Travis also supports a subselector for "draft releases":

deploy:
  provider: script
  ...
  draft: true

When draft is set, this is considered a "release candidate" build, and the Docker tag should instead be: kabanero-operator:<version>-RC

The following RBAC errors are observed in the operator log when applying the default collection:

"error":"serviceaccounts \"appsody-sa\" is forbidden: User \"system:serviceaccount:kabanero:kabanero-operator\" cannot get serviceaccounts in the namespace \"kabanero\": no RBAC policy matched"

and

"error":"clusterrolebindings.rbac.authorization.k8s.io \"appsody-admin\" is forbidden: User \"system:serviceaccount:kabanero:kabanero-operator\" cannot get clusterrolebindings.rbac.authorization.k8s.io at the cluster scope: no RBAC policy matched"

Some consideration is needed as to how much authority the Kabanero operator service account should have, or whether someone should be pre-configuring some of these roles at the cluster scope so that the service account does not need cluster admin authority.

avoid this problem with build-image

operator-sdk build kabanero-operator:latest
# github.com/kabanero-io/kabanero-operator/cmd/manager
cmd/manager/main.go:45:42: cannot use "github.com/operator-framework/operator-sdk/pkg/log/zap".FlagSet() (type *"github.com/operator-framework/operator-sdk/vendor/github.com/spf13/pflag".FlagSet) as type *"github.com/spf13/pflag".FlagSet in argument to "github.com/spf13/pflag".CommandLine.AddFlagSet
cmd/manager/main.go:61:27: cannot use "github.com/operator-framework/operator-sdk/pkg/log/zap".Logger() (type "github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/logr".Logger) as type "github.com/go-logr/logr".Logger in argument to "sigs.k8s.io/controller-runtime/pkg/runtime/log".SetLogger:
        "github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/logr".Logger does not implement "github.com/go-logr/logr".Logger (wrong type for V method)
                have V(int) "github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/logr".InfoLogger
                want V(int) "github.com/go-logr/logr".InfoLogger
cmd/manager/main.go:90:3: cannot use restmapper.NewDynamicRESTMapper (type func(*"github.com/operator-framework/operator-sdk/vendor/k8s.io/client-go/rest".Config) ("github.com/operator-framework/operator-sdk/vendor/k8s.io/apimachinery/pkg/api/meta".RESTMapper, error)) as type func(*"k8s.io/client-go/rest".Config) ("k8s.io/apimachinery/pkg/api/meta".RESTMapper, error) in field value
cmd/manager/main.go:113:11: undefined: metrics.ExposeMetricsPort
Error: failed to build operator binary: (failed to exec []string{"go", "build", "-o", "/home/dacleyra/go/src/github.com/kabanero-io/kabanero-operator/build/_output/bin/kabanero-operator", "-gcflags", "all=-trimpath=${GOPATH}", "-asmflags", "all=-trimpath=${GOPATH}", "github.com/kabanero-io/kabanero-operator/cmd/manager"}: exit status 2)

Reduce the number of steps required to provision the Kabanero Operators

The operator uses a wildcard role assignment right now, but the specific permissions need to be accounted for instead.

Work to be completed:

• Update the RBAC generation annotations in the source
• Run the operator-sdk scripts to regenerate the role yaml

Status reporting seems to be broken for some of the underlying services. Individually test each of the services (Tekton, KNative Eventing, etc) to see that a bad status on those services shows as a bad status on the Kabanero instance.

The .travis.yml file has a workaround which allows forks to build, but it fails on the main repository.

The following messages are present in the log:

The command "cd ../ && mv kabanero-operator $HOME/gopath/src/github.com/kabanero-io" failed and exited with 1 during .

If dependencies.yaml still need to be installed, make sure to pin a version of this file as part of a release.

We need to document that someone who is working with collections (activating, upgrading, etc) will require RBAC authority to create/modify the kind Collection. Likewise, someone wanting to configure a Kabanero instance will need access to the Kind Kabanero.

Some of the dependent operators are not published to OperatorHub. We have several options:

• Wait for the dependencies to arrive
• Embed the dependencies into the Kabanero Operator
• Fork and publish the dependencies

The kabanero-operator CSV says that the Kabanero CRD owns the following resources:

      resources:
      - kind: ConfigMap
        name: ""
        version: v1
      - kind: Deployment
        name: ""
        version: v1
      - kind: Pod
        name: ""
        version: v1
      - kind: Role
        name: ""
        version: v1
      - kind: RoleBinding
        name: ""
        version: v1
      - kind: ServiceAccount
        name: ""
        version: v1

This is not correct - the Kabanero operator creates a Tekton Install, a Knative KnativeEventing and a Knative KnativeServing.

The OKD / OpenShift console uses the resources section as a clue for what to display when viewing an instance of our CRD. It's not broken the way it is, but it's not useful either.

Evaluate the Kabanero operator using available certification compliance tools and make fixes as necessary in order to achieve compliance with these tools. Demonstrate that the Operator fits into the OperatorHub through localized demonstration.

Update the artifact https://github.com/kabanero-io/kabanero-operator/releases/download/0.0.2/kabanero-operators.yaml

to:

• use the kabanero-operator:0.0.2 tag instead of latest
• use the proper namespacing

Then reattach the update to the release and notify other team members (such as doc)

Suggest a test framework for integration testing, likely building upon existing Kubernetes test capabilities already provided by the community. The test framework needs to be able to test such things as whether a migration between platform releases is successful through editing of the resource document.

We may or may not require a golang specific test framework, but please take a look at Ginkgo.

To start with, I suggest that we try to incorporate the following types of tests:

1. Unit. These will be those tests which do not have any golang build flags (see this answer. Simple, fast running tests.
2. Integration. These tests may call the API server. I suggest setting up an API server using: the envtest package in controller-runtime. These need a build flag so that they are not normally run with go test
3. Acceptance. This requires more thought.

Subtasks:

• Add makefile tasks for unit and integration tests. Unit tests will run go test with no tags, integration will run go test -tags=integration. There are a few tests tagged as integration that might need their flag updated.
• Run unit tests in the Travis build by calling the above make target
• Investigate and report on Ginkgo
• Investigate how we can setup the API server using envtest and tear it down at the end of some number of tests. One example of how to do this is here and another is using a test main but there are other ways and there may be some specific ways in Ginkgo.

The kabanero-operators.yaml generated by the release build is not quite correct.
Update travis.yml to use the script located here: contrib/gen_operator_deployment.sh.
This script accepts as input the IMAGE to be used as the kabanero operator deployment's image.
The image value could be set through a travis-ci env variable.

The CSV allows you to list "required" CRDs. The documentation is not clear about how these should be processed. Our initial test seems to show that OLM will ignore any CSV that required CRDs, even if they are installed. This should be investigated further. Is it that:

1. The Operator will not be read from the catalog until all its required CRDs are defined?
2. The Operator will not be read from the catalog unless its required CRSs were installed by OLM?
3. The Operator will not be read from the catalog unless the Operator which owns the required CRDs is available to be installed in/by some other available catalog (such as operator-marketplace)?

This may involve testing on Open Shift 4 to realize the expected behavior, and then coming back to OKD 3.11 to see if it can be duplicated there.

The ideal behavior would be for the user to click on the Kabanero operator in the catalog, and have the dependent operators (Tekton Pipeline, Knative Eventing and Knative Serving) be installed automatically if they are not already installed. This may not be possible, but, we'd like to understand what the actual behavior is.

The collection needs a status which reflects both the current state, version and digests of the collection but also any errors applying collection contents. A common error applying collection contents results from RBAC limitations within the collection contents. Some of the status recording needs to be per resource, while other parts are per collection.

Example:

status:
    # Here available version > current reflects an available update
    availableVersion: 1.0.0.1
    currentVersion: 1.0.0.0
    resources:
    - name: <resource name or path>
      currentDigest: <digest>
      status: applied
    - name: <resource name or path>
      currentDigest: <digest>
      status: failed
      statusMessage: "role" is forbidden from creating objects of kind

Phases and Conditions
Some Kubernetes objects are treated as a state machine and record their status using "phases and conditions" concepts. This allows the status to record such things as whether it is initializing, ready, failed, etc. The use of phases and conditions is advanced and optional at this point, however, some of this idea is present in the above modeling in order to account for individual resource application failures.

For an example of this, see:

• https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/conditions.go#L83
• https://github.com/openshift-knative/knative-eventing-operator/blob/62770a4fec0e75e81377731073c94cb3b1ca82ef/pkg/apis/eventing/v1alpha1/knativeeventing_lifecycle.go

It is not valid to have two kabanero instances in the same namespace, since some of the dependencies cannot be installed twice in the same namespace. The kabanero operator should prevent a second instance from being processed, if a first instance already exists in the namespace.

We have several actions which can only occur in an admission webhook:

• Defaulting of resource values
• Ensuring only one instance of Kabanero in a given namespace context
• Setting the relational reference between a collection and a Kabanero instance

The controller runtime supports admission webhooks which requires that you implement a handler for the admission action.

TLS
Since a webhook introduces an HTTP listener endpoint, this should be secured with TLS which will require valid certificates be generated. If other examples cannot be found, I would suggest looking at how this was done in the Service Catalog project.

When deploying a Kabanero instance, a KnativeEventing instance is created. The status of that instance reports:

    - lastTransitionTime: '2019-07-09T14:53:32Z'
      message: >-
        Install not attempted: The only KnativeEventing resource that matters is
        knative-eventing/knative-eventing
      reason: Ignored
      status: 'False'
      type: Ready

We probably need to stop creating that resource by default, and if we have something specific we need, patch the existing resource in the knative-eventing namespace rather than try to create our own resource.

The inclusion of Che as part of the Kabanero foundation allows enterprises to develop microservices leveraging remote Kubernetes resources. Additionally, it will be a preparation step for enabling multi-architecture implementations, such as IBM p- and z-architectures.

We currently do docker login -u <u> -p <p> but it is better to feed the password through stdin instead. See the docker login docs for an example.

oc adm policy commands are implemented elsewhere and not needed in the QuickStart section

As reported by @dacleyra the dependent operators versions are baked into the dependencies.yaml file and some of these are now out of date.

A fix involves either an update the docker image tags, or regenerate the dependencies file.

The collections controller is a minimal implementation which when applied retrieves the remote contents and applies these contents. Any modification of the resource results in re-execution of this logic.

Digest data is available in the remote repository for both the collection as well as the individual resources. The digest for either one or both of these should be recorded in the status, and then application of the collection be limited to the case where the digest is new.

There is an additional case where the digests changing which may lead to re-application of the collection. This is a bit like "SNAPSHOTS" in a Maven repository, where the HEAD of the repository is changing but the version is not fully committed. This probably should not be supported as a base case, but perhaps could be recognized when a Kabanero instance configuration attribute is present. If no resolution on this is made within the current issue, we should fork this into a separate issue.

The existing index/collection format in https://raw.githubusercontent.com/kabanero-io/kabanero-collection/master/experimental/index.yaml is being deprecated. The new format appears to be worked on here:
kabanero-io/kabanero-collection#40
The operator must use this new format (as well as tolerate the existing format for a short time).

If I delete a Kabanero instance, its Collection instance(s) are not deleted, they must be deleted separately. We need to investigate whether it makes sense to tie the to together, similarly to how the tekton.dev Install instance is deleted when a Kabanero instance is deleted.

In addition to the enhancement to digest checking mentioned in #60, the collection status should have an indicator that a version update is available. The update could be qualified using the semantic version, for example be tagged as a "major" update, but this behavior is optional at this time.

Create a new markdown document and document OKD or OpenShift specifics.

When a pull request is submitted, one of the prerequisite checks is to scan it for open source license compatibility with our project.

Add the ability to install versioned collections from a collections repository and manage the version of the collection that is installed, view errors in the status, etc.

This epic is complete once we arrive at an 'MVP' level of function which is to be determined by the team.

Update kabanero-operator/.travis/prepare_release.sh to include a sed substitution for kabanero-operator:latest -> kabaner-operator:$TRAVIS_TAG

There is no if checking in the collection controller, so it always loads from the default:

Add an "if ! nil" to prevent this

manifestival needs to be able to access the vfs that contains
config/reconciler/knative-eventing
config/reconciler/knative-serving

until so, workaround by copying config to the filesystem using an additional Dockerfile & docker build

There appears to be an ordering and/or timing problem when creating the default collection. Sometimes, Pipeline and Task resources cannot be created by the collection controller, presumably because the Tekton operator has not finished processing the Install resource yet (this causes Tekton to register the Pipeline and Task CRDs). The error can happen in several ways, some of which include (from the Kabanero operator log):

"error":"no matches for kind \"Task\" in version \"tekton.dev/v1alpha1\""

or

"error":"the server could not find the requested resource"

Eventually, the CRDs are registered, but the operator does not seem to always detect this, and so future loops of the collection controller continue to receive the error. We should investigate whether we can fix the timing here... perhaps if the Kabanero controller is going to create a Install, it could wait until the Pipeline and Task CRDs are available before trying to create the Collection instance. It may be that the particular collection does not define a Pipeline or Task, but that's a harder problem to solve.

Document the procedure for performing a release

Release the Kabanero Operator on OperatorHub.io

When the version identifier specified in a collection changes, the resources found in the version need to be applied and the status updated. This needs to be done such that the version can later be "rolled back" to the prior. For example, if resources are removed from the collection, those resources need to be removed. This should likely also be recorded in the status so that you can see what resources changed when a version was applied and what the current version is.

Add the ability to provision a Kafka cluster when provisioning a Kabanero Foundation Instance.

Enable travis ci builds for this project.

Work to be completed:

• Enable travis in a fork by visiting travis-ci.org, in the upper right corner by your username, click settings, then click the 'enable' check for kabanero-operator
• Work through any build failures, committing to the fork, and observing the build results
• Enable travis for github.com/kabanero-io/kabanero-operator (awaiting authorization)

Per kabanero-io/kabanero-foundation#2 the collection controller needs to recognize some priority or featured collections

Currently whether to install the featured collections is on a per Kabanero instance basis, but this should ideally be modeled as per repository. Featured collections are to be installed by default -- if the configuration value is unspecified, then featured collection installation is enabled.

The existing repository modeling is:

repositories: 
- name: myrepo
  url: http://....

Proposed:

repositories: 
- name: myrepo
  url: http://....
  installFeaturedCollections: true

One of the philosophical points of view with the operator pattern is that the operator has a separate lifecycle from the software it manages. This lets you upgrade the operator without disturbing and of the existing software instances, and lets the operator orchestrate roll forward/roll back of a software instance.

The Kabanero instance controller adheres to this point of view, but the collection controller seemingly violates this point of view. We might consider factoring the collection controller off into another deployment at some point in the future.

Tekton dashboard webhook depends on eventing contrib for github source
(githubsources.sources.eventing.knative.dev)

We have makefile targets for deploy, but this is hardcoded to use the publicly available image. The public Kabanero image should be set as the default image URI in the makefile, but all other references should use the IMAGE attribute like the 'build-image'.

Build upon #2, but formally certify.

Enable the ability to activate a Kabanero Collection on a Kabanero Foundation Instance, which extracts Kubernetes assets from the Kabanero Collection Repository and applies these to a local Kubernetes cluster.