operator-framework / kubectl-operator Goto Github PK

Not few of us are struggling with the fact that sometimes one needs an operator in a given version despite all other versions in the catalog/channel.

There are very low options there and all of them rely on automation and manual approval of install plan for such installations (that are expected to have a Manual + startingCSV pair). And everyone's reinventing the wheel using ansible, acm, k8s jobs, disconnected mirrors with only one version, etc. again and again.

I'm not sure if this has been discussed before, but it would be really great if OLM can be smart enough to cover this use case: whenever a subscription is created, if the startingCSV field is defined and the installPlanApproval is properly set to this new value, install that version automatically but let the subscription remain in Manual. It can be a one shot comparison at creation time, no need to add extra logic for recurring modifications.

I am unable to add a particular catalog when running the command kubectl operator catalog add --display-name "IBMCS Operators" --publisher IBM --timeout 15m0s --namespace openshift-marketplace opencloud-operators docker.io/ibmcom/ibm-common-service-catalog:latest which fails with failed to add catalog: get image labels: no match for platform in manifest sha256:3e3fc1e8ff0900bf3ba9b8702a63d65b68d30e5bf4e6b4ce66966c9808fb97eb: not found
Other catalogs are able to be added successfully.

Running against OCP 4.4.9
Manifest is able to be pulled with local docker client.
Running on macOS 10.15.6, kubectl 1.16.6 / oc 4.2

Due to the way the subscription spec defines the Config and Resources sections as concrete structs instead of pointers, it is not possible to keep them unset. We need to update the operator-framework/api with this change, and then in this repo, require the new version of operator-framework/api that fixes it.

Related issues:

• operator-framework/operator-lifecycle-manager#1912
• operator-framework/operator-lifecycle-manager#1940
• operator-framework/operator-lifecycle-manager#1953

API PR: TBD
OLM PR: operator-framework/operator-lifecycle-manager#2010

Hi all,

I am new to kubectl operator , so just an RFE to start with: It would be more intuitive to use the common query syntax for the kubectl operator commands:

e.g:
replace:

kubectl operator catalog list -A

with:

kubectl operator get catalogs --all-namespaces
kubectl operator get catalog --all-namespaces
kubectl operator get catalog -A

Same thing for add and remove , that can respectively changed to create and delete

Right now, the kubectl operator uses an error-prone shortcut to enable users to provide just a --version flag to the install subcommand rather than the full starting CSV.

It starts by using the package name to lookup the CSV name at the head of the channel (the package name can be different than the base name of the CSV, e.g. prometheus vs prometheusoperator). So we can lookup prometheus and resolve to prometheusoperator.0.37.0.

The next step assumes the CSV name at the head of the channel contains a semver version, and it replaces the semver version it finds, with the desired version provided on the command line.

Now we have a guess for the name of the CSV for an earlier version. We blindly put the resulting value into the starting CSV field of the subscription and hope for the best.

We end up making 2 major assumptions, which could both fail:

1. That the CSV name has a particular format,
2. That all CSVs in a channel have the exact same format.

Can this be improved?

It'd be great to have some pointers for installing this in the Readme. Perhaps there is other documentation that could simply be linked?

Currently, in the case of "Openshift Serverless operator", I am able to install the operator with the following command:

k operator install serverless-operator --create-operator-group -v 1.32.1 -c stable

But in order to use the operator, I still have to manually create the "Knative Serving/Knative Eventing" operands/sub-resources to use the operator. Either I have to use the UI or apply the YAML.

I want to do this step from the CLI itself just like the way we install the operator.

I ran:

kubectl operator install etcd -w "default" -C

This seems to result in a successful installation, but there's an extra install plan in the namespace for etcd, which remains un-approved.

According to OLM maintainers, the duplicate install plan issue is a bug. OLM will auto delete all but the latest 5 install plans, so we could delete the duplicate install plan in kubectl operator code, but it doesn't seem strictly necessary.

Story: A user would like to be able to get the standard kubectl autocomplete experience when using the plug-in.

Open question: how difficult is this to implement?

Using OLM v0.16.1 and following the demo script, I am unable to upgrade the cockroachdb operator from 2.1.1 to 2.1.11.

If I update the subscription to set the channel (or re-install and explicitly pass --channel), the available upgrade shows up in the output of kubectl operator list cockroachdb.

Related to #1 and #2, when installing operators at non-channel heads, it is not possible to determine the set of supported install modes in advance. This makes it possible for the operator install command to install a Failed CSV when the channel head CSV and the installed CSV have a different set of supported install modes.

For example, the prometheus operator beta channel has:

• 0.37.0 (HEAD), supporting AllNamespaces
• 0.32.0, not supporting AllNamepspaces

When installing 0.32.0 using the default options, the CSV fails because the set of supported install modes is determined from the channel head. The order of precedence used by the installation selects AllNamespaces first (if supported), which is NOT supported by 0.32.0.

As management of numerous clusters is common in environments support for common syntax with kubectl should be included. Especially --context that allows us to do kubectl operator list --context someothercluster

The catalog source has several metadata fields in the spec that are optional, but nice to have for UX purposes, both on the command line and in a UI. These fields include:

• spec.description
• spec.displayName
• spec.icon
• spec.publisher

Currently (as a prototype) kubectl operator looks for index image labels:

• alpha.operators.operatorframework.io.index.display-name.v1 to set spec.displayName
• alpha.operators.operatorframework.io.index.publisher.v1 to set spec.publisher

When outputting the results of kubectl operator catalog list, kubectl operator includes columns for the display name and publisher. So images without these labels have no value in these columns.

The kubectl operator catalog add command has flags to handle setting these values, but it doesn't make a whole lot of sense to me for an administrator to be setting these fields.

Should there be optional labels added to the image spec to make it possible for index image owners to define the values for these fields?

As of #85, we have begun to add support for the OLMv1 APIs into this plugin via a new subcommand kubectl operator olmv1. So the current state of the world is:

• OLMv1 stuff at kubectl operator olmv1
• OLMv0 stuff at kubectl operator * (except olmv1)

Long term, I think the goal is:

• OLMv1 stuff at kubectl operator
• OLMv0 stuff no longer present.

How do we get from A to B?

Proposal 1

In #84, I proposed the use of an environment variable (I chose EXPERIMENTAL_USE_OLMV1_APIS, but we could pick something else), which would act as a switch controlling the entirety of the kubectl operator functionality:

• EXPERIMENTAL_USE_OLMV1_APIS=on: kubectl operator is OLMv1
• Anything else: kubectl operator is OLMv0

My thought there was that we could perform the migration like this:

• T1. OLMv0 support only
• T2. Default: OLMv0, require EXPERIMENTAL_USE_OLMV1_APIS=on for OLMv1
• T3. Default: OLMv1, require EXPERIMENTAL_USE_OLMV1_APIS=off for OLMv0
• T4. OLMv1 support only

In this proposed model, OLMv1 users can essentially opt-in early (at T2) and then not have to worry about subcommand changes later. Also in this model, OLMv0 users can explicitly opt-out early to avoid a breakage at T3. This gives both sets of users more time to adjust to incoming changes.

Proposal 2

Another option is to perform this same transition, but using subcommand renaming:

• T1. OLMv0 support only at kubectl operator
• T2. OLMv0 support at kubectl operator, OLMv1 support at kubectl operator olmv1
• T3. OLMv1 support at kubectl operator, OLMv0 support at kubectl operator olmv0
• T4. OLMv1 support only at kubectl operator

Proposal 3

Another option is a variation on Proposal 2, except we skip T3 and do a hard cutover:

• T1. OLMv0 support only at kubectl operator
• T2. OLMv0 support at kubectl operator, OLMv1 support at kubectl operator olmv1
• T3. OLMv1 support only at kubectl operator

My two cents

I had always imagined this working like proposal 1, which is similar to the way that Go modules were introduced in Go. IMO, the envvar approach keeps the CLI clean, makes for less typing for OLMv1 users, and will make the transition easy to implement.

In proposal 2 and 3, we'll definitely break early users of the OLMv1 functionality at some point when we remove the olmv1 subcommand and move that functionality under the main subcommand. And I feel like it would be nice if we could help them avoid that.

This plugin is still v0 (and may always be v0), so any of these proposals is fine from a semver standpoint. I'm mainly interested in choosing a strategy that primarily minimizes user impact and secondarily minimizes developer impact.

OLM users right now have a hard time debugging operator installations that have gone awry. They are required to dig deep into internal APIs, such as InstallPlans, to piece together what happened. These APIs are not well documented and the catalog operator logs may be misleading. It would be helpful if we had a command to help piece together what happened when something goes wrong for end users.

I thought about writing a olmdebug CLI tool but it may make more sense for this functionality to live in this project. WDYT about adding a debug command to the tool? Here are some thoughts on the usage:

kubectl operator debug <package>

• Often times users install operators via the UI or create a subscription pointing to a package, they may not know the CSV name specifically. By accepting a package name, the command could determine the CSV involved in the installation and return an amalgamation of CSV status, InstallPlan Status, the catalog involved, etc
• Over time, this command could get more sophisticated and return suggested fixes to the user, similar to how the cargo rust compiler detects compilation errors and returns ways to resolve them.

With the new kubectl operator list-operands command we have the ability to obtain a list of Custom Resources associated with an operator. This list can then be fed to kubectl delete to delete the CRs on-cluster.

For example, kubectl operator list-operands <operator> -o json | jq $filter | kubectl delete -f -

This RFE is for wrapping operand AND operator deletion in one command while ensuring its safe. Some design considerations:

• There already exists an kubectl operator uninstall command that deletes objects on cluster. This command can also delete CRDs, subscriptions, and other resources. We could simply modify this command to meet expected behavior.
• Another option is a new kubectl operator delete command with a narrower scope than uninstall, but that would duplicate logic and potentially confuse users
• Any deletion command would need to handle multi-tenancy and dependency concerns (e.g. only allow deleting CRDs if no other operator owns them, don't allow deleting an operator that another operator depends on, etc.)
• The command should probably have a warning and display the list of CRs to be removed before deleting them, with an additional manual opt in (with a --force override)

If I install cert-manager-1.4.0 using kubectl operator it tells me that an upgrade is available, but when I attempt to use kubectl operator ugrade it refuses.

$ kubectl operator install cert-manager -n operators --channel stable --create-operator-group --version 1.4.0
operatorgroup "operators" created
subscription "cert-manager" created
operator "cert-manager" installed; installed csv is "cert-manager.v1.4.0"

$ kubectl operator list
PACKAGE       SUBSCRIPTION  INSTALLED CSV        CURRENT CSV          STATUS            AGE
cert-manager  cert-manager  cert-manager.v1.4.0  cert-manager.v1.4.0  UpgradeAvailable  40s

$ kubectl operator list-available cert-manager
NAME          CATALOG              CHANNEL  LATEST CSV           AGE
cert-manager  Community Operators  stable   cert-manager.v1.4.3  177m

$ kubectl operator upgrade cert-manager -n operators
failed to upgrade operator: operator is already at latest version

Uninstall

$ kubectl operator uninstall cert-manager -n operators --delete-all
subscription "cert-manager" deleted
customresourcedefinition "certificaterequests.cert-manager.io" deleted
customresourcedefinition "certificates.cert-manager.io" deleted
customresourcedefinition "challenges.acme.cert-manager.io" deleted
customresourcedefinition "clusterissuers.cert-manager.io" deleted
customresourcedefinition "issuers.cert-manager.io" deleted
customresourcedefinition "orders.acme.cert-manager.io" deleted
clusterserviceversion "cert-manager.v1.4.3" deleted
operatorgroup "operators" deleted
operator "cert-manager" uninstalled

If I install cert-manager-1.4.2, then the kubectl operator upgrade command succeeds.

$ kubectl operator install cert-manager -n operators --channel stable --create-operator-group --version 1.4.2
operatorgroup "operators" created
subscription "cert-manager" created
operator "cert-manager" installed; installed csv is "cert-manager.v1.4.2"

$ kubectl operator list
PACKAGE       SUBSCRIPTION  INSTALLED CSV        CURRENT CSV          STATUS          AGE
cert-manager  cert-manager  cert-manager.v1.4.2  cert-manager.v1.4.3  UpgradePending  45s

$ kubectl operator upgrade cert-manager -n operators
operator "cert-manager" upgraded; installed csv is "cert-manager.v1.4.3"

Perhaps I'm misunderstanding how the upgrade mechanism works.

I expected that if I install 1.4.0 and the latest version is 1.4.3, that a manual upgrade would automatically step through each of the newer versions.

Right now, list-available <package> can show the versions at each channel head for a particular package, but it doesn't seem possible to know earlier versions.

Ideally, the packagemanifest API would tell me all versions in each channel.

I have been working on an operator which is listed in the Operator Hub catalog, and it has some resources (clusterrolebinding) which are expected to be created for service accounts in a certain namespace. The operator uses operatorframework.io/suggested-namespace to cajole the OpenShift console to install it into that namespace.

When using OLM without OpenShift, I have found that suggested-namespace is not apparently honored. The result of this is some ServiceAccounts are created in the operators namespace, and the ClusterRoleBindings are bound to service account in the suggested namespace that do not exist. (It was easy enough to update the ClusterRoleBindings by hand, but this is easier done than said, in other words it will be a bad mark on our documentation if we have to explain this...)

While in the long term the plan is to move away from service accounts and impersonate users instead, I'm not sure how to handle this in the short term. I understood that the OpenShift Console can be used outside of OpenShift, and that can be used to install operators and create namespaces, but that Operator Lifecycle Manager will not create namespaces by itself.

Would this be a reasonable feature addition to the kubectl-operator plugin? Is there somewhere else it should go besides OpenShift Console?

I haven't been through every page of the docs, but is there already a nice way to install the operator into a user-specified namespace at install time? (Would it help if I can create the namespace some other way?) I'm looking for the best way to resolve this, and someone suggested this kubectl plugin project might be a good place for this function to land.

Skimming through the options, I found there is a -n flag for namespace, but it does not appear to have the desired effect.

kubectl operator install -n flux-system flux --create-operator-group

(Flux still lands in the operators namespace.)

In #50 I have reported that I tried to install the Flux operator with kubectl-operator, and I learned it includes some service accounts and some ClusterRoleBindings which should be associated with them.

The Flux operator sets operatorframework.io/preferred-namespace, but as #50 explains, kubectl-operator does not honor it today. However there is a -n or --namespace option which I assumed is meant to set the target namespace for the operator lifecycle manager to install the operator itself into.

kubectl operator install -n flux-system flux --create-operator-group

Question

How to handle the case that one operator package is provided by multiple catalogsources?

Background

When users want to migrate the same operator from one catalogsource to another catalogsource or some other user case, there will be more than one catalogsources providing the same operator package. It causes there are multiple packageManifests using the same namespace in the same namespace, like

oc get packagemanifests -n ibm-common-services | grep ibm-common-service-operator
ibm-common-service-operator                          IBMCS Operators       24h
ibm-common-service-operator                          IBMD Operators        16h

In this case, the kubectl operator can't distinguish the difference between the packageManifest from different catalogsources and it uses the get command to fetch the packageManifest, which causes it will choose a random packageManifest from catalogsources.

This the result of the kubectl operator list-available command. It could have different results when executing the command twice in the same cluster.

➜  ~ kubectl operator list-available ibm-common-service-operator

NAME                         CATALOG          CHANNEL    LATEST CSV                                AGE
ibm-common-service-operator  IBMCS Operators  beta       ibm-common-service-operator.v3.7.1        24h
ibm-common-service-operator  IBMCS Operators  dev        ibm-common-service-operator.v101.101.101  24h
ibm-common-service-operator  IBMCS Operators  stable-v1  ibm-common-service-operator.v3.6.3        24h
ibm-common-service-operator  IBMCS Operators  v3         ibm-common-service-operator.v3.7.2        24h
➜  ~ kubectl operator list-available ibm-common-service-operator

NAME                         CATALOG         CHANNEL    LATEST CSV                                AGE
ibm-common-service-operator  IBMD Operators  beta       ibm-common-service-operator.v3.7.1        16h
ibm-common-service-operator  IBMD Operators  dev        ibm-common-service-operator.v101.101.101  16h
ibm-common-service-operator  IBMD Operators  stable-v1  ibm-common-service-operator.v3.6.3        16h
ibm-common-service-operator  IBMD Operators  v3         ibm-common-service-operator.v3.7.1        16h

Expect to know

1. Should I expect kubectl operator to handle this kind of scenario better? For example, using list command and filter the package name to get all the candidate packageManifests.

2. What is the best way to get the target PackageManifest? What I can think of is using list command and filter the result. Is there a best practice for it?

cc @cdjohnson

There has been some recent progress with OLMv1 and the new Operator API that would be useful to explore here. We can use the kubectl-operator CLI to help drive demos and provide feedback to core OLMv1 maintainers about the UX of the new API.

Acceptance criteria

• Experimental OLMv1 support is hidden and disabled by default
• Experimental OLMv1 support is enabled via an environment variable
• When OLMv1 support is enabled, OLMv0 commands are disabled
• operator install <packageName> creates an Operator object with metadata.name and spec.packageName set to the provided package name
• operator uninstall <packageName> deletes the named Operator object

There was a breaking change in operator-controller recently where Operator kind was renamed to ClusterExtension.

Now kubectl operator olmv1 commands fail with errors such as this:

failed to install operator: no matches for kind "Operator" in version "operators.operatorframework.io/v1alpha1"

See operator-framework/operator-controller#568 for more details

See operator-framework/operator-controller#813

Tried installing cockroachdb operator using the commands as given in demo.sh with the binary built from master.

Steps followed:

1. Install operator - kubectl-operator install cockroachdb --create-operator-group -v 2.1.1
   Output - Operator is installed successfully.

2. Uninstall operator - kubectl-operator uninstall cockroachdb --delete-crds --delete-operator-groups
   Output:

subscription "cockroachdb" deleted
uninstall operator: delete clusterserviceversion "": resource name may not be empty

When I check for the csv available in cluster after running the uninstall command, I get this:

➜  kubectl-operator git:(main) k get csv --all-namespaces | grep -i cockroachdb
default           cockroachdb.v2.1.1   CockroachDB      2.1.1     cockroachdb.v2.0.9   Succeeded
kube-node-lease   cockroachdb.v2.1.1   CockroachDB      2.1.1     cockroachdb.v2.0.9   Succeeded
kube-public       cockroachdb.v2.1.1   CockroachDB      2.1.1     cockroachdb.v2.0.9   Succeeded
kube-system       cockroachdb.v2.1.1   CockroachDB      2.1.1     cockroachdb.v2.0.9   Succeeded
olm               cockroachdb.v2.1.1   CockroachDB      2.1.1     cockroachdb.v2.0.9   Succeeded
operators         cockroachdb.v2.1.1   CockroachDB      2.1.1     cockroachdb.v2.0.9   Succeeded

The crds are not deleted either:

➜  kubectl-operator git:(main) k get crds --all-namespaces | grep -i cockroachdb
cockroachdbs.charts.helm.k8s.io               2020-11-03T03:01:41Z

Expected output: The operator is uninstalled successfully and all the resources are deleted.