The CI pipeline for projects-operator is currently a concourse pipeline hosted on an internal Pivotal concourse deployment. As we move towards open sourcing the projects-operator, it may make sense to move to a more OSS-friendly CI workflow.

In an ideal world the solution we come up with would be able to successfully run the acceptance test suite, without requiring any external infrastrucutre (as this would require someone to gatekeep the passwords for said infrastructure).

One way I think we could achieve this is by updating the acceptance tests to run against kind, and use the kind Github action for CI workflows.

This would require us to configure the kind cluster for OIDC support, pointing to an openldap server that was itself deployed inside the cluster. I'm not totally sure if this is possible, but I don't see an obvious reason why it wouldn't be.

I wanted to open this issue to start a discussion before investing too much more time in it. What do y'all think?

There is a new release of controller-runtime, v0.5.1, that contains a couple of features which look like they are potentially of interest to us:

• Add controllerutil.SetOwnerReference
• Webhook support in envtest

controllerutil.SetOwnerReference in particular is of interest as we are currently using controllerutil.SetControllerReference to set non-controller OwnerRefs which is a slight misuse of the function. The webhook support in envtest should help us improve our tests to just be local.

Note: there is a dependabot PR that contains the bump to v0.5.1, #24.

We are currently using apiextensions.k8s.io/v1beta1 as the apiVersion for our CRDs. As of k8s 1.16.0 this is deprecated and the new apiVersion is apiextensions.k8s.io/v1. See https://v1-16.docs.kubernetes.io/docs/setup/release/notes/#api-changes for a (possibly non-exhaustive list) of changes. With kubebuilder this can be accomplished by passing in crd:crdVersions=v1 to controller-gen, see https://book.kubebuilder.io/reference/controller-gen.html#generators.

Note that this would then render subsequent releases of projects-operator incompatible with versions of kubernetes before 1.16.0 so we may want to hold off on this for a little bit, although notionally only 1.16+ are currently maintained: https://kubernetes.io/docs/setup/release/version-skew-policy/#supported-versions.

Currently we default to setting DevMode in projects-opeartor (both webhook and manager). This is nice but should probably be configurable.

I was curious if you all had considered creating a CLI that can do basic CRUD operations against Project and ProjectAccess resources. On the one hand, you can do everything you'd want with kubectl, but I'm wondering if creating a CLI could provide some sensible guardrails.

We (tanzu build service) just integrated the most recent changes from version 0.7.0 and we noticed that the ProjectAccess resource is namespace scoped. Given it operates on cluster resources, was there a reason it is not a cluster resource as well?

@ashwin-venkatesh

We don't currently support kubectl explain for any of the CRDs we have in projects-operator. The method for doing so is specifying a structural scheme, see https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#specifying-a-structural-schema.

The initial switch to a structural schema would come with #62. However in order to gain a useful output from kubectl explain we would also need to include useful descriptions (via comments above our types and fields) that describe the purpose of the CRDs as well as the spec fields.

• Ensure optional fields are marked as +optional
• Remove omitempty for essential fields

Optional vs Required fields

Suggested finalizer string: finalizer.projects.pivotal.io

The projects-operator is currently built using kubebuilder and deployed via Helm, with the corresponding helm chart living at /helm/projects-operator.

One problem with this approach is that kubebuilder does not currently have native support for templating out helm config. It currently only knows how to generate kustomize config.

This is a problem because it is then a manual (and error-prone) task to copy-and-edit the kustomize-generated config into helm-formatted config whenever we make changes to the projects-operator.

We have made a few attempts in the past to make this better, for example using the /scripts/helmify-yaml script, and custom make tasks, however it's still not intuitive and it requires us to keep thing up to date with newer versions of kubebuilder. Also this is only a partial solution as it only deals with rbac.

Perhaps we could improve this?

One idea @gmrodgers and I have been talking about is to make use of kubebuilder plugins (NB: currently experimental, design doc here).

It's not clear to us exactly if/how this would work, but based on comments from the operator-sdk team it seems like this should be possible.

Perhaps we should spend some time investigating if there is a helm plugin for kubebuilder in the works, and if not, should contribute one and then use it here?

@teddyking and @gmrodgers

controller-manager supposedly has some support for metrics which is the reason we are including kube-rbac-proxy as a dependency, though this functionality is currently unused. We should consider either using this functionality or removing the kube-rbac-proxy dependency.

It seems like klog is the standard logger to use for the kubernetes ecosystem, it has support for the logr interface via the klogr package so we should be able to swap out the current zap initialisations fairly directly.

Looks like with the introduction of the projects-create handler in #25, the tests now leave around a project. It looks like it's missing an AfterEach that the other tests are using.

cc @teddyking

The make install target uses the helmify-yaml script, which uses bsd sed syntax, rather than gnu sed syntax, so it works on macs but not linux. The issue is that gnu sed does not tolerate a space between -i and the suffix, while bsd sed requires the space.

Dependabot couldn't parse the go.mod found at /go.mod.

The error Dependabot encountered was:

go: k8s.io/[email protected] requires
	go.etcd.io/[email protected] requires
	github.com/grpc-ecosystem/[email protected] requires
	gopkg.in/[email protected]: invalid version: git fetch --unshallow -f origin in /opt/go/gopath/pkg/mod/cache/vcs/748bced43cf7672b862fbc52430e98581510f4f2c34fb30c0064b7102a68ae2c: exit status 128:
	fatal: The remote end hung up unexpectedly

View the update logs.

Instructions for development and contributing are currently in the README.md, rather than in their own separate CONTRIBUTING.md.

Additionally, the instructions for running tests are split across two separate sections, Deployment and testing workflow and Tests.

As part of moving this project to open source we should probably update the API Group to reflect the Pivotal acquisition. I would suggest that the API Group should be updated from projects.pivotal.io to projects.vmware.com, this is in line with the general guidance that an API Group should be a subdomain owned by the originating company.

Context:
As of now, whenever we need to apply a new RBAC role to the operator the process is as follow:

• Creates an annotation in the projects_controller.go (i.e: // +kubebuilder:rbac:groups=rbac.authorization.k8s.io,resources=rolebindings,verbs=watch;list;create;get;update;patch )
• Run make manifests in order to create the config/rbac/[roles-files]
• Copy and paste the content of the file in the helm/projects-operator/templates/rbac.yaml file

This is not a great development experience, it would be nice if this were simplified. Ideally, the files used by helm should be autogenerated.

https://github.com/pivotal/projects-operator/blob/dbe9fdf6f83fa9ce7b192fa8787b046d15ddc4b0/controllers/project_controller.go#L123

Hi all! We encountered this bug today when trying to delete projects. Namespaces in our cluster have a default "kubernetes" finalizer that will not allow them to be deleted (for various reasons that are not important). Thus, the controller will hang when attempting to delete those namespaces and the failure is opaque.

The error in the provided link is ignored and the code enters an infinite loop waiting for the namespace to disappear.

The projects-operator has a custom-built webhook that is serving CREATE project and CREATE/UPDATE projectaccess. While doing this work we created a standalone binary and manually wired it all up in our helm templating.

However it seems like kubebuilder has some pretty sweet support for webhooks, so it might be nice for us to buy into this.

Acceptance

• project-operator acceptance tests all pass
• we are using a kubebuilder webhook, not a custom manual one

Context on conditions

Acceptance

1. k get project -o yaml -w
2. k apply -f my-project.yaml
3. k delete -f my-project.yaml
4. Relevant changes to conditions reported in status in the watch

We use kapp delete to uninstall our deployment in CI. This will remove everything we've installed on the cluster including all projects. the project controller and webhook, and the project CRDs. Unfortunately, this creates a deadlock in the following way:

1. The project controller and webhooks are deleted first
2. The deletion attempts to delete projects but it cannot because they cannot be finalized by the controller
3. The deletion attempts to delete the project CRD but it cannot because projects still exist

One solution might be to add the project controller as an owner reference to all created projects.

Another solution may be to drop the finalizer from projects as they already have an owner reference for their corresponding namespaces.

@ashwin-venkatesh @matthewmcnew

Dependabot couldn't parse the go.mod found at /go.mod.

The error Dependabot encountered was:

go: k8s.io/[email protected] requires
	k8s.io/[email protected] requires
	gonum.org/v1/[email protected] requires
	modernc.org/[email protected]: reading modernc.org/cc/go.mod at revision v1.0.0: unknown revision v1.0.0

View the update logs.

https://github.com/pivotal/projects-operator/blob/afa814a2cd007b7fa850e2ebf9322f23c2e932f9/helm/projects-operator/templates/projectaccess-role.yaml#L16

Does this cluster role require the get verb? We may be able to just get away with create and delete, preventing unnecessary access of a project list.