vmware-tanzu / projects-operator Goto Github PK
View Code? Open in Web Editor NEWProvides a `Project` CRD and controller for k8s to help with organising resources
License: Other
Provides a `Project` CRD and controller for k8s to help with organising resources
License: Other
I was curious if you all had considered creating a CLI that can do basic CRUD operations against Project and ProjectAccess resources. On the one hand, you can do everything you'd want with kubectl, but I'm wondering if creating a CLI could provide some sensible guardrails.
The projects-operator
has a custom-built webhook that is serving CREATE project
and CREATE/UPDATE projectaccess
. While doing this work we created a standalone binary and manually wired it all up in our helm templating.
However it seems like kubebuilder has some pretty sweet support for webhooks, so it might be nice for us to buy into this.
Suggested finalizer string: finalizer.projects.pivotal.io
The make install target uses the helmify-yaml script, which uses bsd sed syntax, rather than gnu sed syntax, so it works on macs but not linux. The issue is that gnu sed does not tolerate a space between -i
and the suffix, while bsd sed requires the space.
Looks like with the introduction of the projects-create handler in #25, the tests now leave around a project. It looks like it's missing an AfterEach
that the other tests are using.
cc @teddyking
Currently we default to setting DevMode
in projects-opeartor (both webhook and manager). This is nice but should probably be configurable.
Instructions for development and contributing are currently in the README.md, rather than in their own separate CONTRIBUTING.md.
Additionally, the instructions for running tests are split across two separate sections, Deployment and testing workflow
and Tests
.
We are currently using apiextensions.k8s.io/v1beta1
as the apiVersion
for our CRDs. As of k8s 1.16.0 this is deprecated and the new apiVersion
is apiextensions.k8s.io/v1
. See https://v1-16.docs.kubernetes.io/docs/setup/release/notes/#api-changes for a (possibly non-exhaustive list) of changes. With kubebuilder
this can be accomplished by passing in crd:crdVersions=v1
to controller-gen
, see https://book.kubebuilder.io/reference/controller-gen.html#generators.
Note that this would then render subsequent releases of projects-operator incompatible with versions of kubernetes before 1.16.0 so we may want to hold off on this for a little bit, although notionally only 1.16+ are currently maintained: https://kubernetes.io/docs/setup/release/version-skew-policy/#supported-versions.
There is a new release of controller-runtime, v0.5.1, that contains a couple of features which look like they are potentially of interest to us:
Add controllerutil.SetOwnerReference
Webhook support in envtest
controllerutil.SetOwnerReference
in particular is of interest as we are currently using controllerutil.SetControllerReference
to set non-controller OwnerRefs which is a slight misuse of the function. The webhook support in envtest
should help us improve our tests to just be local.
Note: there is a dependabot PR that contains the bump to v0.5.1, #24.
The CI pipeline for projects-operator
is currently a concourse pipeline hosted on an internal Pivotal concourse deployment. As we move towards open sourcing the projects-operator, it may make sense to move to a more OSS-friendly CI workflow.
In an ideal world the solution we come up with would be able to successfully run the acceptance test suite, without requiring any external infrastrucutre (as this would require someone to gatekeep the passwords for said infrastructure).
One way I think we could achieve this is by updating the acceptance tests to run against kind, and use the kind Github action for CI workflows.
This would require us to configure the kind cluster for OIDC support, pointing to an openldap server that was itself deployed inside the cluster. I'm not totally sure if this is possible, but I don't see an obvious reason why it wouldn't be.
I wanted to open this issue to start a discussion before investing too much more time in it. What do y'all think?
We don't currently support kubectl explain
for any of the CRDs we have in projects-operator. The method for doing so is specifying a structural scheme, see https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#specifying-a-structural-schema.
The initial switch to a structural schema would come with #62. However in order to gain a useful output from kubectl explain
we would also need to include useful descriptions (via comments above our types and fields) that describe the purpose of the CRDs as well as the spec fields.
Dependabot couldn't parse the go.mod found at /go.mod
.
The error Dependabot encountered was:
go: k8s.io/[email protected] requires
k8s.io/[email protected] requires
gonum.org/v1/[email protected] requires
modernc.org/[email protected]: reading modernc.org/cc/go.mod at revision v1.0.0: unknown revision v1.0.0
Does this cluster role require the get verb? We may be able to just get away with create and delete, preventing unnecessary access of a project list.
Dependabot couldn't parse the go.mod found at /go.mod
.
The error Dependabot encountered was:
go: k8s.io/[email protected] requires
go.etcd.io/[email protected] requires
github.com/grpc-ecosystem/[email protected] requires
gopkg.in/[email protected]: invalid version: git fetch --unshallow -f origin in /opt/go/gopath/pkg/mod/cache/vcs/748bced43cf7672b862fbc52430e98581510f4f2c34fb30c0064b7102a68ae2c: exit status 128:
fatal: The remote end hung up unexpectedly
Context:
As of now, whenever we need to apply a new RBAC role to the operator the process is as follow:
projects_controller.go
(i.e: // +kubebuilder:rbac:groups=rbac.authorization.k8s.io,resources=rolebindings,verbs=watch;list;create;get;update;patch
)make manifests
in order to create the config/rbac/[roles-files]
helm/projects-operator/templates/rbac.yaml
fileThis is not a great development experience, it would be nice if this were simplified. Ideally, the files used by helm should be autogenerated.
controller-manager supposedly has some support for metrics which is the reason we are including kube-rbac-proxy as a dependency, though this functionality is currently unused. We should consider either using this functionality or removing the kube-rbac-proxy dependency.
k get project -o yaml -w
k apply -f my-project.yaml
k delete -f my-project.yaml
status
in the watchHi all! We encountered this bug today when trying to delete projects. Namespaces in our cluster have a default "kubernetes" finalizer that will not allow them to be deleted (for various reasons that are not important). Thus, the controller will hang when attempting to delete those namespaces and the failure is opaque.
The error in the provided link is ignored and the code enters an infinite loop waiting for the namespace to disappear.
The projects-operator is currently built using kubebuilder and deployed via Helm, with the corresponding helm chart living at /helm/projects-operator.
One problem with this approach is that kubebuilder does not currently have native support for templating out helm config. It currently only knows how to generate kustomize config.
This is a problem because it is then a manual (and error-prone) task to copy-and-edit the kustomize-generated config into helm-formatted config whenever we make changes to the projects-operator.
We have made a few attempts in the past to make this better, for example using the /scripts/helmify-yaml script, and custom make tasks, however it's still not intuitive and it requires us to keep thing up to date with newer versions of kubebuilder. Also this is only a partial solution as it only deals with rbac.
Perhaps we could improve this?
One idea @gmrodgers and I have been talking about is to make use of kubebuilder plugins (NB: currently experimental, design doc here).
It's not clear to us exactly if/how this would work, but based on comments from the operator-sdk team it seems like this should be possible.
Perhaps we should spend some time investigating if there is a helm plugin for kubebuilder in the works, and if not, should contribute one and then use it here?
@teddyking and @gmrodgers
+optional
omitempty
for essential fieldsWe (tanzu build service) just integrated the most recent changes from version 0.7.0 and we noticed that the ProjectAccess
resource is namespace scoped. Given it operates on cluster resources, was there a reason it is not a cluster resource as well?
@ashwin-venkatesh
As part of moving this project to open source we should probably update the API Group to reflect the Pivotal acquisition. I would suggest that the API Group should be updated from projects.pivotal.io
to projects.vmware.com
, this is in line with the general guidance that an API Group should be a subdomain owned by the originating company.
We use kapp delete
to uninstall our deployment in CI. This will remove everything we've installed on the cluster including all projects. the project controller and webhook, and the project CRDs. Unfortunately, this creates a deadlock in the following way:
One solution might be to add the project controller as an owner reference to all created projects.
Another solution may be to drop the finalizer from projects as they already have an owner reference for their corresponding namespaces.
@ashwin-venkatesh @matthewmcnew
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.