projectcapsule / capsule Goto Github PK

View Code? Open in Web Editor NEW

1.5K 22.0 148.0 5.01 MB

Multi-tenancy and policy-based framework for Kubernetes.

Home Page: https://capsule.clastix.io

License: Apache License 2.0

Makefile 1.49% Dockerfile 0.17% Shell 2.20% Go 94.82% Smarty 1.22% JavaScript 0.10%

kubernetes multi-tenancy operator tenant namespaces kubernetes-namespaces multi-tenant-operator kubernetes-operator

capsule's People

Contributors

Stargazers

Watchers

Forkers

ioggstream adheipsingh hvbe wgsateam crixo eionel paolocarta valentinouberti gernest isgasho donhighmsft frodopwns doytsujin paolosanacore rizkyramadhanch unai-ttxu stratio davideimola willcode2surf ludusrusso stg-0 mauriziocasciano rogervaas thien-nguyenthanh ptx96 sftim gyliu513 zcubbs nmarcetic danielchudc kubealex henriaw viveksyngh spagno cawfeecoder abirdcfly rasheedamir zhm022726 brightzheng100 93lucasp mixitgit slushysnowman tanalam2411 abdulhaseeb2 titansmc mohan9456 franklinharry tkuennen uestc-keep nsxaci jayasurya9666 fouadsemaan pandry bgrant0607 goend akasurde alessandroargentieri newrefar bobsongplus abhinandanbaheti yunis-ali gkarthiks chazzrobbz maxgio92 cybernetics mastrogiovanni igorgorovoy openfnord klarrio-asml-dp nitishschauhan lalyos rahimigit jmckee805 al-pragliola psolarcz jgduarte-stratio sagar-jadhav gg-big-org crt-fork prometherion kamranrad1993 h4wkmoon aslafy-z smileisak gabrilabs75 abdennour seifrajhi yarelm firewires frisso mocini-reevo glassofwhiskey nzacharia christiengoergy 315i klarrio jackbrannan thedigitaloctopus rakshpatel slimm609

capsule's Issues

Align the docs to latest bug fixing and enhancements

Review the docs in order to align them to new bug fixing and enhancements. Even if planning to host the documentation to mkdocs or hugo, here we are more focused on provide an updated simple getting started docs.

CI is broken due to kustomize missing Namespace creation

There's an odd behavior during the CI: kustomize doesn't create the expected Namespace (capsule-system).

cd config/manager && /snap/bin/kustomize edit set image controller=quay.io/clastix/capsule:latest
/snap/bin/kustomize build config/default | kubectl apply -f -
namespace/system created
customresourcedefinition.apiextensions.k8s.io/tenants.capsule.clastix.io created
clusterrole.rbac.authorization.k8s.io/capsule-proxy-role created
clusterrole.rbac.authorization.k8s.io/capsule-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/capsule-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/capsule-proxy-rolebinding created
mutatingwebhookconfiguration.admissionregistration.k8s.io/capsule-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/capsule-validating-webhook-configuration created
Error from server (NotFound): error when creating "STDIN": namespaces "capsule-system" not found
Error from server (NotFound): error when creating "STDIN": namespaces "capsule-system" not found
Error from server (NotFound): error when creating "STDIN": namespaces "capsule-system" not found
Error from server (NotFound): error when creating "STDIN": namespaces "capsule-system" not found
Error from server (NotFound): error when creating "STDIN": namespaces "capsule-system" not found
Makefile:51: recipe for target 'deploy' failed

Make configurable the Resource Quota enforcement at tenant level

Describe the feature

It would be useful to make configurable via CLI argument, eg. --force-tenant-quota=true the resource quota enforcement at tenant level. The default should be true and leave the cluster admin to disable the quota resource enforcement at tenant level.

What would the new user story look like?

We would to address case where the cluster admin wants assign resources only at namespace level so the total resource assignment (at tenant level) is statically calculated as number of

assigned_resources_in_namespace x namespaceQuota

For example, with --force-tenant-quota=false, the cluster admin can assign 128GB of RAM per namespace and a namespace quota of 3 to a given tenant. So the permitted amount of RAM for that tenant will be 128GB x 3 = 384GB but each namespace has a strict quota of 128GB.

On the opposite, with --force-tenant-quota=true, the cluster admin can assign 384GB to the tenant and a namespace quota of 3. So the permitted amount of RAM for a single namespace will be 384GB but this amount of RAM has to be shared between all namespaces.

Expected behavior

When --force-tenant-quota=true (default) we have the current behaviour with creation of ResourceQuota at namespace level and cross namespace (tenant) quota check.

When --force-tenant-quota=false the creation of ResourceQuota at namespace level is still in place but no check at cross namespace level (tenant). Only check at namespace level performed by regular kubernetes ResourceQuota admission controller.

Document how to secure the Capsule environment with PSP

Making a CaaS platform means Tenant owner Pods could run on the same node: an attacker could start a privileged one, getting root access on the machine, and obtain privilege escalation.

A document explaining how to set up a simple Pod Security Policy would be great, explaining why a Cluster Admin should deny getting Pods running as privileged, denying to perform host mounting, or host using the host network, etc.

I'm pretty confident that Capsule is not responsible to set this up and must be performed by the Cluster Admin, rather.

How to pronunce it

How to pronunce "capsule"?
It should be pronounced as /ˈkæpsjuːl/ with a bit of french accent :)

How to get rid of Capsule

We need to document how to uninstall the Capsule Operator from the current setup and how to clean everything it installs: pods, web-hooks, roles, crds, ...

Users management

At the current state of Capsule (just the first commit), we don't have multi-user management for the Tenant resources: it means just the owner can operate over all the clusters through the credentials provided by the Cluster Admin as mentioned here.

We need the support of the community to drive this development, so please: let's use this issue to propose a way to implement this feature.

Ensure all images come from a trusted registry declared in the Tenant

Describe the feature

A Tenant could just use a specific set of images from a trusted registry.

Should we use a list of allowed registries?

What would the new user story look like?

setup a Validating Admission webhook for every Pod
retrieve the Tenant from the namespace indexer
check if the image starts with the desired registry name

Expected behavior

if the Pod is running on a non-Capsule namespace, return Allowed without further actions
if the Pod is running on a Capsule namespace with a non allowed registry, return Forbidden

Certificate TLS on initial setup is signed by a different CA

Bug description

On Capsule installation, the certificate used by Capsule to handle webhook requests is signed by a different CA.

Error from server (InternalError): Internal error occurred: failed calling webhook "owner.namespace.capsule.clastix.io": Post https://capsule.capsule-system.svc:443/mutate-v1-namespace-owner-reference?timeout=30s: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Clastix")

Only regenerating it through deletion of the tls.{crt,key} keys of capsule-tls solves the issue.

How to reproduce

Install capsule via kubectl apply -f deploy
Wait for getting it up and running
Create a Namespace and getting the said error

Expected behavior

No needs to generate certificates back manually: it should be totally automatic.

Logs

{"level":"info","ts":1595687878.9992175,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"secret-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1595687879.1016169,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"secret-controller"}
{"level":"info","ts":1595687879.2018154,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"secret-controller","worker count":1}
{"level":"info","ts":1595687879.2021117,"logger":"controller_secret","msg":"Reconciling TLS/CA Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"} 
{"level":"info","ts":1595687885.3081074,"logger":"controller_secret","msg":"Reconciliation completed, processing back in 4319h59m50s","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":1595687885.308173,"logger":"controller_secret","msg":"Reconciling TLS/CA Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls","Request.Namespace":"capsule-system","
Request.Name":"capsule-ca"} 
{"level":"info","ts":1595687885.792041,"logger":"controller_secret","msg":"Handling CA Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls","Request.Namespace":"capsule-system","Request
.Name":"capsule-ca"}
{"level":"info","ts":1595687885.7920654,"logger":"controller_secret","msg":"Updating CA secret with new PEM and RSA","Request.Namespace":"capsule-system","Request.Name":"capsule-tls","Request.Namespace":"ca
psule-system","Request.Name":"capsule-ca"}
{"level":"info","ts":1595687885.9240122,"logger":"controller_secret","msg":"Reconciliation completed, processing back in 87648h0m0s","Request.Namespace":"capsule-system","Request.Name":"capsule-tls","Reques
t.Namespace":"capsule-system","Request.Name":"capsule-ca"}
{"level":"info","ts":1595687885.924053,"logger":"controller_secret","msg":"Reconciling TLS/CA Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls","Request.Namespace":"capsule-system","
Request.Name":"capsule-ca","Request.Namespace":"capsule-system","Request.Name":"capsule-ca"}
{"level":"info","ts":1595687885.9242811,"logger":"controller_secret","msg":"Handling CA Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls","Request.Namespace":"capsule-system","Reques
t.Name":"capsule-ca","Request.Namespace":"capsule-system","Request.Name":"capsule-ca"}
{"level":"info","ts":1595687885.9242892,"logger":"controller_secret","msg":"Updating CA secret with new PEM and RSA","Request.Namespace":"capsule-system","Request.Name":"capsule-tls","Request.Namespace":"ca
psule-system","Request.Name":"capsule-ca","Request.Namespace":"capsule-system","Request.Name":"capsule-ca"} 
{"level":"info","ts":1595687885.9460676,"logger":"controller_secret","msg":"Reconciliation completed, processing back in 87648h0m0s","Request.Namespace":"capsule-system","Request.Name":"capsule-tls","Reques
t.Namespace":"capsule-system","Request.Name":"capsule-ca","Request.Namespace":"capsule-system","Request.Name":"capsule-ca"}

Additional context

Capsule version:

{"level":"info","ts":1595689088.1010258,"logger":"cmd","msg":"Operator Version: 0.0.1"}
{"level":"info","ts":1595689088.1010656,"logger":"cmd","msg":"Go Version: go1.13.8"}
{"level":"info","ts":1595689088.101077,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1595689088.1010873,"logger":"cmd","msg":"Version of operator-sdk: v0.18.1"}

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.4", GitCommit:"224be7bdce5a9dd0c2fd0d46b83865648e2fe0ba", GitTreeState:"clean", BuildDate:"2019-12-11T12:47:40Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-30T20:19:45Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Enable short name for Tenant

Describe the feature

We are lazy people, typing kubectl get tenants is too much hassle.

Would be great having tnt as shortname: kubectl get tnt

What would the new user story look like?

Annotate the Tenant CRD with the Kubebuilder one (should be shortName)

Expected behavior

Getting Tenant resources via tnt resource name using kubectl rather than typing tenant{s}

The controller namespace is hard-coded

Bug description

The namespace of Capsule controller is hard-coded to capsule-system.

How to reproduce

Use the Helm chart to deploy Capsule and specify a different namespace than capsule-system

$ helm install capsule capsule-helm-chart --set force_tenant_prefix=false -n foo

Expected behavior

The capsule controlled should be deployed on the requested namespace

Logs

Additional context

Add an optional external admission controller to Capsule Operator

Describe the feature

Capsule implements its own set of Admission Controllers through the Dynamic Admission Controller, providing callbacks to add further validation or resource patching. Currently these admission controllers address only a fixed set of use cases.

As long term evolution for this project we can evaluate to make the admission controllers (both the mutating and validating flavours) programmable in order to cover more and more use cases.

A viable option would be to include a Policy Engine, eg. Open Policy Agent (OPA), as embedded library and provide a programmable interface in Rego, see https://www.openpolicyagent.org/docs/latest/integration/#integrating-with-the-go-api. Since this will create another dependancy we have evaluate with care this option.
~~Let's to see also feedbacks from community.~~

e2e tests against multiple Kubernetes versions

Right now, the Capsule e2e test suite is running only on Kubernetes 1.18: would be great to perform it over the latest available releases, also to discover eventually incompatibilities.

Ability for the tenant owner to list namespaces in the tenant

Describe the feature

Ability for the tenant owner to list namespaces in the tenant. Currently, the tenant owner can create and delete namespaces in his tenants but listing them is prevented by RBAC. And this is correct, according to RBAC rules, otherwise the tenant owner can list all the namespaces in the cluster (namespace is a cluster-wide resource).

Generally speaking, the ability to list only the owned namespaces has been long discussed in the community, see kubernetes/kubernetes#48537, kubernetes/kubernetes#58262, and kubernetes/kubernetes#61958 and the objections described there still hold.

As reported in this comment there are no ACL-filtered APIs in core kubernetes. Filtering resources through ACL breaks resourceVersion semantics that allow watch to function properly.

What would the new user story look like?

As tenant owner, I would be able to create, list, and delete namespaces in my tenant.

Expected behavior

See above

Evalute to upgrade to OperatorSDK 0.19

Operator SDK 0.19 has been released two weeks ago and is providing more tight integration with Kubebuilder and a better webhook scaffold generation and management, too.

We should consider migrating the actual code-base in order to use this latest version, also to get automated YAML templates generation according to the annotations/code generation comments instead of manually reporting it and follow the opinionated workflow to allow more contributions.

Capsule always assumes first found tenant as an owner for namespace

Current implementation of owner_refrence webhook use following logic to set tenant for a namespace
https://github.com/clastix/capsule/blob/a99153cbe70432d7e21ad5a2610ba737b1ba27d3/pkg/webhook/owner_reference/patching.go#L84-L93
So basically we find all Tenants for a user\group who created a namespace, and select first one from it (and I think that this logic should be rewritten if future because it's a bit strange for me - a good option will be to set force-tenant-prefix by default, so all this magic will gone)

Let the tenant owner to know the assigned Ingress Classes

Describe the feature

Let the tenant owner (and other any tenant user) to know the list of Ingress Classes available in the tenant. Currently, because of RBAC, the tenant owner cannot get the available Ingress classes. The suggested approach is to put a readonly annotation in each namespace of the tenant. The annotation reports the list of Ingress Classes assigned to the tenant. For example: capsule.clastix.io/ingress-classes: "haproxy,nginx,standard".

What would the new user story look like?

What are the prerequisites for this?
Tenant owner creates a new namespace
Tenant owner check the annotations in the namespace and get list of assigned Ingress Classes
Tenant owner creates an ingress in the namespace using the one of the available Ingress Classes
Capsule controller is happy of this and lets the ingress to be created.

Expected behavior

As tenant owner (or as any other tenant user) I want to know which Ingress Classes are available in my tenant.

Adding Owner and Age as additional printer columns

Describe the feature

An additional printer column reporting the Tenant owner name would be great.

What would the new user story look like?

Implementation is really easy, just need to add the kubebuilderprintcolumn annotation into the Tenant type.

Feel free to add a diagram if that helps explain things.

Expected behavior

# kubectl get tenants.capsule.clastix.io
NAME   NAMESPACE QUOTA   NAMESPACE COUNT   OWNER
oil    3                 3                 alice

Scaffold the e2e testing during CI with Kind

Describe the feature

KinD is a top-notch project to perform e2e tests during the Continous Integration.

Actually, Capsule doesn't provide a real test suite and probably also the current code-base doesn't allow to perform other kinds of tests (except for the Certificate Authority package).

Would be great implementing full e2e during CI with GitHub Actions, also evaluating a little refactoring to adopt other kinds of tests.

What would the new user story look like?

Try to set up KinD action
Write a simple e2e test

Further tests can be tracked by future issues.

Expected behavior

Getting merge blocked in case of bugged PR.

Ingress Class is not correctly allowed with Kubernetes 1.17

Bug description

It seems the e2e test suite is failing (spec ingress_class_test.go) when running on Kubernetes 1.17 with Ingress Class specified through .spec.ingressClassName field rather than annotation (API group: ingresses.extensions.k8s.io/v1beta1).

Although using an available Ingress Class as the Tenant specified, the webhook is returning always an errored response.

The issue appeared during the e2e test suite implementation for the CI

How to reproduce

make e2e enduring to use Kubernetes 1.17

Expected behavior

Despite the Kubernetes version, Capsule should allow Ingress resources with Ingress Class specified in any available way.

Logs

From the test suite:

      <*errors.StatusError | 0xc000171900>: {
          ErrStatus: {
              TypeMeta: {Kind: "", APIVersion: ""},
              ListMeta: {
                  SelfLink: "",
                  ResourceVersion: "",
                  Continue: "",
                  RemainingItemCount: nil,
              },
              Status: "Failure",
              Message: "admission webhook \"extensions.ingress.capsule.clastix.io\" denied the request: A valid Ingress Class must be used",
              Reason: "",
              Details: nil,
              Code: 400,
          },
      }
      admission webhook "extensions.ingress.capsule.clastix.io" denied the request: A valid Ingress Class must be used

From the manager:

2020-08-22T08:45:56.935Z    DEBUG   controller-runtime.webhook.webhooks     received request        {"webhook": "/validating-v1-extensions-ingress", "UID": "df5cc1ef-4bb5-4371-b8b5-1c76ceb95479", "kind": "extensions/v1beta1, Kind=Ingress", "resource": {"group":"extensions","version":"v1beta1","resource":"ingresses"}}
2020-08-22T08:45:56.935Z    DEBUG   controller-runtime.webhook.webhooks     wrote response  {"webhook": "/validating-v1-extensions-ingress", "UID": "df5cc1ef-4bb5-4371-b8b5-1c76ceb95479", "allowed": false, "result": {}, "resultError": "got runtime.Object without object metadata: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:,Message:A valid Ingress Class must be used,Reason:,Details:nil,Code:400,}"}

Additional context

Capsule version: N.R.
Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.4", GitCommit:"224be7bdce5a9dd0c2fd0d46b83865648e2fe0ba", GitTreeState:"clean", BuildDate:"2019-12-11T12:47:40Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/
amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2020-01-14T00:09:19Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/a
md64"}

Webhook closures not working

Bug description

Closures not working in webhooks middleware

How to reproduce

Steps to reproduce the behaviour:

build capsule from master, load it to kind
try to create namespace as cluster-admin
You will got following error

k create ns bla
Error from server (You do not have any Tenant assigned: please, reach out the system administrators): admission webhook "owner.namespace.capsule.clastix.io" denied the request: You do not have any Tenant assigned: please, reach out the system administrators

Expected behavior

Namespace should be created, as kind cluster-admin is not a member of capsule-user-group

Support for additional labels and annotations to be inherited by tenant namespaces

Describe the feature

The idea is to leave an additional section in the Tenant manifest where the Cluster admin can specify additional labels and/or annotations to apply to the namespaces belonging to the tenant. Initially, there are no specific semantics assigned to these labels, they just will be assigned to the namespaces in the tenant. This can be useful to the adopters of Capsule to implement their specific use cases.

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: oil
spec:
  additionalNamespaceLabels:
    capsule.clastix.io/backup: "true"
    capsule.clastix.io/billable: "true"
    capsule.clastix.io/do_stuff: "once"
...

What would the new user story look like?

The cluster admin creates a tenant with this additional fields
Tenant owner creates a new Namespace
All the namespaces created in this tenant will inherit the labels
Depending on the case, additional business logics can be implemented by using the labels.

Please note the specific business logic is not part of the Capsule. Instead it is left to the cluster admin and or to the tenant owner.

Expected behavior

All the namespaces created in the tenant inherit the labels.

Helm 3 chart

A Helm 3 chart would be useful.

Support for Group as RoleBinding.Subject.Kind created by Capsule

Describe the feature

We always use groups for rbac, but right now Tenant Owner is always set to a User. So i would like to add flag --use-groups-as-tenant-owner , which will modify func (r *TenantReconciler) ownerRoleBinding(tenant *capsulev1alpha1.Tenant) function, so

	s := []rbacv1.Subject{
		{
			Kind: "User",
			Name: tenant.Spec.Owner,
		},
	}

will be created with Kind:"Group"

Let the tenant owner to know the assigned Storage Classes

Describe the feature

Let the tenant owner (and other any tenant user) to know the list of Storage Classes available in the tenant. Currently, because of RBAC, the tenant owner cannot get the available storage classes. The suggested approach is to put a readonly annotation in each namespace of the tenant. The annotation reports the list of Storage Classes assigned to the tenant. For example: capsule.clastix.io/storage-classes: "ceph,longhorn,standard".

What would the new user story look like?

What are the prerequisites for this?
Tenant owner creates a new namespace
Tenant owner check the annotations in the namespace and get list of assigned Storage Classes
Tenant owner creates a pvc in the namespace using the one of the available Storage Classes
Capsule controller is happy of this and lets the pvc to be created.

Expected behavior

As tenant owner (or as any other tenant user) I want to know which Storage Classes are available in my tenant.

Tenant Resource Quota admission controller doesn't block resources creation

Bug description

Tenant Resource Quota admission controller doesn't block resources creation

How to reproduce

As cluster admin:

cmp~/Shared/capsule$ kubectl get tenants
NAME   NAMESPACE QUOTA   NAMESPACE COUNT   OWNER   AGE
oil    3                 3                 alice   161m

A tenant owner:

$ kubectl -n oil-production get quota
NAME            AGE    REQUEST                                      LIMIT
capsule-oil-0   161m   requests.cpu: 0/8, requests.memory: 0/16Gi   limits.cpu: 0/8, limits.memory: 0/16Gi
capsule-oil-1   161m   pods: 0/10                                   
capsule-oil-2   161m   requests.storage: 0/100Gi                    
$ kubectl -n oil-development get quota
NAME            AGE    REQUEST                                      LIMIT
capsule-oil-0   161m   requests.cpu: 0/8, requests.memory: 0/16Gi   limits.cpu: 0/8, limits.memory: 0/16Gi
capsule-oil-1   161m   pods: 0/10                                   
capsule-oil-2   161m   requests.storage: 0/100Gi                    
$ kubectl -n oil-test get quota
NAME            AGE    REQUEST                                      LIMIT
capsule-oil-0   161m   requests.cpu: 0/8, requests.memory: 0/16Gi   limits.cpu: 0/8, limits.memory: 0/16Gi
capsule-oil-1   161m   pods: 0/10                                   
capsule-oil-2   161m   requests.storage: 0/100Gi

check used quota at tenant level

$ kubectl -n oil-production get quota capsule-oil-1 -o yaml | grep quota.capsule.clastix.io/used-pods
    quota.capsule.clastix.io/used-pods: "0"

scale up pods

kubectl -n oil-production scale rs nginx --replicas=3
kubectl -n oil-development scale rs nginx --replicas=3
kubectl -n oil-test scale rs nginx --replicas=5

and check quota:

kubectl -n oil-production get quota capsule-oil-1 -o yaml | grep quota.capsule.clastix.io/used-pods
    quota.capsule.clastix.io/used-pods: "11"

kubectl -n oil-production describe quota capsule-oil-1 
Name:       capsule-oil-1
Namespace:  oil-production
Resource    Used  Hard
--------    ----  ----
pods        3     3                   <-- see hard quota!!!

kubectl -n oil-test describe quota capsule-oil-1 
Name:       capsule-oil-1
Namespace:  oil-test
Resource    Used  Hard
--------    ----  ----
pods        5     5

Expected behavior

Tenant Resource Quota admission controller should block resource creation crossing the set quota.

Logs

Capsule controller has been started with following args but I do see only INFO

      containers:
      - args:
        - --metrics-addr=127.0.0.1:8080
        - --enable-leader-election
        - --zap-log-level=10
        - --zap-devel
        command:
        - /manager
        image: quay.io/clastix/capsule:latest

2020-08-10T10:22:00.793Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 0   {"Request.Name": "oil"}
2020-08-10T10:22:00.942Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-test"}
2020-08-10T10:22:00.942Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:22:00.942Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:22:01.093Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-test"}
2020-08-10T10:22:01.093Z        INFO    controllers.Tenant      Ensuring RoleBinding for owner  {"Request.Name": "oil"}
2020-08-10T10:22:01.096Z        INFO    controllers.Tenant      Role Binding sync result: updated       {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-development"}
2020-08-10T10:22:01.096Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-development"}
2020-08-10T10:22:01.097Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-production"}
2020-08-10T10:22:01.097Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-production"}
2020-08-10T10:22:01.097Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-test"}
2020-08-10T10:22:01.097Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-test"}
2020-08-10T10:22:01.097Z        INFO    controllers.Tenant      Ensuring Namespace count        {"Request.Name": "oil"}
2020-08-10T10:22:01.107Z        INFO    controllers.Tenant      Tenant reconciling completed    {"Request.Name": "oil"}
2020-08-10T10:32:29.193Z        INFO    controllers.Tenant      Starting processing of Namespaces       {"Request.Name": "oil", "items": 3}
2020-08-10T10:32:29.203Z        INFO    controllers.Tenant      Starting processing of Network Policies {"Request.Name": "oil", "items": 1}
2020-08-10T10:32:29.204Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:29.220Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:29.220Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:29.252Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:29.252Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:29.262Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:29.262Z        INFO    controllers.Tenant      Starting processing of Node Selector    {"Request.Name": "oil"}
2020-08-10T10:32:29.263Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-development"}
2020-08-10T10:32:29.263Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-production"}
2020-08-10T10:32:29.263Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-test"}
2020-08-10T10:32:29.263Z        INFO    controllers.Tenant      Starting processing of Limit Ranges     {"Request.Name": "oil", "items": 1}
2020-08-10T10:32:29.263Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:32:29.290Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:29.290Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:32:29.330Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:29.330Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:32:29.340Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:29.341Z        INFO    controllers.Tenant      Starting processing of Resource Quotas  {"Request.Name": "oil", "items": 3}
2020-08-10T10:32:29.342Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:32:29.358Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:32:29.358Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 30Mi     {"Request.Name": "oil"}
2020-08-10T10:32:29.380Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:32:29.381Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 600m  {"Request.Name": "oil"}
2020-08-10T10:32:29.459Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:32:29.459Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 300Mi      {"Request.Name": "oil"}
2020-08-10T10:32:29.485Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:32:29.485Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 300m        {"Request.Name": "oil"}
2020-08-10T10:32:29.519Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:29.520Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:32:29.520Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 3   {"Request.Name": "oil"}
2020-08-10T10:32:29.546Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-development"}
2020-08-10T10:32:29.546Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:32:29.546Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:32:29.565Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-development"}
2020-08-10T10:32:29.565Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:32:29.568Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:32:29.568Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 30Mi     {"Request.Name": "oil"}
2020-08-10T10:32:29.579Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:32:29.579Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 600m  {"Request.Name": "oil"}
2020-08-10T10:32:29.595Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:32:29.595Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 300Mi      {"Request.Name": "oil"}
2020-08-10T10:32:29.607Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:32:29.607Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 300m        {"Request.Name": "oil"}
2020-08-10T10:32:29.631Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:29.631Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:32:29.631Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 3   {"Request.Name": "oil"}
2020-08-10T10:32:29.748Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-production"}
2020-08-10T10:32:29.748Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:32:29.748Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:32:29.919Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-production"}
2020-08-10T10:32:29.919Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:32:29.950Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:32:29.950Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 30Mi     {"Request.Name": "oil"}
2020-08-10T10:32:30.098Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:32:30.098Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 600m  {"Request.Name": "oil"}
2020-08-10T10:32:30.248Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:32:30.248Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 300Mi      {"Request.Name": "oil"}
2020-08-10T10:32:30.398Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:32:30.398Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 300m        {"Request.Name": "oil"}
2020-08-10T10:32:30.548Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:30.548Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:32:30.548Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 3   {"Request.Name": "oil"}
2020-08-10T10:32:30.697Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-test"}
2020-08-10T10:32:30.697Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:32:30.697Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:32:30.850Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-test"}
2020-08-10T10:32:30.850Z        INFO    controllers.Tenant      Ensuring RoleBinding for owner  {"Request.Name": "oil"}
2020-08-10T10:32:30.856Z        INFO    controllers.Tenant      Role Binding sync result: updated       {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-test"}
2020-08-10T10:32:30.856Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-test"}
2020-08-10T10:32:30.857Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-development"}
2020-08-10T10:32:30.857Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-development"}
2020-08-10T10:32:30.857Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-production"}
2020-08-10T10:32:30.857Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-production"}
2020-08-10T10:32:30.857Z        INFO    controllers.Tenant      Ensuring Namespace count        {"Request.Name": "oil"}
2020-08-10T10:32:30.871Z        INFO    controllers.Tenant      Tenant reconciling completed    {"Request.Name": "oil"}
2020-08-10T10:32:30.871Z        INFO    controllers.Tenant      Starting processing of Namespaces       {"Request.Name": "oil", "items": 3}
2020-08-10T10:32:30.884Z        INFO    controllers.Tenant      Starting processing of Network Policies {"Request.Name": "oil", "items": 1}
2020-08-10T10:32:30.884Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:30.887Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:30.887Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:30.891Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:30.891Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:30.894Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:30.894Z        INFO    controllers.Tenant      Starting processing of Node Selector    {"Request.Name": "oil"}
2020-08-10T10:32:30.894Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-development"}
2020-08-10T10:32:30.895Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-production"}
2020-08-10T10:32:30.895Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-test"}
2020-08-10T10:32:30.895Z        INFO    controllers.Tenant      Starting processing of Limit Ranges     {"Request.Name": "oil", "items": 1}
2020-08-10T10:32:30.895Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:32:30.898Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:30.898Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:32:30.903Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:30.903Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:32:30.907Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:30.907Z        INFO    controllers.Tenant      Starting processing of Resource Quotas  {"Request.Name": "oil", "items": 3}
2020-08-10T10:32:30.907Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:32:30.912Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:32:30.912Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 300Mi      {"Request.Name": "oil"}
2020-08-10T10:32:31.050Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:32:31.050Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 300m        {"Request.Name": "oil"}
2020-08-10T10:32:31.198Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:32:31.198Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 30Mi     {"Request.Name": "oil"}
2020-08-10T10:32:31.347Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:32:31.347Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 600m  {"Request.Name": "oil"}
2020-08-10T10:32:31.497Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:31.498Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:32:31.498Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 3   {"Request.Name": "oil"}
2020-08-10T10:32:31.647Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-development"}
2020-08-10T10:32:31.648Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:32:31.648Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:32:31.797Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-development"}
2020-08-10T10:32:31.797Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:32:31.846Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:32:31.846Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 30Mi     {"Request.Name": "oil"}
2020-08-10T10:32:31.997Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:32:31.997Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 600m  {"Request.Name": "oil"}
2020-08-10T10:32:32.147Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:32:32.147Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 300Mi      {"Request.Name": "oil"}
2020-08-10T10:32:32.303Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:32:32.303Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 300m        {"Request.Name": "oil"}
2020-08-10T10:32:32.450Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:32.451Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:32:32.451Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 3   {"Request.Name": "oil"}
2020-08-10T10:32:32.600Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-production"}
2020-08-10T10:32:32.601Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:32:32.601Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:32:32.747Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-production"}
2020-08-10T10:32:32.747Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:32:32.795Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:32:32.796Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 30Mi     {"Request.Name": "oil"}
2020-08-10T10:32:32.947Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:32:32.947Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 600m  {"Request.Name": "oil"}
2020-08-10T10:32:33.097Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:32:33.097Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 300Mi      {"Request.Name": "oil"}
2020-08-10T10:32:33.247Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:32:33.247Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 300m        {"Request.Name": "oil"}
2020-08-10T10:32:33.398Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:33.398Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:32:33.398Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 3   {"Request.Name": "oil"}
2020-08-10T10:32:33.546Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-test"}
2020-08-10T10:32:33.547Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:32:33.547Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:32:33.697Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-test"}
2020-08-10T10:32:33.697Z        INFO    controllers.Tenant      Ensuring RoleBinding for owner  {"Request.Name": "oil"}
2020-08-10T10:32:33.701Z        INFO    controllers.Tenant      Role Binding sync result: updated       {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-development"}
2020-08-10T10:32:33.701Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-development"}
2020-08-10T10:32:33.702Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-production"}
2020-08-10T10:32:33.702Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-production"}
2020-08-10T10:32:33.702Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-test"}
2020-08-10T10:32:33.702Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-test"}
2020-08-10T10:32:33.702Z        INFO    controllers.Tenant      Ensuring Namespace count        {"Request.Name": "oil"}
2020-08-10T10:32:33.712Z        INFO    controllers.Tenant      Tenant reconciling completed    {"Request.Name": "oil"}
2020-08-10T10:32:43.303Z        INFO    controllers.Tenant      Starting processing of Namespaces       {"Request.Name": "oil", "items": 3}
2020-08-10T10:32:43.357Z        INFO    controllers.Tenant      Starting processing of Network Policies {"Request.Name": "oil", "items": 1}
2020-08-10T10:32:43.358Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:43.388Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:43.388Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:43.410Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:43.410Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:43.427Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:43.427Z        INFO    controllers.Tenant      Starting processing of Node Selector    {"Request.Name": "oil"}
2020-08-10T10:32:43.427Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-development"}
2020-08-10T10:32:43.428Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-production"}
2020-08-10T10:32:43.428Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-test"}
2020-08-10T10:32:43.428Z        INFO    controllers.Tenant      Starting processing of Limit Ranges     {"Request.Name": "oil", "items": 1}
2020-08-10T10:32:43.428Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:32:43.441Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:43.441Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:32:43.447Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:43.447Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:32:43.460Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:43.460Z        INFO    controllers.Tenant      Starting processing of Resource Quotas  {"Request.Name": "oil", "items": 3}
2020-08-10T10:32:43.460Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:32:43.463Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:32:43.463Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 1200m {"Request.Name": "oil"}
2020-08-10T10:32:43.499Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:32:43.499Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 600Mi      {"Request.Name": "oil"}
2020-08-10T10:32:43.513Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:32:43.513Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 600m        {"Request.Name": "oil"}
2020-08-10T10:32:43.534Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:32:43.535Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 60Mi     {"Request.Name": "oil"}
2020-08-10T10:32:43.547Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:43.547Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:32:43.547Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 6   {"Request.Name": "oil"}
2020-08-10T10:32:43.579Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-development"}
2020-08-10T10:32:43.579Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:32:43.579Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:32:43.603Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-development"}
2020-08-10T10:32:43.603Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:32:43.609Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:32:43.609Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 600m        {"Request.Name": "oil"}
2020-08-10T10:32:43.621Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:32:43.621Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 60Mi     {"Request.Name": "oil"}
2020-08-10T10:32:43.632Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:32:43.632Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 1200m {"Request.Name": "oil"}
2020-08-10T10:32:43.644Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:32:43.644Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 600Mi      {"Request.Name": "oil"}
2020-08-10T10:32:43.657Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:43.657Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:32:43.657Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 6   {"Request.Name": "oil"}
2020-08-10T10:32:43.717Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-production"}
2020-08-10T10:32:43.717Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:32:43.717Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:32:43.898Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-production"}
2020-08-10T10:32:43.899Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:32:43.914Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:32:43.914Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 1200m {"Request.Name": "oil"}
2020-08-10T10:32:44.066Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:32:44.066Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 600Mi      {"Request.Name": "oil"}
2020-08-10T10:32:44.219Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:32:44.219Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 600m        {"Request.Name": "oil"}
2020-08-10T10:32:44.366Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:32:44.366Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 60Mi     {"Request.Name": "oil"}
2020-08-10T10:32:44.517Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:44.517Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:32:44.517Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 6   {"Request.Name": "oil"}
2020-08-10T10:32:44.667Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-test"}
2020-08-10T10:32:44.667Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:32:44.667Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:32:44.816Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-test"}
2020-08-10T10:32:44.816Z        INFO    controllers.Tenant      Ensuring RoleBinding for owner  {"Request.Name": "oil"}
2020-08-10T10:32:44.820Z        INFO    controllers.Tenant      Role Binding sync result: updated       {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-development"}
2020-08-10T10:32:44.820Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-production"}
2020-08-10T10:32:44.821Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-production"}
2020-08-10T10:32:44.821Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-test"}
2020-08-10T10:32:44.821Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-test"}
2020-08-10T10:32:44.821Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-development"}
2020-08-10T10:32:44.821Z        INFO    controllers.Tenant      Ensuring Namespace count        {"Request.Name": "oil"}
2020-08-10T10:32:44.833Z        INFO    controllers.Tenant      Tenant reconciling completed    {"Request.Name": "oil"}
2020-08-10T10:32:44.833Z        INFO    controllers.Tenant      Starting processing of Namespaces       {"Request.Name": "oil", "items": 3}
2020-08-10T10:32:44.845Z        INFO    controllers.Tenant      Starting processing of Network Policies {"Request.Name": "oil", "items": 1}
2020-08-10T10:32:44.845Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:44.850Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:44.850Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:44.854Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:44.854Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:44.858Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:44.858Z        INFO    controllers.Tenant      Starting processing of Node Selector    {"Request.Name": "oil"}
2020-08-10T10:32:44.859Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-development"}
2020-08-10T10:32:44.859Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-production"}
2020-08-10T10:32:44.859Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-test"}
2020-08-10T10:32:44.859Z        INFO    controllers.Tenant      Starting processing of Limit Ranges     {"Request.Name": "oil", "items": 1}
2020-08-10T10:32:44.859Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:32:44.863Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:44.863Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:32:44.866Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:44.866Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:32:44.871Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:44.871Z        INFO    controllers.Tenant      Starting processing of Resource Quotas  {"Request.Name": "oil", "items": 3}
2020-08-10T10:32:44.871Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:32:44.876Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:32:44.876Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 1200m {"Request.Name": "oil"}
2020-08-10T10:32:45.016Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:32:45.016Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 600Mi      {"Request.Name": "oil"}
2020-08-10T10:32:45.166Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:32:45.166Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 600m        {"Request.Name": "oil"}
2020-08-10T10:32:45.315Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:32:45.315Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 60Mi     {"Request.Name": "oil"}
2020-08-10T10:32:45.466Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:45.466Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:32:45.466Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 6   {"Request.Name": "oil"}
2020-08-10T10:32:45.615Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-development"}
2020-08-10T10:32:45.615Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:32:45.615Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:32:45.765Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-development"}
2020-08-10T10:32:45.765Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:32:45.814Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:32:45.814Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 600m        {"Request.Name": "oil"}
2020-08-10T10:32:45.965Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:32:45.965Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 60Mi     {"Request.Name": "oil"}
2020-08-10T10:32:46.115Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:32:46.115Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 1200m {"Request.Name": "oil"}
2020-08-10T10:32:46.265Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:32:46.265Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 600Mi      {"Request.Name": "oil"}
2020-08-10T10:32:46.416Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:46.416Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:32:46.416Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 6   {"Request.Name": "oil"}
2020-08-10T10:32:46.564Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-production"}
2020-08-10T10:32:46.565Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:32:46.565Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:32:46.726Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-production"}
2020-08-10T10:32:46.726Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:32:46.764Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:32:46.764Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 600m        {"Request.Name": "oil"}
2020-08-10T10:32:46.918Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:32:46.918Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 60Mi     {"Request.Name": "oil"}
2020-08-10T10:32:47.065Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:32:47.065Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 1200m {"Request.Name": "oil"}
2020-08-10T10:32:47.215Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:32:47.215Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 600Mi      {"Request.Name": "oil"}
2020-08-10T10:32:47.366Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:47.366Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:32:47.366Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 6   {"Request.Name": "oil"}
2020-08-10T10:32:47.515Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-test"}
2020-08-10T10:32:47.516Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:32:47.516Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:32:47.667Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-test"}
2020-08-10T10:32:47.667Z        INFO    controllers.Tenant      Ensuring RoleBinding for owner  {"Request.Name": "oil"}
2020-08-10T10:32:47.673Z        INFO    controllers.Tenant      Role Binding sync result: updated       {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-development"}
2020-08-10T10:32:47.674Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-production"}
2020-08-10T10:32:47.674Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-production"}
2020-08-10T10:32:47.674Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-test"}
2020-08-10T10:32:47.674Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-test"}
2020-08-10T10:32:47.674Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-development"}
2020-08-10T10:32:47.674Z        INFO    controllers.Tenant      Ensuring Namespace count        {"Request.Name": "oil"}
2020-08-10T10:32:47.683Z        INFO    controllers.Tenant      Tenant reconciling completed    {"Request.Name": "oil"}
2020-08-10T10:33:11.273Z        INFO    controllers.Tenant      Starting processing of Namespaces       {"Request.Name": "oil", "items": 3}
2020-08-10T10:33:11.283Z        INFO    controllers.Tenant      Starting processing of Network Policies {"Request.Name": "oil", "items": 1}
2020-08-10T10:33:11.284Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:11.292Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:33:11.292Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:11.318Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:33:11.318Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:11.347Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:33:11.347Z        INFO    controllers.Tenant      Starting processing of Node Selector    {"Request.Name": "oil"}
2020-08-10T10:33:11.347Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-development"}
2020-08-10T10:33:11.347Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-production"}
2020-08-10T10:33:11.347Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-test"}
2020-08-10T10:33:11.347Z        INFO    controllers.Tenant      Starting processing of Limit Ranges     {"Request.Name": "oil", "items": 1}
2020-08-10T10:33:11.347Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:33:11.353Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:33:11.353Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:33:11.361Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:33:11.362Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:33:11.375Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:33:11.375Z        INFO    controllers.Tenant      Starting processing of Resource Quotas  {"Request.Name": "oil", "items": 3}
2020-08-10T10:33:11.375Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:33:11.414Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:33:11.414Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 1800m {"Request.Name": "oil"}
2020-08-10T10:33:11.507Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:33:11.507Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 900Mi      {"Request.Name": "oil"}
2020-08-10T10:33:11.608Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:33:11.609Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 900m        {"Request.Name": "oil"}
2020-08-10T10:33:11.654Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:33:11.654Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 90Mi     {"Request.Name": "oil"}
2020-08-10T10:33:11.730Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:33:11.730Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:33:11.730Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 11  {"Request.Name": "oil"}
2020-08-10T10:33:11.769Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-development"}
2020-08-10T10:33:11.769Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:33:11.769Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:33:11.806Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-development"}
2020-08-10T10:33:11.806Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:33:11.822Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:33:11.822Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 2200m {"Request.Name": "oil"}
2020-08-10T10:33:11.869Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:33:11.869Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 1100Mi     {"Request.Name": "oil"}
2020-08-10T10:33:11.889Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:33:11.889Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 1100m       {"Request.Name": "oil"}
2020-08-10T10:33:11.925Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:33:11.925Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 110Mi    {"Request.Name": "oil"}
2020-08-10T10:33:11.939Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:33:11.939Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:33:11.939Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 11  {"Request.Name": "oil"}
2020-08-10T10:33:12.000Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-production"}
2020-08-10T10:33:12.001Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:33:12.001Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:33:12.010Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-production"}
2020-08-10T10:33:12.010Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:33:12.014Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:33:12.014Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 2200m {"Request.Name": "oil"}
2020-08-10T10:33:12.135Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:33:12.135Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 1100Mi     {"Request.Name": "oil"}
2020-08-10T10:33:12.282Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:33:12.282Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 1100m       {"Request.Name": "oil"}
2020-08-10T10:33:12.434Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:33:12.434Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 110Mi    {"Request.Name": "oil"}
2020-08-10T10:33:12.599Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:33:12.599Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:33:12.599Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 11  {"Request.Name": "oil"}
2020-08-10T10:33:12.731Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-test"}
2020-08-10T10:33:12.731Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:33:12.731Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:33:12.881Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-test"}
2020-08-10T10:33:12.882Z        INFO    controllers.Tenant      Ensuring RoleBinding for owner  {"Request.Name": "oil"}
2020-08-10T10:33:12.886Z        INFO    controllers.Tenant      Role Binding sync result: updated       {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-development"}
2020-08-10T10:33:12.886Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-development"}
2020-08-10T10:33:12.886Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-production"}
2020-08-10T10:33:12.887Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-production"}
2020-08-10T10:33:12.887Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-test"}
2020-08-10T10:33:12.887Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-test"}
2020-08-10T10:33:12.887Z        INFO    controllers.Tenant      Ensuring Namespace count        {"Request.Name": "oil"}
2020-08-10T10:33:12.898Z        INFO    controllers.Tenant      Tenant reconciling completed    {"Request.Name": "oil"}
2020-08-10T10:33:12.898Z        INFO    controllers.Tenant      Starting processing of Namespaces       {"Request.Name": "oil", "items": 3}
2020-08-10T10:33:12.909Z        INFO    controllers.Tenant      Starting processing of Network Policies {"Request.Name": "oil", "items": 1}
2020-08-10T10:33:12.909Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:12.914Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:33:12.914Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:12.917Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:33:12.917Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:12.921Z        INFO    controllers.Tenant      Network Policy sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:33:12.921Z        INFO    controllers.Tenant      Starting processing of Node Selector    {"Request.Name": "oil"}
2020-08-10T10:33:12.921Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-development"}
2020-08-10T10:33:12.921Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-production"}
2020-08-10T10:33:12.921Z        INFO    controllers.Tenant      Namespace Node  sync result: unchanged  {"Request.Name": "oil", "name": "oil-test"}
2020-08-10T10:33:12.921Z        INFO    controllers.Tenant      Starting processing of Limit Ranges     {"Request.Name": "oil", "items": 1}
2020-08-10T10:33:12.921Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:33:12.925Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:33:12.925Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:33:12.928Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:33:12.928Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/limit-range notin (0)    {"Request.Name": "oil"}
2020-08-10T10:33:12.931Z        INFO    controllers.Tenant      LimitRange sync result: unchanged       {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:33:12.931Z        INFO    controllers.Tenant      Starting processing of Resource Quotas  {"Request.Name": "oil", "items": 3}
2020-08-10T10:33:12.931Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:33:12.935Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:33:12.935Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 1100Mi     {"Request.Name": "oil"}
2020-08-10T10:33:13.082Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:33:13.082Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 1100m       {"Request.Name": "oil"}
2020-08-10T10:33:13.237Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:33:13.237Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 110Mi    {"Request.Name": "oil"}
2020-08-10T10:33:13.399Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:33:13.399Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 2200m {"Request.Name": "oil"}
2020-08-10T10:33:13.531Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:33:13.532Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:33:13.532Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 11  {"Request.Name": "oil"}
2020-08-10T10:33:13.680Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-development"}
2020-08-10T10:33:13.681Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:33:13.681Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:33:13.830Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-development"}
2020-08-10T10:33:13.830Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:33:13.880Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:33:13.880Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 1100Mi     {"Request.Name": "oil"}
2020-08-10T10:33:14.032Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:33:14.032Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 1100m       {"Request.Name": "oil"}
2020-08-10T10:33:14.181Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:33:14.181Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 110Mi    {"Request.Name": "oil"}
2020-08-10T10:33:14.330Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:33:14.330Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 2200m {"Request.Name": "oil"}
2020-08-10T10:33:14.481Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:33:14.482Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:33:14.482Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 11  {"Request.Name": "oil"}
2020-08-10T10:33:14.630Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-production"}
2020-08-10T10:33:14.630Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:33:14.630Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:33:14.781Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-production"}
2020-08-10T10:33:14.781Z        INFO    controllers.Tenant      Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)     {"Request.Name": "oil"}
2020-08-10T10:33:14.829Z        INFO    controllers.Tenant      Desired hard limits.cpu quota is 8      {"Request.Name": "oil"}
2020-08-10T10:33:14.829Z        INFO    controllers.Tenant      Computed limits.cpu quota for the whole Tenant is 2200m {"Request.Name": "oil"}
2020-08-10T10:33:14.981Z        INFO    controllers.Tenant      Desired hard limits.memory quota is 16Gi        {"Request.Name": "oil"}
2020-08-10T10:33:14.981Z        INFO    controllers.Tenant      Computed limits.memory quota for the whole Tenant is 1100Mi     {"Request.Name": "oil"}
2020-08-10T10:33:15.130Z        INFO    controllers.Tenant      Desired hard requests.cpu quota is 8    {"Request.Name": "oil"}
2020-08-10T10:33:15.130Z        INFO    controllers.Tenant      Computed requests.cpu quota for the whole Tenant is 1100m       {"Request.Name": "oil"}
2020-08-10T10:33:15.280Z        INFO    controllers.Tenant      Desired hard requests.memory quota is 16Gi      {"Request.Name": "oil"}
2020-08-10T10:33:15.281Z        INFO    controllers.Tenant      Computed requests.memory quota for the whole Tenant is 110Mi    {"Request.Name": "oil"}
2020-08-10T10:33:15.431Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:33:15.431Z        INFO    controllers.Tenant      Desired hard pods quota is 10   {"Request.Name": "oil"}
2020-08-10T10:33:15.431Z        INFO    controllers.Tenant      Computed pods quota for the whole Tenant is 11  {"Request.Name": "oil"}
2020-08-10T10:33:15.580Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-test"}
2020-08-10T10:33:15.581Z        INFO    controllers.Tenant      Desired hard requests.storage quota is 100Gi    {"Request.Name": "oil"}
2020-08-10T10:33:15.581Z        INFO    controllers.Tenant      Computed requests.storage quota for the whole Tenant is 0       {"Request.Name": "oil"}
2020-08-10T10:33:15.730Z        INFO    controllers.Tenant      Resource Quota sync result: unchanged   {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-test"}
2020-08-10T10:33:15.730Z        INFO    controllers.Tenant      Ensuring RoleBinding for owner  {"Request.Name": "oil"}
2020-08-10T10:33:15.734Z        INFO    controllers.Tenant      Role Binding sync result: updated       {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-development"}
2020-08-10T10:33:15.734Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-development"}
2020-08-10T10:33:15.734Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-production"}
2020-08-10T10:33:15.734Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-production"}
2020-08-10T10:33:15.734Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-test"}
2020-08-10T10:33:15.735Z        INFO    controllers.Tenant      Role Binding sync result: unchanged     {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-test"}
2020-08-10T10:33:15.735Z        INFO    controllers.Tenant      Ensuring Namespace count        {"Request.Name": "oil"}
2020-08-10T10:33:15.743Z        INFO    controllers.Tenant      Tenant reconciling completed    {"Request.Name": "oil"}

Additional context

Capsule version: (capsule --version)
Kubernetes version: (kubectl version)

NetworkPolicy pruning is deleting also Tenant ones

Bug description

If the Tenant owner is going to deploy manually some NetworkPolicy resources on their namespaces and a reconciliation occurs, these are going to be deleted.

How to reproduce

Given the following np.yaml:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: dont-delete-me-capsule
spec:
  podSelector:
    matchLabels:
  ingress:
  - from:
    - podSelector: {}

kubectl apply -f np.yaml
reconcile the Tenant, restarting Capsule for example
kubectl get networkpolicies.networking.k8s.io -w and notice how it's deleted

Expected behavior

User NetworkPolicy resource should not be deleted.

Logs

Irrelevant.

Additional context

Capsule version: 0.0.1
Kubernetes version: Irrelevant

Tenant Resource Quota is not honoured

Bug description

We are testing the Tenant Resource Quota feature and discovered that Resource Quota at tenant level is not honoured.

How to reproduce

Steps to reproduce the behaviour:

Tenant definition:

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  annotations:
...
Spec:
  Ingress Classes:
    default
  Limit Ranges:

  Namespace Quota:  3
  Network Policies:

  Node Selector:
    kubernetes.io/os:  linux
  Owner:               pippouser
  Resource Quotas:
    Hard:
      limits.cpu:       8
      limits.memory:    16Gi
      requests.cpu:     8
      requests.memory:  16Gi
    Scopes:
      NotTerminating
    Hard:
      Pods:  10          <--- Hard Pod Quota is 10
    Hard:
      requests.storage:  100Gi
  Storage Classes:
    standard
Status:
  Namespaces:
    usl1
    usl2
    usl3
  Size:  3

We created 3 namespaces: usl1, usl2, and usl3

With empty namespaces, we have:

kaas~/Shared/capsule$ kubectl get resourcequota -n usl1
NAME                    AGE   REQUEST                                      LIMIT
capsule-plutotenant-0   18h   requests.cpu: 0/8, requests.memory: 0/16Gi   limits.cpu: 0/8, limits.memory: 0/16Gi
capsule-plutotenant-1   18h   pods: 0/10                                   
capsule-plutotenant-2   18h   requests.storage: 0/100Gi                    
kaas~/Shared/capsule$ 
kaas~/Shared/capsule$ kubectl get resourcequota -n usl2
NAME                    AGE   REQUEST                                      LIMIT
capsule-plutotenant-0   18h   requests.cpu: 0/8, requests.memory: 0/16Gi   limits.cpu: 0/8, limits.memory: 0/16Gi
capsule-plutotenant-1   18h   pods: 0/10                                   
capsule-plutotenant-2   18h   requests.storage: 0/100Gi                    
kaas~/Shared/capsule$ kubectl get resourcequota -n usl3
NAME                    AGE   REQUEST                                      LIMIT
capsule-plutotenant-0   18h   requests.cpu: 0/8, requests.memory: 0/16Gi   limits.cpu: 0/8, limits.memory: 0/16Gi
capsule-plutotenant-1   18h   pods: 0/10                                   
capsule-plutotenant-2   18h   requests.storage: 0/100Gi

Then we create 3 pods for each namespace for a total of 9 pods:

kaas~/Shared/capsule$ kubectl apply -f nginx-rs.yaml -n usl1
kaas~/Shared/capsule$ kubectl apply -f nginx-rs.yaml -n usl2
kaas~/Shared/capsule$ kubectl apply -f nginx-rs.yaml -n usl3

kaas~/Shared/capsule$ kubectl get pods -n usl1
NAME          READY   STATUS    RESTARTS   AGE
nginx-282bz   1/1     Running   0          18s
nginx-7z4p8   1/1     Running   0          18s
nginx-lgwch   0/1     Running   0          18s
aas~/Shared/capsule$ kubectl get pods -n usl2
NAME          READY   STATUS    RESTARTS   AGE
nginx-6g27f   1/1     Running   0          18s
nginx-hs7r2   1/1     Running   0          18s
nginx-ndz69   1/1     Running   0          18s
kaas~/Shared/capsule$ kubectl get pods -n usl3
NAME          READY   STATUS              RESTARTS   AGE
nginx-cndnv   1/1     Running             0          18s
nginx-q9g6f   1/1     Running             0          18s
nginx-t5jld   0/1     Running             0          18s

kaas~/Shared/capsule$ kubectl get resourcequota -n usl1
NAME                    AGE   REQUEST                                            LIMIT
capsule-plutotenant-0   18h   requests.cpu: 300m/8, requests.memory: 30Mi/16Gi   limits.cpu: 600m/8, limits.memory: 300Mi/16Gi
capsule-plutotenant-1   18h   pods: 3/10                                         
capsule-plutotenant-2   18h   requests.storage: 0/100Gi                          
kaas~/Shared/capsule$ kubectl get resourcequota -n usl2
NAME                    AGE   REQUEST                                            LIMIT
capsule-plutotenant-0   18h   requests.cpu: 300m/8, requests.memory: 30Mi/16Gi   limits.cpu: 600m/8, limits.memory: 300Mi/16Gi
capsule-plutotenant-1   18h   pods: 3/10                                         
capsule-plutotenant-2   18h   requests.storage: 0/100Gi                          
kaas~/Shared/capsule$ kubectl get resourcequota -n usl3
NAME                    AGE   REQUEST                                            LIMIT
capsule-plutotenant-0   18h   requests.cpu: 300m/8, requests.memory: 30Mi/16Gi   limits.cpu: 600m/8, limits.memory: 300Mi/16Gi
capsule-plutotenant-1   18h   pods: 3/10                                         
capsule-plutotenant-2   18h   requests.storage: 0/100Gi 

kaas~/Shared/capsule$ kubectl get resourcequota capsule-plutotenant-1 -o yaml -n usl1
apiVersion: v1
kind: ResourceQuota
metadata:
  annotations:
    quota.capsule.clastix.io/used-pods: "9"
  creationTimestamp: "2020-07-27T15:39:38Z"
  labels:
    capsule.clastix.io/resource-quota: "1"
    capsule.clastix.io/tenant: plutotenant


kaas~/Shared/capsule$ kubectl get resourcequota capsule-plutotenant-1 -o yaml -n usl2
apiVersion: v1
kind: ResourceQuota
metadata:
  annotations:
    quota.capsule.clastix.io/used-pods: "9"
  creationTimestamp: "2020-07-27T15:51:47Z"
  labels:
    capsule.clastix.io/resource-quota: "1"
    capsule.clastix.io/tenant: plutotenant

kaas~/Shared/capsule$ kubectl get resourcequota capsule-plutotenant-1 -o yaml -n usl3
apiVersion: v1
kind: ResourceQuota
metadata:
  annotations:
    quota.capsule.clastix.io/used-pods: "9"
  creationTimestamp: "2020-07-27T15:51:51Z"
  labels:
    capsule.clastix.io/resource-quota: "1"
    capsule.clastix.io/tenant: plutotenant

Then scale up in namespace usl1 for a total of 12 pods in the tanant

kubectl scale rs nginx --replicas=6 -n usl1
replicaset.apps/nginx scaled


kaas~/Shared/capsule$ kubectl get pods -n usl1
NAME          READY   STATUS    RESTARTS   AGE
nginx-282bz   1/1     Running   0          6m39s
nginx-56df5   1/1     Running   0          83s
nginx-7z4p8   1/1     Running   0          6m39s
nginx-lgwch   1/1     Running   0          6m39s
nginx-ppn8v   1/1     Running   0          83s
nginx-rvcr7   1/1     Running   0          83s


kaas~/Shared/capsule$ kubectl get pods -n usl2
NAME          READY   STATUS    RESTARTS   AGE
nginx-6g27f   1/1     Running   0          5m49s
nginx-hs7r2   1/1     Running   0          5m49s
nginx-ndz69   1/1     Running   0          5m49s
kaas~/Shared/capsule$ kubectl get pods -n usl3
NAME          READY   STATUS    RESTARTS   AGE
nginx-cndnv   1/1     Running   0          5m49s
nginx-q9g6f   1/1     Running   0          5m49s
nginx-t5jld   1/1     Running   0          5m49s



kaas~/Shared/capsule$ kubectl get resourcequota  -n usl1
NAME                    AGE   REQUEST                                            LIMIT
capsule-plutotenant-0   18h   requests.cpu: 600m/8, requests.memory: 60Mi/16Gi   limits.cpu: 1200m/8, limits.memory: 600Mi/16Gi
capsule-plutotenant-1   18h   pods: 6/10                                         
capsule-plutotenant-2   18h   requests.storage: 0/100Gi                          
kaas~/Shared/capsule$ 
kaas~/Shared/capsule$ 
kaas~/Shared/capsule$ kubectl get resourcequota  -n usl2
NAME                    AGE   REQUEST                                            LIMIT
capsule-plutotenant-0   18h   requests.cpu: 300m/8, requests.memory: 30Mi/16Gi   limits.cpu: 600m/8, limits.memory: 300Mi/16Gi
capsule-plutotenant-1   18h   pods: 3/10                                         
capsule-plutotenant-2   18h   requests.storage: 0/100Gi                          
kaas~/Shared/capsule$ kubectl get resourcequota  -n usl3
NAME                    AGE   REQUEST                                            LIMIT
capsule-plutotenant-0   18h   requests.cpu: 300m/8, requests.memory: 30Mi/16Gi   limits.cpu: 600m/8, limits.memory: 300Mi/16Gi
capsule-plutotenant-1   18h   pods: 3/10                                         
capsule-plutotenant-2   18h   requests.storage: 0/100Gi     



kaas~/Shared/capsule$ kubectl get resourcequota capsule-plutotenant-1 -o yaml -n usl1
apiVersion: v1
kind: ResourceQuota
metadata:
  annotations:
    quota.capsule.clastix.io/used-pods: "12"
  creationTimestamp: "2020-07-27T15:39:38Z"
  labels:
    capsule.clastix.io/resource-quota: "1"
    capsule.clastix.io/tenant: plutotenant


kaas~/Shared/capsule$ kubectl get resourcequota capsule-plutotenant-1 -o yaml -n usl2
apiVersion: v1
kind: ResourceQuota
metadata:
  annotations:
    quota.capsule.clastix.io/used-pods: "12"
  creationTimestamp: "2020-07-27T15:51:47Z"
  labels:
    capsule.clastix.io/resource-quota: "1"
    capsule.clastix.io/tenant: plutotenant


kaas~/Shared/capsule$ kubectl get resourcequota capsule-plutotenant-1 -o yaml -n usl3
apiVersion: v1
kind: ResourceQuota
metadata:
  annotations:
    quota.capsule.clastix.io/used-pods: "12"
  creationTimestamp: "2020-07-27T15:51:51Z"
  labels:
    capsule.clastix.io/resource-quota: "1"
    capsule.clastix.io/tenant: plutotenant

Expected behavior

We expect the Capsule controller to block the creation of pods exceeding the quota specified in the tenant (10).

Logs

{"level":"info","ts":1595931276.4034417,"logger":"controller_tenant","msg":"Starting processing of Network Policies","Request.Name":"plutotenant","items":1}
{"level":"info","ts":1595931276.4037812,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/network-policy notin (0)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.4126463,"logger":"controller_tenant","msg":"Network Policy sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl1"}
{"level":"info","ts":1595931276.412777,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/network-policy notin (0)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.416444,"logger":"controller_tenant","msg":"Network Policy sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl2"}
{"level":"info","ts":1595931276.4164975,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/network-policy notin (0)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.4207706,"logger":"controller_tenant","msg":"Network Policy sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl3"}
{"level":"info","ts":1595931276.4208238,"logger":"controller_tenant","msg":"Starting processing of Node Selector","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.421098,"logger":"controller_tenant","msg":"Namespace Node  sync result: unchanged","Request.Name":"plutotenant","name":"usl1"}
{"level":"info","ts":1595931276.421387,"logger":"controller_tenant","msg":"Namespace Node  sync result: unchanged","Request.Name":"plutotenant","name":"usl2"}
{"level":"info","ts":1595931276.4216444,"logger":"controller_tenant","msg":"Namespace Node  sync result: unchanged","Request.Name":"plutotenant","name":"usl3"}
{"level":"info","ts":1595931276.4216664,"logger":"controller_tenant","msg":"Starting processing of Limit Ranges","Request.Name":"plutotenant","items":1}
{"level":"info","ts":1595931276.4219756,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/limit-range notin (0)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.4289002,"logger":"controller_tenant","msg":"LimitRange sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl1"}
{"level":"info","ts":1595931276.4289649,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/limit-range notin (0)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.436696,"logger":"controller_tenant","msg":"LimitRange sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl2"}
{"level":"info","ts":1595931276.4367547,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/limit-range notin (0)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.4407997,"logger":"controller_tenant","msg":"LimitRange sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl3"}
{"level":"info","ts":1595931276.4409108,"logger":"controller_tenant","msg":"Starting processing of Resource Quotas","Request.Name":"plutotenant","items":3}
{"level":"info","ts":1595931276.441031,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.4495492,"logger":"controller_tenant","msg":"Desired hard limits.memory quota is 16Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.449604,"logger":"controller_tenant","msg":"Computed limits.memory quota for the whole Tenant is 1100Mi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.5032845,"logger":"controller_tenant","msg":"Desired hard requests.cpu quota is 8","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.5033338,"logger":"controller_tenant","msg":"Computed requests.cpu quota for the whole Tenant is 1100m","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.524165,"logger":"controller_tenant","msg":"Desired hard requests.memory quota is 16Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.5242355,"logger":"controller_tenant","msg":"Computed requests.memory quota for the whole Tenant is 110Mi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.545166,"logger":"controller_tenant","msg":"Desired hard limits.cpu quota is 8","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.545233,"logger":"controller_tenant","msg":"Computed limits.cpu quota for the whole Tenant is 2200m","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.5652263,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl1"}
{"level":"info","ts":1595931276.565492,"logger":"controller_tenant","msg":"Desired hard pods quota is 10","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.5655282,"logger":"controller_tenant","msg":"Computed pods quota for the whole Tenant is 12","Request.Name":"plutotenant"}

{"level":"info","ts":1595931276.5886817,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-1","namespace":"usl1"}
{"level":"info","ts":1595931276.5889187,"logger":"controller_tenant","msg":"Desired hard requests.storage quota is 100Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.5889587,"logger":"controller_tenant","msg":"Computed requests.storage quota for the whole Tenant is 0","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.6020565,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-2","namespace":"usl1"}
{"level":"info","ts":1595931276.6021152,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.6058068,"logger":"controller_tenant","msg":"Desired hard requests.memory quota is 16Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.6058373,"logger":"controller_tenant","msg":"Computed requests.memory quota for the whole Tenant is 120Mi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.6272814,"logger":"controller_tenant","msg":"Desired hard limits.cpu quota is 8","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.6274571,"logger":"controller_tenant","msg":"Computed limits.cpu quota for the whole Tenant is 2400m","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.654262,"logger":"controller_tenant","msg":"Desired hard limits.memory quota is 16Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.654315,"logger":"controller_tenant","msg":"Computed limits.memory quota for the whole Tenant is 1200Mi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.6750116,"logger":"controller_tenant","msg":"Desired hard requests.cpu quota is 8","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.6750698,"logger":"controller_tenant","msg":"Computed requests.cpu quota for the whole Tenant is 1200m","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.7004423,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl2"}
{"level":"info","ts":1595931276.700643,"logger":"controller_tenant","msg":"Desired hard pods quota is 10","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.70067,"logger":"controller_tenant","msg":"Computed pods quota for the whole Tenant is 12","Request.Name":"plutotenant"}

{"level":"info","ts":1595931276.7474844,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-1","namespace":"usl2"}
{"level":"info","ts":1595931276.7477975,"logger":"controller_tenant","msg":"Desired hard requests.storage quota is 100Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.747816,"logger":"controller_tenant","msg":"Computed requests.storage quota for the whole Tenant is 0","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.8970833,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-2","namespace":"usl2"}
{"level":"info","ts":1595931276.8971539,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.9466813,"logger":"controller_tenant","msg":"Desired hard limits.cpu quota is 8","Request.Name":"plutotenant"}
{"level":"info","ts":1595931276.9467242,"logger":"controller_tenant","msg":"Computed limits.cpu quota for the whole Tenant is 2400m","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.0970955,"logger":"controller_tenant","msg":"Desired hard limits.memory quota is 16Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.0971472,"logger":"controller_tenant","msg":"Computed limits.memory quota for the whole Tenant is 1200Mi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.2474694,"logger":"controller_tenant","msg":"Desired hard requests.cpu quota is 8","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.24754,"logger":"controller_tenant","msg":"Computed requests.cpu quota for the whole Tenant is 1200m","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.3976142,"logger":"controller_tenant","msg":"Desired hard requests.memory quota is 16Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.3976903,"logger":"controller_tenant","msg":"Computed requests.memory quota for the whole Tenant is 120Mi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.5486085,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl3"}
{"level":"info","ts":1595931277.5487823,"logger":"controller_tenant","msg":"Desired hard pods quota is 10","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.5487983,"logger":"controller_tenant","msg":"Computed pods quota for the whole Tenant is 12","Request.Name":"plutotenant"}

{"level":"info","ts":1595931277.6983695,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-1","namespace":"usl3"}
{"level":"info","ts":1595931277.6986752,"logger":"controller_tenant","msg":"Desired hard requests.storage quota is 100Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.6986911,"logger":"controller_tenant","msg":"Computed requests.storage quota for the whole Tenant is 0","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.8470774,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-2","namespace":"usl3"}
{"level":"info","ts":1595931277.847203,"logger":"controller_tenant","msg":"Ensuring RoleBinding for owner","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.8518553,"logger":"controller_tenant","msg":"Role Binding sync result: updated","Request.Name":"plutotenant","name":"namespace:admin","namespace":"usl3"}
{"level":"info","ts":1595931277.8533523,"logger":"controller_tenant","msg":"Role Binding sync result: unchanged","Request.Name":"plutotenant","name":"namespace:deleter","namespace":"usl3"}
{"level":"info","ts":1595931277.853605,"logger":"controller_tenant","msg":"Role Binding sync result: unchanged","Request.Name":"plutotenant","name":"namespace:admin","namespace":"usl1"}
{"level":"info","ts":1595931277.8565931,"logger":"controller_tenant","msg":"Role Binding sync result: unchanged","Request.Name":"plutotenant","name":"namespace:deleter","namespace":"usl1"}
{"level":"info","ts":1595931277.8570623,"logger":"controller_tenant","msg":"Role Binding sync result: unchanged","Request.Name":"plutotenant","name":"namespace:admin","namespace":"usl2"}
{"level":"info","ts":1595931277.8574808,"logger":"controller_tenant","msg":"Role Binding sync result: unchanged","Request.Name":"plutotenant","name":"namespace:deleter","namespace":"usl2"}
{"level":"info","ts":1595931277.8576417,"logger":"controller_tenant","msg":"Tenant reconciling completed","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.8585966,"logger":"controller_tenant","msg":"Starting processing of Network Policies","Request.Name":"plutotenant","items":1}
{"level":"info","ts":1595931277.8588207,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/network-policy notin (0)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.8683193,"logger":"controller_tenant","msg":"Network Policy sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl1"}
{"level":"info","ts":1595931277.8683808,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/network-policy notin (0)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.8711429,"logger":"controller_tenant","msg":"Network Policy sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl2"}
{"level":"info","ts":1595931277.8711782,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/network-policy notin (0)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.8740993,"logger":"controller_tenant","msg":"Network Policy sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl3"}
{"level":"info","ts":1595931277.8741276,"logger":"controller_tenant","msg":"Starting processing of Node Selector","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.8742213,"logger":"controller_tenant","msg":"Namespace Node  sync result: unchanged","Request.Name":"plutotenant","name":"usl1"}
{"level":"info","ts":1595931277.8743088,"logger":"controller_tenant","msg":"Namespace Node  sync result: unchanged","Request.Name":"plutotenant","name":"usl2"}
{"level":"info","ts":1595931277.874399,"logger":"controller_tenant","msg":"Namespace Node  sync result: unchanged","Request.Name":"plutotenant","name":"usl3"}
{"level":"info","ts":1595931277.8744168,"logger":"controller_tenant","msg":"Starting processing of Limit Ranges","Request.Name":"plutotenant","items":1}
{"level":"info","ts":1595931277.8744342,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/limit-range notin (0)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.8781052,"logger":"controller_tenant","msg":"LimitRange sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl1"}
{"level":"info","ts":1595931277.8781438,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/limit-range notin (0)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.881174,"logger":"controller_tenant","msg":"LimitRange sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl2"}
{"level":"info","ts":1595931277.8812094,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/limit-range notin (0)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.8843713,"logger":"controller_tenant","msg":"LimitRange sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl3"}
{"level":"info","ts":1595931277.8846133,"logger":"controller_tenant","msg":"Starting processing of Resource Quotas","Request.Name":"plutotenant","items":3}
{"level":"info","ts":1595931277.8847358,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.8952508,"logger":"controller_tenant","msg":"Desired hard limits.memory quota is 16Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931277.8953485,"logger":"controller_tenant","msg":"Computed limits.memory quota for the whole Tenant is 1200Mi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931278.047916,"logger":"controller_tenant","msg":"Desired hard requests.cpu quota is 8","Request.Name":"plutotenant"}
{"level":"info","ts":1595931278.0479865,"logger":"controller_tenant","msg":"Computed requests.cpu quota for the whole Tenant is 1200m","Request.Name":"plutotenant"}
{"level":"info","ts":1595931278.1967275,"logger":"controller_tenant","msg":"Desired hard requests.memory quota is 16Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931278.1967673,"logger":"controller_tenant","msg":"Computed requests.memory quota for the whole Tenant is 120Mi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931278.3484957,"logger":"controller_tenant","msg":"Desired hard limits.cpu quota is 8","Request.Name":"plutotenant"}
{"level":"info","ts":1595931278.3485723,"logger":"controller_tenant","msg":"Computed limits.cpu quota for the whole Tenant is 2400m","Request.Name":"plutotenant"}
{"level":"info","ts":1595931278.5031614,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl1"}
{"level":"info","ts":1595931278.5039475,"logger":"controller_tenant","msg":"Desired hard pods quota is 10","Request.Name":"plutotenant"}
{"level":"info","ts":1595931278.504314,"logger":"controller_tenant","msg":"Computed pods quota for the whole Tenant is 12","Request.Name":"plutotenant"}

{"level":"info","ts":1595931278.647681,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-1","namespace":"usl1"}
{"level":"info","ts":1595931278.647966,"logger":"controller_tenant","msg":"Desired hard requests.storage quota is 100Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931278.647989,"logger":"controller_tenant","msg":"Computed requests.storage quota for the whole Tenant is 0","Request.Name":"plutotenant"}
{"level":"info","ts":1595931278.796046,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-2","namespace":"usl1"}
{"level":"info","ts":1595931278.796111,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931278.8456223,"logger":"controller_tenant","msg":"Desired hard limits.memory quota is 16Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931278.8456635,"logger":"controller_tenant","msg":"Computed limits.memory quota for the whole Tenant is 1200Mi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931278.9972732,"logger":"controller_tenant","msg":"Desired hard requests.cpu quota is 8","Request.Name":"plutotenant"}
{"level":"info","ts":1595931278.997557,"logger":"controller_tenant","msg":"Computed requests.cpu quota for the whole Tenant is 1200m","Request.Name":"plutotenant"}
{"level":"info","ts":1595931279.1488643,"logger":"controller_tenant","msg":"Desired hard requests.memory quota is 16Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931279.1489353,"logger":"controller_tenant","msg":"Computed requests.memory quota for the whole Tenant is 120Mi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931279.2972162,"logger":"controller_tenant","msg":"Desired hard limits.cpu quota is 8","Request.Name":"plutotenant"}
{"level":"info","ts":1595931279.2972736,"logger":"controller_tenant","msg":"Computed limits.cpu quota for the whole Tenant is 2400m","Request.Name":"plutotenant"}
{"level":"info","ts":1595931279.4480398,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl2"}
{"level":"info","ts":1595931279.4482088,"logger":"controller_tenant","msg":"Desired hard pods quota is 10","Request.Name":"plutotenant"}
{"level":"info","ts":1595931279.44823,"logger":"controller_tenant","msg":"Computed pods quota for the whole Tenant is 12","Request.Name":"plutotenant"}

{"level":"info","ts":1595931279.5966516,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-1","namespace":"usl2"}
{"level":"info","ts":1595931279.596853,"logger":"controller_tenant","msg":"Desired hard requests.storage quota is 100Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931279.5969186,"logger":"controller_tenant","msg":"Computed requests.storage quota for the whole Tenant is 0","Request.Name":"plutotenant"}
{"level":"info","ts":1595931279.7484195,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-2","namespace":"usl2"}
{"level":"info","ts":1595931279.7484918,"logger":"controller_tenant","msg":"Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2)","Request.Name":"plutotenant"}
{"level":"info","ts":1595931279.79746,"logger":"controller_tenant","msg":"Desired hard requests.memory quota is 16Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931279.7975004,"logger":"controller_tenant","msg":"Computed requests.memory quota for the whole Tenant is 120Mi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931279.9465752,"logger":"controller_tenant","msg":"Desired hard limits.cpu quota is 8","Request.Name":"plutotenant"}
{"level":"info","ts":1595931279.946632,"logger":"controller_tenant","msg":"Computed limits.cpu quota for the whole Tenant is 2400m","Request.Name":"plutotenant"}
{"level":"info","ts":1595931280.0979872,"logger":"controller_tenant","msg":"Desired hard limits.memory quota is 16Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931280.0980456,"logger":"controller_tenant","msg":"Computed limits.memory quota for the whole Tenant is 1200Mi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931280.2471662,"logger":"controller_tenant","msg":"Desired hard requests.cpu quota is 8","Request.Name":"plutotenant"}
{"level":"info","ts":1595931280.2472095,"logger":"controller_tenant","msg":"Computed requests.cpu quota for the whole Tenant is 1200m","Request.Name":"plutotenant"}
{"level":"info","ts":1595931280.398391,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-0","namespace":"usl3"}
{"level":"info","ts":1595931280.3986878,"logger":"controller_tenant","msg":"Desired hard pods quota is 10","Request.Name":"plutotenant"}
{"level":"info","ts":1595931280.3987398,"logger":"controller_tenant","msg":"Computed pods quota for the whole Tenant is 12","Request.Name":"plutotenant"}

{"level":"info","ts":1595931280.5474253,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-1","namespace":"usl3"}
{"level":"info","ts":1595931280.547838,"logger":"controller_tenant","msg":"Desired hard requests.storage quota is 100Gi","Request.Name":"plutotenant"}
{"level":"info","ts":1595931280.5479205,"logger":"controller_tenant","msg":"Computed requests.storage quota for the whole Tenant is 0","Request.Name":"plutotenant"}
{"level":"info","ts":1595931280.6980507,"logger":"controller_tenant","msg":"Resource Quota sync result: unchanged","Request.Name":"plutotenant","name":"capsule-plutotenant-2","namespace":"usl3"}
{"level":"info","ts":1595931280.6980958,"logger":"controller_tenant","msg":"Ensuring RoleBinding for owner","Request.Name":"plutotenant"}
{"level":"info","ts":1595931280.7020915,"logger":"controller_tenant","msg":"Role Binding sync result: updated","Request.Name":"plutotenant","name":"namespace:admin","namespace":"usl1"}
{"level":"info","ts":1595931280.702371,"logger":"controller_tenant","msg":"Role Binding sync result: unchanged","Request.Name":"plutotenant","name":"namespace:deleter","namespace":"usl1"}
{"level":"info","ts":1595931280.7025545,"logger":"controller_tenant","msg":"Role Binding sync result: unchanged","Request.Name":"plutotenant","name":"namespace:admin","namespace":"usl2"}
{"level":"info","ts":1595931280.7032402,"logger":"controller_tenant","msg":"Role Binding sync result: unchanged","Request.Name":"plutotenant","name":"namespace:deleter","namespace":"usl2"}
{"level":"info","ts":1595931280.7034364,"logger":"controller_tenant","msg":"Role Binding sync result: unchanged","Request.Name":"plutotenant","name":"namespace:admin","namespace":"usl3"}
{"level":"info","ts":1595931280.7036405,"logger":"controller_tenant","msg":"Role Binding sync result: unchanged","Request.Name":"plutotenant","name":"namespace:deleter","namespace":"usl3"}
{"level":"info","ts":1595931280.7036653,"logger":"controller_tenant","msg":"Tenant reconciling completed","Request.Name":"plutotenant"}

In a standard stand-alone installation of Capsule,
you'd get this by running kubectl -n capsule-system logs deploy/capsule.

Additional context

Capsule version: (capsule --version)

sh-4.4# capsule --version
{"level":"info","ts":1595932434.7580533,"logger":"cmd","msg":"Operator Version: 0.0.1"}
{"level":"info","ts":1595932434.7581751,"logger":"cmd","msg":"Go Version: go1.14.5"}
{"level":"info","ts":1595932434.7582006,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1595932434.7582278,"logger":"cmd","msg":"Version of operator-sdk: v0.18.1"}

Kubernetes version: (kubectl version)
v1.18.5

Document how to increase logs level

It would be nice to have a configurable logs level for the capsule controller, e.g. info, warning, debug, trace, .. This will help during troubleshooting.

Getting an error message from Capsule Operator about CA certificate validity

Bug description

Getting an error message from Capsule Operator about CA certificate validity:

Error from server (InternalError): Internal error occurred: failed calling webhook "owner.namespace.capsule.clastix.io": Post https://capsule-webhook-service.capsule-system.svc:443/mutate-v1-namespace-owner-reference?timeout=30s: x509: certificate signed by unknown authority

Webhook handlers should work as piped middleware

Describe the feature

It appears we have to replicate or not for some webhook certain business logic.

As an example, we're always checking if the user impacting the webhook is a Capsule user or not but this could be a blocker for future interactions and we should rely on a smart handlers encapsulation, as HTTP closures.

What would the new user story look like?

Separate the Webhook wrapper from the Handlers
Define the types to operate with
Ensure everything is working fine

Expected behavior

The test suite must be green upon changes.

Tenant backup and restore of a Tenant

Describe the feature

It looks that Velero is the most suitable solution to handle Kubernetes cluster backups, although meaning adding a new dependency besides the already required admission plugins.

However, we should ignore specific Tenant that doesn't need a backup strategy due to end-user business logic (e.g. tiering).

Would be great if we could easily label the Namespace resources to let Velero skips all the underlying resources.

What would the new user story look like?

feature request implemented in Velero
adding a .spec.backup boolean flag into Tenant definition
Capsule ensures that the selected label is available on all the handled Namespace resources
Capsule ensures the PV used by the PVC is labeled with the said one since Persistent Volume are cluster-scoped resources.

Expected behavior

Toggling the backup flag into Tenant definition, Velero should ignore or not the backup of the Tenant resources.

Capsule should control at runtime the namespace:provisioner and namespace:deleter Cluster Role and CRB resources

Bug description

Found this via #67: although adding the CLI flag we have to changes also the bindings, otherwise, the Capsule user wouldn't be able to provision new namespaces or delete these.

How to reproduce

N.R.

Expected behavior

Once Capsule is running, the given resources should be updated or created at runtime according to the specified Capsule user group.

Logs

N.R.

Additional context

Capsule version: N.R.
Kubernetes version: N.R.

Namespace Controller is not using predicates

Bug description

Not properly a bug but, rather, a code improvement.

https://github.com/clastix/capsule/blob/66969e334cebdb53ba63a420116a02f62f37a2fc/pkg/controller/namespace/namespace_controller.go#L132-L135

This kind of business logic should be handled at the predicate level rather than the reconciliation loop.

How to reproduce

Restart the Capsule pod
Notice how it iterates all over the available namespaces

Expected behavior

Capsule Namespace Controller should iterate only on managed Namespaces to don't waste cycles.

Logs

{"level":"info","ts":1595927277.4415448,"logger":"controller_namespace","msg":"Reconciling Namespace","Request.Namespace":"","Request.Name":"kube-public"}
{"level":"info","ts":1595927277.441783,"logger":"controller_namespace","msg":"Reconciling Namespace","Request.Namespace":"","Request.Name":"kube-system"}
{"level":"info","ts":1595927277.4419541,"logger":"controller_namespace","msg":"Reconciling Namespace","Request.Namespace":"","Request.Name":"local-path-storage"}
{"level":"info","ts":1595927277.442173,"logger":"controller_namespace","msg":"Reconciling Namespace","Request.Namespace":"","Request.Name":"oil-dev"}
{"level":"info","ts":1595927277.453179,"logger":"controller_namespace","msg":"Namespace reconciliation processed","Request.Namespace":"","Request.Name":"oil-dev"}
{"level":"info","ts":1595927277.4532309,"logger":"controller_namespace","msg":"Reconciling Namespace","Request.Namespace":"","Request.Name":"oil-prod"}
{"level":"info","ts":1595927277.4586139,"logger":"controller_namespace","msg":"Namespace reconciliation processed","Request.Namespace":"","Request.Name":"oil-prod"}
{"level":"info","ts":1595927277.4586496,"logger":"controller_namespace","msg":"Reconciling Namespace","Request.Namespace":"","Request.Name":"capsule-system"}
{"level":"info","ts":1595927277.4586751,"logger":"controller_namespace","msg":"Reconciling Namespace","Request.Namespace":"","Request.Name":"default"}
{"level":"info","ts":1595927277.4586918,"logger":"controller_namespace","msg":"Reconciling Namespace","Request.Namespace":"","Request.Name":"kube-node-lease"}
{"level":"info","ts":1595927455.349841,"logger":"controller_namespace","msg":"Reconciling Namespace","Request.Namespace":"","Request.Name":"oil-staging"}
{"level":"info","ts":1595927455.365413,"logger":"controller_namespace","msg":"Namespace reconciliation processed","Request.Namespace":"","Request.Name":"oil-staging"}
{"level":"info","ts":1595927455.3654807,"logger":"controller_namespace","msg":"Reconciling Namespace","Request.Namespace":"","Request.Name":"oil-staging"}
{"level":"info","ts":1595927455.3830864,"logger":"controller_namespace","msg":"Namespace reconciliation processed","Request.Namespace":"","Request.Name":"oil-staging"}
{"level":"info","ts":1595927455.413415,"logger":"controller_namespace","msg":"Reconciling Namespace","Request.Namespace":"","Request.Name":"oil-staging"}
{"level":"info","ts":1595927455.4246178,"logger":"controller_namespace","msg":"Namespace reconciliation processed","Request.Namespace":"","Request.Name":"oil-staging"}

Additional context

Capsule version:

{"level":"info","ts":1595928190.2007432,"logger":"cmd","msg":"Operator Version: 0.0.1"}
{"level":"info","ts":1595928190.200779,"logger":"cmd","msg":"Go Version: go1.13.8"}
{"level":"info","ts":1595928190.2007892,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1595928190.2008016,"logger":"cmd","msg":"Version of operator-sdk: v0.18.1"}

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.4", GitCommit:"224be7bdce5a9dd0c2fd0d46b83865648e2fe0ba", GitTreeState:"clean", BuildDate:"2019-12-11T12:47:40Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-30T20:19:45Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Simple implementation of control naming of namespaces

Describe the feature

We may need a simple implementation of control naming of namespaces.
Currently during creation of a new namespace by tenant-owner we receive an error message from k8s if the name of our selected namespace is already used by another tenant.

For example:

[ansible@k8s-caas-ansible-installer ~/capsule/hack]$ kubectl create ns production
Error from server (AlreadyExists): namespaces "production" already exists

What would the new user story look like?

Tenant owner creates a new Namespace (eg. production like the example above)
Capsule creates the new namespace having the name "tenantname-production" where the "tenantname" is the name of the tenant and "production" is the name of the namespace.
This is going to be attached to the Tenant
All the magic happens in the background

Tenant NetworkPolicies added by the tenant owner cannot be deleted nor patched

Bug description

Tenant NetworkPolicies added by the tenant owner cannot be deleted nor patched

$ cat networkpolicy.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  labels:
  name: custom-network-policy-0
  namespace:
spec:
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          capsule.clastix.io/tenant: oil
    - podSelector: {}
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

How to reproduce

Steps to reproduce the behaviour:

Given additional network policies the tenant owner has in a namespace

$ kubectl get networkpolicies
NAME                      POD-SELECTOR   AGE
capsule-oil-0             <none>         17m
custom-network-policy     <none>         18m
custom-network-policy-0   <none>         13m

the tenant gets deny when deleting or patching

kubectl apply -f networkpolicy.yaml
Error from server (Capsule Network Policies cannot be updated: please, reach out the system administrators): error when applying patch:
{"metadata":{"labels":null}}
to:
Resource: "networking.k8s.io/v1, Resource=networkpolicies", GroupVersionKind: "networking.k8s.io/v1, Kind=NetworkPolicy"
Name: "custom-network-policy", Namespace: "usl1"
for: "networkpolicy.yaml": admission webhook "validating.network-policy.capsule.clastix.io" denied the request: Capsule Network Policies cannot be updated: please, reach out the system administrators
6:41

kubectl delete -f networkpolicy.yaml 
Error from server (Capsule Network Policies cannot be deleted: please, reach out the system administrators): error when deleting "networkpolicy.yaml": admission webhook "validating.network-policy.capsule.clastix.io" denied the request: Capsule Network Policies cannot be deleted: please, reach out the system administrators

Expected behavior

The tenant owner can create, patch and delete additional network policies in his namespaces

Logs

2020-08-04T16:43:17.003Z        DEBUG   controller-runtime.webhook.webhooks     received request        {"webhook": "/validating-v1-network-policy", "UID": "e3ab89ac-3311-45b8-9c59-056a253bdcef", "kind": "networking.k8s.io/v1, Kind=NetworkPolicy", "resource": {"group":"networking.k8s.io","version":"v1","resource":"networkpolicies"}}
2020-08-04T16:43:17.003Z        DEBUG   controller-runtime.webhook.webhooks     wrote response  {"webhook": "/validating-v1-network-policy", "UID": "e3ab89ac-3311-45b8-9c59-056a253bdcef", "allowed": false, "result": {}, "resultError": "got runtime.Object without object metadata: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:,Message:,Reason:Capsule Network Policies cannot be deleted: please, reach out the system administrators,Details:nil,Code:403,}"}

Additional context

The tenant nodeSelector is not updated correctly

Bug description

When updating the tenant nodeSelector in the tenant manifest, the new selector is just appended in the tenant namespaces instead of substituted.

How to reproduce

Steps to reproduce the behaviour:

Having a tenant manifest with a given node selector

  nodeSelector:
    kubernetes.io/os: linux

as tenant owner, create a namespace

$ kubectl create ns usl1

and check the namespace is correctly labeled

apiVersion: v1
kind: Namespace
metadata:
  name: usl1
  annotations:
    scheduler.alpha.kubernetes.io/node-selector: kubernetes.io/os=linux

as cluster admin, update the tenant above as following:

  nodeSelector:
    kubernetes.io/hostname: cmp

back as tenant owner and check the namespace

apiVersion: v1
kind: Namespace
metadata:
  name: usl1
  annotations:
    scheduler.alpha.kubernetes.io/node-selector: kubernetes.io/hostname=cmp,kubernetes.io/os=linux

Expected behavior

As cluster admin, it should be possible to update correctly the tenant node selector

Logs

Additional context

Capsule version: build for #30
Kubernetes version: irrelevant

Support for custom user group

Describe the feature

Hi guys, thank's a lot for a great tool for managing multi-tenant k8s :)

We are now trying to adopt it in our environment and found that right now it's impossible to use custom UserInfo.Group in webhooks so they apply only for specific users, as it's hardcoded as capsulev1alpha1.GroupVersion.Group. As we already have a dedicated AD group for k8s users, it will be good if we can use it with Capsule too, without a need to provision one more group and move users into it.

I will submit a PR with this small patch.

Thanks,
Max

Capsule should not run as Cluster Admin

Describe the feature

During the reconciliation loop, Capsule will ensure that the admin Role is going to be granted to the Tenant owner for their Namespace resources: this is actually mandatory with the current RBAC setup, otherwise, Capsule wouldn't be able to grant RBAC permissions that are not currently held.

What would the new user story look like?

We should investigate how to decrease the RBAC permissions to the minimum but, at the same time, allows Capsule to grant admin role to Tenants.

Probably we need to use the bind verb at RBAC for the admin resources at the cluster scope.

Expected behavior

Capsule would be running as a non Cluster admin and eligible to bind admin role to tenant owners without getting the following error:

user "system:serviceaccount:capsule-system:capsule" (groups=["system:serviceaccounts" "system:serviceaccounts:capsule-system" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: (...)

Namespace count additional printer column is empty when creating a new Tenant

Bug description

Not properly a bug, it's just something stylish: when creating a Tenant the additional printer column NAMESPACE COUNT is not reporting zero

NAME   NAMESPACE QUOTA   NAMESPACE COUNT
oil    3

Once the Tenant owner creates a Namespace and deletes it, the counter is successfully computed.

NAME   NAMESPACE QUOTA   NAMESPACE COUNT
oil    3                 0

How to reproduce

Create the Tenant
Get the Tenant from the CLI

Expected behavior

After Tenant creation, kubectl get tenants should output zero namespaces.

NAME   NAMESPACE QUOTA   NAMESPACE COUNT
oil    3                 0

Logs

Irrelevant.

Additional context

Capsule version: 0.0.1
Kubernetes version: irrelevant

Review RetryOnConflict usages

Bug description

In multiple places, there are retry.RetryOnConflict usages that are unnecessary or misused.

https://github.com/clastix/capsule/blob/ee469ce2e78f478c123478203746745478e4ae4b/controllers/tenant_controller.go#L566-L569

https://github.com/clastix/capsule/blob/a29db95ad3d04fce6664fd7c6693f935ac44b328/controllers/namespace_controller.go#L172-L181

For both the above Update calls, a conflicting update should never happen given the leader's election.

https://github.com/clastix/capsule/blob/a29db95ad3d04fce6664fd7c6693f935ac44b328/controllers/namespace_controller.go#L163-L166

The above needs to retrieve the namespace and update it inside the callback.

Provide an endpoint for liveness and readiness probes

Describe the feature

As best practice, we should define readiness and liveness probes for any container providing a service and this is also true for capsule controller being a critical service.

Currently the capsule controller exposes only the web-hook port 9443 but this port is not suitable for probing:

$ curl -k https://10.38.0.29:9443
404 page not found

$ curl -k https://10.38.0.29:9443/validating-v1-pvc
{"response":{"uid":"","allowed":false,"status":{"metadata":{},"message":"contentType=, expected application/json","code":400}}}

Expected behavior

We need a couple of endpoints for liveness and readiness probing, for example:

$ curl -k https://10.38.0.29:10080/health
ok

$ curl -k https://10.38.0.29:10080/ready
ok

so we can write readiness and liveness probes:

    livenessProbe:
      failureThreshold: 5
      httpGet:
        path: /health
        port: 10080
        scheme: HTTP
      initialDelaySeconds: 60
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    readinessProbe:
      failureThreshold: 5
      httpGet:
        path: /ready
        port: 10080
        scheme: HTTP
      initialDelaySeconds: 60
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5

Readiness should return ok only when all the stuff are ready, for example when the CA generation succeeded.

Fix typos in the docs

Pretty straightforward, there are several typos in the documentation that sounds terrible in the English language.

Help is wanted, so please don't hesitate to refer to this issue while contributing to fixing small typos on Capsule! 🎉

Document how to contribute

A document explaining how to contribute to this project would be great, explaining how to setup the development environment, needed tools, coding style, rules of conduct, etc..

Tenant without Limit Range or Network Policy resources is not able to reconciliate

Bug description

Actually, Capsule is not able to handle a Tenant with no Resource Quota or Limit Range resource due to the pruning funcion.
https://github.com/clastix/capsule/blob/7661bbb0ad770d8f9773e3c02aa76ac1b53a602d/controllers/tenant_controller.go#L126-L156

The issue is the notIn condition, since if keys are mandatory and with an empty list the requirement is using an empty string slice.

How to reproduce

Deploy the following Tenant

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: example
spec:
  limitRanges: []
  resourceQuota: []
...

Check Capsule logs

Expected behavior

The Tenant reconciliation loop should be performed without errors.

Logs

{"level":"info","ts":1597244835.3131397,"logger":"controllers.Tenant","msg":"Starting processing of Namespaces","Request.Name":"oil","items":1}
{"level":"info","ts":1597244835.3191206,"logger":"controllers.Tenant","msg":"Starting processing of Network Policies","Request.Name":"oil","items":0}
{"level":"error","ts":1597244835.3191724,"logger":"controllers.Tenant","msg":"Cannot sync NetworkPolicy items","Request.Name":"oil","error":"for 'in', 'notin' operators, values set can't be empty","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\ngithub.com/clastix/capsule/controllers.TenantReconciler.Reconcile\n\t/workspace/controllers/tenant_controller.go:88\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90"}
{"level":"error","ts":1597244835.319267,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"tenant","request":"/oil","error":"for 'in', 'notin' operators, values set can't be empty","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90"}

Additional context

Capsule version:

{"level":"info","ts":1597244816.9130142,"logger":"setup","msg":"Operator Version: "}                                                                                                       
{"level":"info","ts":1597244816.9131448,"logger":"setup","msg":"Go Version: go1.13.14"}
{"level":"info","ts":1597244816.9131947,"logger":"setup","msg":"Go OS/Arch: linux/amd64"}

Kubernetes version: N.R.

Support for additional labels and annotations to be inherited by tenant services

Describe the feature

The idea is to leave an additional section in the Tenant manifest where the Cluster admin can specify additional labels and/or annotations to apply to the services created in the tenant. Initially, there are no specific semantics assigned to these labels, they just will be assigned to the services and endpoints in the tenant. This can be useful to the adopters of Capsule to implement their specific use cases.

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: oil
spec:
  additionalServiceLabels:
    capsule.clastix.io/do_stuff: "always"

...

What would the new user story look like?

The cluster admin creates a tenant with this additional fields
Tenant owner creates a new Namespace
Tenant owner creates new services in the tenant namespace
All the services and related endpoints created in this tenant will inherit the labels
Depending on the case, additional business logics can be implemented by using the labels.

Please note the specific business logic is not part of the Capsule. Instead it is left to the cluster admin and or to the tenant owner.

Expected behavior

All the services and related endpoints created in the tenant inherit the labels.

Control logs verbosity

Moving to Operator SDK 0.19, the --zap-logger for controlling logs verbosity is no more available.

Meet the multi-tenancy benchmarks requirements

Pass the benchmarks test as specified by SIG multi-tenancy benchmarks.

Target for Capsule should be the L2 profile, being the L3 out-of-scope:

Level 3
This profile extends the "Level 2" profile. Items in this profile:

are intended for use where a higher-level of multi-tenancy is paramount

allow usage of all Kubernetes features. For example, a tenant may install their own CRD versions.

Support for protected-namespace-regexp option

Describe the feature

We had a concept of protected namespaces, e.g. we do not allow users to create namespaces ending on -system$ , as they had special monitoring and other rules, which are applied to such NS. So I would like to add additional flag --protected-namespace-regexp and modify tenant_prefix webhook, so will also reject creation of namespaces, which are covered by this regexp.

Dedicated PodSecurityPolicy per Tenant

Describe the feature

It would be nice to let the cluster admin to set a dedicated Pod Security Policy (PSP) per tenant. This is likely to be a requirement in a multi-tenancy environment.

A PSP is defined at cluster scope. To assign the PSP to users/serviceaccounts in a tenant we can use RoleBindings for each namespace created in the tenant. The Capsule should enforce the creation of RoleBinding assigning the PSP.

What would the new user story look like?

The cluster admin creates a PSP for a given tenant:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp:restricted
spec:
  privileged: false
  # Required to prevent escalations to root.
  allowPrivilegeEscalation: false
  ...

then creates a ClusterRole to assign the PSP

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp:restricted
rules:
- apiGroups:
  - extensions
  resources:
  - podsecuritypolicies
  resourceNames:
  - restricted # the psp we are giving access to
  verbs:
  - use

the cluster admin assign such PSP to the tenant:

  apiVersion: capsule.clastix.io/v1alpha1
  kind: Tenant
  metadata:
      name: oil
  spec:
    podSecurityPolicies:
    - psp:restricted
 ...

Expected behavior

For each namespace created into the tenant, the Capsule controller creates a RoleBinding assigning the PSP to all the services accounts in such namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: psp:restricted
  namespace:
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: psp:restricted
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts