clusternet / clusternet Goto Github PK

What happened:

kind/security

Dependabot alerts multiple security vulnerabilities in some dependencies.

• github.com/containerd/containerd
  Upgrade github.com/containerd/containerd to version 1.4.8 or later

• github.com/opencontainers/runc
  Upgrade github.com/opencontainers/runc to version 1.0.0-rc95 or later

• github.com/dgrijalva/jwt-go

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Clusternet version:
  • Clusternet-agent version (user clusternet-agent --version=json):
  • Clusternet-hub version (user clusternet-hub --version=json):
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

What happened:

When following the guide: https://github.com/clusternet/clusternet#visit-managedcluster-with-rbac

It will be a problem about how to pass the user cred to managed cluster.
I suffer from "forbidden: User "system:anonymous" cannot get path "/api"" error .
api-server does not set -anonymous-auth=false explicitly.

What you expected to happen:

Using curl + token to add Child Cluster API, or use kubectl to access it .

How to reproduce it (as minimally and precisely as possible):

Neither proxy/https mode or proxy/direct mode, you will have to find a way to satisfy Auth of **Child Cluster **.
So I tried to specific cert/key in your kube.conf or token in curl header (--header "Authorization: Bearer $TOKEN" )

It will be looked like (curl to access http proxy child cluster )

curl --header "Authorization: Bearer $TOKEN"  \
https://${HUB_CLUSTER_IP}:6443/apis/proxies.clusternet.io/v1alpha1/sockets/7a93727c-6609-45bc-8c3e-6556cb89cc2b/proxy/https/${CHILD_CLUSTER_IP}:6443/api/v1/nodes

At first, I thought the TOKEN should be CHILD-Cluster authorized token . So I picked an admin privilege token of child cluster.

Before sending it to Hub Cluster, I verified the token with directly access the Child cluster without clusterNet. token is good.✅

curl --header "Authorization: Bearer $TOKEN" https://${CHILD_CLUSTER_IP}:6443/api/v1/nodes -k
# the result is good, return the nodes to me 

But with clusterNet proxy ,using the same $TOKEN (Child Cluster token), it failed。❌

curl --header "Authorization: Bearer $TOKEN"  https://${HUB_CLUSTER_IP}:6443/apis/proxies.clusternet.io/v1alpha1/sockets/7a93727c-6609-45bc-8c3e-656cb89cc2b/proxy/https/${CHILD_CLUSTER_IP}:6443/api/v1/nodes -k
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

I believe it's due to Hub Cluster reject this token . So the first gate(Hub Cluster AuthZ) blocks request.

Ok, I change the $TOKEN to Hub Cluster admin token.
Now I can access Child Cluster's /healthz (this API does not require auth), ✅ like below

curl --header "Authorization: Bearer $TOKEN"  https://${HUB_CLUSTER_IP}:6443/apis/proxies.clusternet.io/v1alpha1/sockets/7a93727c-6609-45bc-8c3e-656cb89cc2b/proxy/https/${CHILD_CLUSTER_IP}:6443/healthz  -k

But it still failed for other API( like /apis/v1/nodes ), ❌ error will be :

 "message": "forbidden: User \"system:anonymous\" cannot get path \"/ping\"",

Same problem , I'm also confused the user config in kube-config.
For Child Cluster configuration in kube.conf:

• The ca cert(certificate-authority-data) should be Hub-Cluster cert
• But the user , I copied client-certificate-data and client-key-data from child cluster kube config file.

Below is my kubectl config, the failure was

kubectl config use-context k8s-21-child
kubectl get no  -v=9

Error:
[] GET https://172.**.**.20:6443/apis/proxies.clusternet.io/v1alpha1/sockets/7a93727c-6609-45bc-8c3e-6556cb89cc2b/proxy/https/172.**.**.21:6443/api?timeout=32s 403 Forbidden in 16 milliseconds
[] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"forbidden: User \"system:anonymous\" cannot get path \"/api\"","reason":"Forbidden","details":{},"code":403}

[root@my-172-**-**-20 ]# kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://172.**.**.20:6443
  name: k8s-20-parent
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://172.**.**.20:6443/apis/proxies.clusternet.io/v1alpha1/sockets/7a93727c-6609-45bc-8c3e-6556cb89cc2b/proxy/https/172.**.**.21:6443
  name: k8s-21-child
contexts:
- context:
    cluster: k8s-20-parent
    user: kubernetes-admin20
  name: k8s-20-parent
- context:
    cluster: k8s-21-child
    user: kubernetes-admin20
  name: k8s-21-child
current-context: k8s-21-child
kind: Config
preferences: {}
users:
- name: kubernetes-admin20
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED
- name: kubernetes-admin21
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

Anything else we need to know?:

Environment:

• Clusternet version:
  • Clusternet-agent version (user clusternet-agent --version=json): latest . 0.4.0
  • Clusternet-hub version (user clusternet-hub --version=json): 0.4.0
• Kubernetes version (use kubectl version): k8s: 1.19.13(build by kubeadm ). kubectl binary : v1.18.20
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

What would you like to be added:

Allowing visiting child clusters with RBAC rules.

Why is this needed:

Currently only http scheme is supported when visiting child clusters.

What happened:

When the description apply failed, the description controller will not reconcile

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Clusternet version:
  • Clusternet-agent version (user clusternet-agent --version=json):
  • Clusternet-hub version (user clusternet-hub --version=json):
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

What happened:

Tracking Kubernetes version skew issues when using with Clusternet

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

status:
conditions:

• lastProbeTime: null
  lastTransitionTime: "2021-08-30T09:20:12Z"
  message: Marking condition "cloud.tencent.com/load-balancer-backendgroup-ready"
  to True.
  reason: LoadBalancerNetworkGroupReady
  status: "True"
  type: cloud.tencent.com/load-balancer-backendgroup-ready
• lastProbeTime: null
  lastTransitionTime: "2021-08-30T09:20:12Z"
  status: "True"
  type: Initialized
• lastProbeTime: null
  lastTransitionTime: "2021-08-30T09:20:12Z"
  message: 'containers with unready status: [clusternet-hub]'
  reason: ContainersNotReady
  status: "False"
  type: Ready
• lastProbeTime: null
  lastTransitionTime: "2021-08-30T09:20:12Z"
  message: 'containers with unready status: [clusternet-hub]'
  reason: ContainersNotReady
  status: "False"
  type: ContainersReady
• lastProbeTime: null
  lastTransitionTime: "2021-08-30T09:20:12Z"
  status: "True"
  type: PodScheduled
  containerStatuses:
• image: ghcr.io/clusternet/clusternet-hub:v0.3.0
  imageID: ""
  lastState: {}
  name: clusternet-hub
  ready: false
  restartCount: 0
  started: false
  state:
  waiting:
  reason: ContainerCreating
  hostIP: 10.228.49.71
  phase: Pending
  qosClass: BestEffort
  startTime: "2021-08-30T09:20:12Z"

What would you like to be added:

• #359
• #360

Why is this needed:

When a workload is scheduled to multiple clusters, the aggregated status is unknown.

 k get mcls -A
NAMESPACE          NAME                       CLUSTER ID                             SYNC MODE   KUBERNETES   READYZ   AGE
clusternet-9p2pf   clusternet-cluster-pn4g4   e0a73457-33c1-4528-b47f-caf422d29760   Push        v1.21.1      true     39h

@dixudx I know the cluster heartbeat triggered by agent which cause cluster status show alive forever even if agent get removed from target cluster. This issue cause CLI user and third party integrator confuse, could you please do this enhancement in clustenet, thanks in advance.

What would you like to be added:

Hi, Greeting:

In use case, the client will do the following step to access Child Cluster:
(1) list all child clusters from hub cluster
(2) retrieve the cluster id and api-server endpoint, for one child
(3) using prepared cert or token , to access child cluster by AA API

But the key point is : how to get the real child cluster "api-server endpoint"?

The apiserverURL in mcls is APIServer's Cluster IP(https://10.96.0.1:443), instead of external reachable IP.

kubectl clusternet get mcls.clusters.clusternet.io -n clusternet-g4wj5 -o yaml

apiVersion: v1
items:
- apiVersion: clusters.clusternet.io/v1beta1
  kind: ManagedCluster
....
  status:
    apiserverURL: https://10.96.0.1:443
    clusterCIDR: 10.96.0.0/12
....

Why is this needed:

What would you like to be added:

Why is this needed:

What happened:

  subscribers:
  - clusterAffinity:
      matchExpressions:
      - key: clusters.clusternet.io/cluster-name
        operator: In
        values:
        - cls-27rq50l2

E0923 10:46:39.327280       1 subscription.go:358] error syncing 'clusternet-subs/wordpress3-sub': values[0][cls-27rq50l2]: Invalid value: "clusters.clusternet.io/cluster-name": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?'), requeuing

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Clusternet version:
  • Clusternet-agent version (user clusternet-agent --version=json):
  • Clusternet-hub version (user clusternet-hub --version=json):
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

What would you like to be added:

Support add username, password to repo in HelmChart

Why is this needed:

apiVersion: apps.clusternet.io/v1alpha1
kind: HelmChart
metadata:
  name: example
  namespace: default
spec:
  repo: https://private-repo.com/test
  chartPullSecret:
    name: my-private-repo-auth
    namespace: kube-system
  chart: example
  version: 0.0.1
  targetNamespace: abc

apiVersion: v1
data:
  username: xxx
  password: xxxx
kind: Secret
metadata:
  name: my-private-repo-auth
  namespace: kube-system
type: Opaque

What would you like to be added:

Why is this needed:

When CRDs/APIServices are created or updated, shadow apis/scheme should be updated accordingly.

What would you like to be added:

Abstract a set of apis for cluster-agent，like virtual-kubelet provider

Why is this needed:

Scenario: I don’t have permission to install the agent component in the member cluster, and hope to be able to clusternet-agent as a pluggable component

Whenever a child cluster joins the parent cluster, it would get a generated clusterID. If there're multiple child clusters, how can I tell which clsrr/mcls belongs to which child cluster? Is there anyway I can specify this clusterID in clusternet-agent manifest?

What happened:

clusternet-hub print error msg , but It seems that application distribution is normal.

I0909 07:06:10.472336       1 httplog.go:89] "HTTP" verb="GET" URI="/apis/proxies.clusternet.io/v1alpha1" latency="1.19169ms" userAgent="Go-http-client/2.0" srcIP="172.17.0.1:21105" resp=200
E0909 07:06:10.472513       1 writers.go:117] apiserver was unable to write a JSON response: http2: stream closed
E0909 07:06:10.472528       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
I0909 07:06:10.472549       1 httplog.go:89] "HTTP" verb="GET" URI="/apis/shadow/v1alpha1" latency="3.710677ms" userAgent="Go-http-client/2.0" srcIP="172.17.0.1:21105" resp=200
E0909 07:06:10.472561       1 wrap.go:54] timeout or abort while handling: GET "/apis/shadow/v1alpha1"
E0909 07:06:10.472845       1 writers.go:117] apiserver was unable to write a JSON response: http2: stream closed
I0909 07:06:10.472940       1 httplog.go:89] "HTTP" verb="GET" URI="/apis/proxies.clusternet.io/v1alpha1" latency="2.995015ms" userAgent="Go-http-client/2.0" srcIP="172.17.0.1:21105" resp=200
I0909 07:06:10.472948       1 httplog.go:89] "HTTP" verb="GET" URI="/apis/shadow/v1alpha1" latency="2.875673ms" userAgent="Go-http-client/2.0" srcIP="172.17.0.1:21105" resp=200
E0909 07:06:10.472963       1 wrap.go:54] timeout or abort while handling: GET "/apis/shadow/v1alpha1"
I0909 07:06:10.473122       1 httplog.go:89] "HTTP" verb="GET" URI="/apis/shadow/v1alpha1" latency="740.108µs" userAgent="Go-http-client/2.0" srcIP="172.17.0.1:21105" resp=200
E0909 07:06:10.473733       1 writers.go:130] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0909 07:06:10.475531       1 writers.go:117] apiserver was unable to write a JSON response: http2: stream closed
I0909 07:06:10.475573       1 httplog.go:89] "HTTP" verb="GET" URI="/apis/shadow/v1alpha1" latency="2.595901ms" userAgent="Go-http-client/2.0" srcIP="172.17.0.1:21105" resp=200
E0909 07:06:10.475587       1 wrap.go:54] timeout or abort while handling: GET "/apis/shadow/v1alpha1"
I0909 07:06:10.476185       1 httplog.go:89] "HTTP" verb="GET" URI="/apis/proxies.clusternet.io/v1alpha1" latency="7.400501ms" userAgent="Go-http-client/2.0" srcIP="172.17.0.1:21105" resp=200
I0909 07:06:10.477526       1 httplog.go:89] "HTTP" verb="GET" URI="/apis/shadow/v1alpha1" latency="4.885754ms" userAgent="Go-http-client/2.0" srcIP="172.17.0.1:21105" resp=200
E0909 07:06:10.483204       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0909 07:06:10.487024       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0909 07:06:10.489381       1 writers.go:130] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0909 07:06:10.490731       1 writers.go:130] apiserver was unable to write a fallback JSON response: http: Handler timeout
I0909 07:06:10.611183       1 httplog.go:89] "HTTP" verb="GET" URI="/apis/shadow/v1alpha1/virtualmachineinstancereplicasets?allowWatchBookmarks=true&resourceVersion=601545&timeout=7m56s&timeoutSeconds=476&watch=true" latency="7m56.008358351s" userAgent="kube-controller-manager/v1.20.2 (linux/amd64) kubernetes/faecb19/metadata-informers" srcIP="172.17.0.1:20471" resp=0
I0909 07:06:10.618348       1

What you expected to happen:

normal

How to reproduce it (as minimally and precisely as possible):

Install clusternet components via readme

Anything else we need to know?:

I am using minikube, the host cluster and sub-cluster are the same.

Environment:

• Clusternet version:
  • Clusternet-agent version (user clusternet-agent --version=json):
  • Clusternet-hub version (user clusternet-hub --version=json):

{
  "programName": "clusternet-hub",
  "major": "0",
  "minor": "4",
  "gitVersion": "v0.4.0",
  "gitCommit": "4fb52f73a246b321715b5b93637650389c57d6dc",
  "gitTreeState": "clean",
  "buildDate": "2021-09-08T05:31:55Z",
  "goVersion": "go1.14.15",
  "compiler": "gc",
  "platform": "linux/amd64"
}
{
  "programName": "clusternet-agent",
  "major": "0",
  "minor": "4",
  "gitVersion": "v0.4.0",
  "gitCommit": "4fb52f73a246b321715b5b93637650389c57d6dc",
  "gitTreeState": "clean",
  "buildDate": "2021-09-08T05:16:42Z",
  "goVersion": "go1.14.15",
  "compiler": "gc",
  "platform": "linux/amd64"
}

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:58:09Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):

minikube on macos:
minikube version: v1.18.1
commit: 09ee84d530de4a92f00f1c5dbc34cead092b95bc

• Kernel (e.g. uname -a):

Linux minikube 4.19.171 #1 SMP Tue Feb 23 14:35:29 PST 2021 x86_64 GNU/Linux

• Others:

What would you like to be added:

Why is this needed:

What happened:

I deploy applications to Multiple Clusters with reference to https://github.com/clusternet/clusternet/blob/main/docs/tutorials/deploying-applications-to-multiple-clusters.md
while result to a failure. There is no Description created.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

installed with helm the version is v0.5.0.
these message may help to figure out..

[root@global-master-1 clusternet]# kubectl create -f examples/applications/
globalization.apps.clusternet.io/mysql-global-overrides-demo created
globalization.apps.clusternet.io/nginx-global-overrides-demo created
helmchart.apps.clusternet.io/mysql created
localization.apps.clusternet.io/mysql-local-overrides-demo created
localization.apps.clusternet.io/nginx-local-overrides-demo-higher-priority created
localization.apps.clusternet.io/nginx-local-overrides-demo-lower-priority created
namespace/foo created
deployment.apps/my-nginx created
service/my-nginx-svc created
subscription.apps.clusternet.io/app-demo created


[root@global-master-1 clusternet]# kubectl  get helmcharts.apps.clusternet.io
NAME    CHART   VERSION   REPO                                 STATUS   AGE
mysql   mysql   8.6.2     https://charts.bitnami.com/bitnami   Found    5s

[root@global-master-1 clusternet]# kubectl  describe base -n clusternet-h487b
Spec:
  Feeds:
    API Version:  apps.clusternet.io/v1alpha1
    Kind:         HelmChart
    Name:         mysql
    Namespace:    default
    API Version:  v1
    Kind:         Namespace
    Name:         foo
    API Version:  v1
    Kind:         Service
    Name:         my-nginx-svc
    Namespace:    foo
    API Version:  apps/v1
    Kind:         Deployment
    Name:         my-nginx
    Namespace:    foo
Events:
  Type     Reason                Age                From            Message
  ----     ------                ----               ----            -------
  Warning  VerifyingHelmChart    9s (x10 over 12s)  clusternet-hub  HelmChart default/mysql is in verifying
  Warning  FailedSynced          9s (x10 over 12s)  clusternet-hub  HelmChart default/mysql is in verifying
  Warning  NonexistentNamespace  1s (x2 over 6s)    clusternet-hub  Base clusternet-h487b/app-demo is using a nonexistent Namespace foo
  Warning  FailedSynced          1s (x2 over 6s)    clusternet-hub  Base clusternet-h487b/app-demo is using a nonexistent Namespace foo

What would you like to be added:

As title said, third-party operators may import clusternet apis and use client-go to interact with clusternet.

Why is this needed:

What would you like to be added:

Support to add the label of the mcls object through parameters when the agent is started.

like :

/usr/local/bin/clusternet-agent --labels="region=shenzhen,zone=zone1"

Why is this needed:

Although the label can be added by manually editing the mcls object, it is still hoped that the label can be specified directly in the agent through parameters (such as area and other information)

The project README currently states

"thousands of millions" would be "billions" of clusters. Can this project really scale that high? Do you have any performance benchmarks to substantiate the claim? And do you have any metrics on what kind of resources this would cost?

The reason I ask is we have thousands (potentially hundreds of thousands) of agents that we'd like to manage in one central hub. This project appears to do what we need it to do, but the claims seem outlandish (sorry if I sounds skeptical) given that Kubernetes itself has much more conservative limitations as to the number of resources it can manage and reconcile in a timely manner.

What would you like to be added:

Example:

status:
  allocatable:
    cpu: "8"
    memory: 15753Mi
  capacity:
    cpu: "8"
    memory: 15753Mi
  clusterCidr: 10.44.0.0/24
  serviceCidr: 10.45.0.0/24

Why is this needed:

What happened:

When I deploy clusternet-agent, the container logs error:

E0626 13:41:08.159174       1 leaderelection.go:329] error initially creating leader election record: namespaces "edge-system" not found

What you expected to happen:

No error log, no need namespace edge-system

How to reproduce it (as minimally and precisely as possible):

With docker image: ghcr.io/clusternet/clusternet-agent :v0.1.0, you will reproduce it.

Because I haven't find any code about edge-system, and I find there was a default namespace named edge-system [#4 ].

Maybe we need update the image in ghcr.io.

What would you like to be added:

(1)
kubectl clusternet edit deploy xxxchange a label of deploy or any other change for workload.

but the desc object will not be updated, and the real deploy in child clusters not updated neither.

This is a normal case in Day2 , updating an existing workload from hub cluster .

(2)
I manually edit the workload on child cluster, e.g. : edit the deployment.
I expect the agent will watch the desc and reconcile that deployment to original state (aka, rollback my manual change)
But it didn't happen.
KubeFed can achieve this .

Why is this needed:

i just want to ensure if global cluster breakdown,can we make a work cluster to be global cluster,just like redis.
it can't provide service if global cluster breakdown and have have only one global cluster

What would you like to be added:

Why is this needed:

Want to know the roadmap of this project, for example, including cross-cluster service discovery.

What happened:

default go client enable the redirect func, but clusternet proxy will response 301 with a internal IP, which the client can't access.

What you expected to happen:

should not response 301.

How to reproduce it (as minimally and precisely as possible):

curl -s -f -I -L -k https://{pub_IP}:6443/apis/proxies.clusternet.io/v1alpha1/sockets/xxxxxxxx/proxy/direct

Anything else we need to know?:

Environment:

• Clusternet version:
  • Clusternet-agent version (user clusternet-agent --version=json):
  • Clusternet-hub version (user clusternet-hub --version=json):
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

What would you like to be added:

Bump default namespace to clusternet-system

What would you like to be added:

Why is this needed:

What would you like to be added:

Supporting UI, just like Rancher

Why is this needed:

Easy to use.

What happened:

Error log:

#  kubectl  logs -n clusternet-system -l app=clusternet-agent
...
E0627 07:00:04.512265       1 leaderelection.go:325] error retrieving resource lock edge-system/self-cluster: leases.coordination.k8s.io "self-cluster" is forbidden: User "system:serviceaccount:clusternet-system:clusternet-agent" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "edge-system"
I0627 07:00:04.512287       1 leaderelection.go:248] failed to acquire lease edge-system/self-cluster
E0627 07:00:11.405519       1 leaderelection.go:325] error retrieving resource lock edge-system/self-cluster: leases.coordination.k8s.io "self-cluster" is forbidden: User "system:serviceaccount:clusternet-system:clusternet-agent" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "edge-system"
I0627 07:00:11.405572       1 leaderelection.go:248] failed to acquire lease edge-system/self-cluster

[root@beijing ~]# kubectl api-resources | grep lease
leases                                         coordination.k8s.io/v1                 true         Lease

What you expected to happen:

Run correctly in k8s 1.20 and newer

How to reproduce it (as minimally and precisely as possible):

Deploy the clusternet-agent in k8s 1.20

Anything else we need to know?:

The lease resources version is changed in 1.20

# in k8s 1.20.0
# kubectl api-resources | grep lease
leases                                         coordination.k8s.io/v1                 true         Lease

# in k8s 1.19.7
# kubectl api-resources | grep lease
leases                                               coordination.k8s.io            true         Lease

Environment:

• Clusternet version: v0.1.0

What happened:

In different k8s clusters, service-cluster-ip-range is in different paths.
For some clusters, service-cluster-ip-range is located under the path spec.containers[0].command, for some clusters, service-cluster-ip-range is located under the path spec.containers[0].args

We need try to find the cluster cidr from spec.containers[0].command and spec.containers[0].args

In the kind test cluster

➜  ~ kubectl get po kube-apiserver-cluster1-control-plane -n kube-system -oyaml
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver-cluster1-control-plane
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=172.18.0.3
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --runtime-config=
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=10.96.0.0/16
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

In the TKE cluster

$ kubectl get po kube-apiserver-172.16.0.176 -n kube-system -oyaml
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver-172.16.0.176
  namespace: kube-system
spec:
  containers:
  - args:
    - --proxy-client-key-file=/etc/kubernetes/certs/kube-apiserver-front-proxy-client.key
    - --client-ca-file=/etc/kubernetes/certs/apiserver-client-ca.crt
    - --etcd-certfile=/etc/kubernetes/certs/kube-apiserver-etcd-client.crt
    - --service-account-key-file=/etc/kubernetes/certs/kube-apiserver-service-account.key
    - --requestheader-client-ca-file=/etc/kubernetes/certs/kube-apiserver-proxy-request-header-client-ca.crt
    - --etcd-cafile=/etc/kubernetes/certs/kube-apiserver-etcd-ca.crt
    - --kubelet-client-key=/etc/kubernetes/certs/kube-apiserver-kubelet-client-key.key
    - --requestheader-allowed-names=
    - --service-cluster-ip-range=10.27.252.0/22
    - --proxy-client-cert-file=/etc/kubernetes/certs/kube-apiserver-front-proxy-client.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-username-headers=X-Remote-User
    - --bind-address=0.0.0.0
    - --tls-private-key-file=/etc/kubernetes/certs/kube-apiserver.key
    - --etcd-servers=https://etcd.cls-nsbwqz74.ccs.tencent-cloud.com:2379
    - --kubelet-client-certificate=/etc/kubernetes/certs/kube-apiserver-kubelet-client-cert.crt
    - --allow-privileged=true
    - --requestheader-group-headers=X-Remote-Group
    - --secure-port=60002
    - --tls-cert-file=/etc/kubernetes/certs/kube-apiserver.crt
    - --etcd-keyfile=/etc/kubernetes/certs/kube-apiserver-etcd-client.key
    - --cloud-provider=qcloud
    - --etcd-prefix=cls-nsbwqz74
    - --authorization-mode=RBAC,Node
    - --token-auth-file=/etc/kubernetes/known_tokens.csv
    - --enable-admission-plugins=PersistentVolumeLabel,NodeRestriction
    - --external-hostname=cls-nsbwqz74.ccs.tencent-cloud.com
    - --kubelet-preferred-address-types=InternalIP,ExternalIP
    - --advertise-address=169.254.128.3
    - --cloud-config=/etc/kubernetes/qcloud.conf
    - --enable-aggregator-routing=true
    - --basic-auth-file=/etc/kubernetes/basic_auth.csv
    - --enable-bootstrap-token-auth=true
    - --audit-log-maxage=7
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    - --audit-log-path=/etc/kubernetes/kubernetes.audit
    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    command:
    - kube-apiserver

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Clusternet version:
  • Clusternet-agent version (user clusternet-agent --version=json):
  • Clusternet-hub version (user clusternet-hub --version=json):
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

What happened:

Append a new matchLabels "clusters.clusternet.io/cluster-id": "c817745f-f46b-4a2b-95fd-86521c81cc79" in the subscription, the original matchLabels "clusters.clusternet.io/cluster-id": "46919132-9639-4c72-a187-975d8cfe0f6c" will lost

☁  ~  k get sub helm-demo -ojson | jq '.spec.subscribers'
[
  {
    "clusterAffinity": {
      "matchLabels": {
        "clusters.clusternet.io/cluster-id": "c817745f-f46b-4a2b-95fd-86521c81cc79"
      }
    }
  }
]

What you expected to happen:

☁  ~  k get sub helm-demo -ojson | jq '.spec.subscribers'
[
  {
    "clusterAffinity": {
      "matchLabels": {
        "clusters.clusternet.io/cluster-id": "46919132-9639-4c72-a187-975d8cfe0f6c"
        "clusters.clusternet.io/cluster-id": "c817745f-f46b-4a2b-95fd-86521c81cc79"
      }
    }
  }
]

How to reproduce it (as minimally and precisely as possible):

There are 2 managed cluster, and a subscription in my test env.

☁  ~  k get mcls -A
NAMESPACE          NAME                       CLUSTER ID                             SYNC MODE   KUBERNETES   READYZ   AGE
clusternet-4jxfq   clusternet-cluster-l2bmc   46919132-9639-4c72-a187-975d8cfe0f6c   Dual        v1.21.1      true     23h
clusternet-dwgrd   clusternet-cluster-kx2b2   c817745f-f46b-4a2b-95fd-86521c81cc79   Dual        v1.21.1      true     23h
☁  ~  cat sub.yaml
apiVersion: apps.clusternet.io/v1alpha1
kind: Subscription
metadata:
  name: helm-demo
  namespace: default
spec:
  subscribers:
    - clusterAffinity:
        matchLabels:
          clusters.clusternet.io/cluster-id: 46919132-9639-4c72-a187-975d8cfe0f6c
  feeds:
    - apiVersion: apps.clusternet.io/v1alpha1
      kind: HelmChart
      name: mysql
      namespace: default
    - apiVersion: apps.clusternet.io/v1alpha1
      kind: HelmChart
      namespace: default
      feedSelector:
        matchLabels:
          app: wordpress

Apply this subscription, and check hr, looks good so far.

☁  ~  k apply -f sub.yaml
subscription.apps.clusternet.io/helm-demo created
☁  ~  k get hr -A
NAMESPACE          NAME                  CHART       VERSION   REPO                                 STATUS     AGE
clusternet-4jxfq   helm-demo-mysql       mysql       8.6.2     https://charts.bitnami.com/bitnami   deployed   20s
clusternet-4jxfq   helm-demo-wordpress   wordpress   11.0.17   https://charts.bitnami.com/bitnami   deployed   20s

Edit the subscription, and append a new matchLabels clusters.clusternet.io/cluster-id: c817745f-f46b-4a2b-95fd-86521c81cc79 in this sub.

Expect result:,

☁  ~  k get sub helm-demo -ojson | jq '.spec.subscribers'
[
  {
    "clusterAffinity": {
      "matchLabels": {
        "clusters.clusternet.io/cluster-id": "46919132-9639-4c72-a187-975d8cfe0f6c"
        "clusters.clusternet.io/cluster-id": "c817745f-f46b-4a2b-95fd-86521c81cc79"
      }
    }
  }
]

After edit subscription,

☁  ~  k get sub helm-demo -ojson | jq '.spec.subscribers'
[
  {
    "clusterAffinity": {
      "matchLabels": {
        "clusters.clusternet.io/cluster-id": "c817745f-f46b-4a2b-95fd-86521c81cc79"
      }
    }
  }
]

Anything else we need to know?:

Environment:

• Clusternet version:

  • Clusternet-agent version (user clusternet-agent --version=json):
  • Clusternet-hub version (user clusternet-hub --version=json): "ghcr.io/clusternet/clusternet-hub:v0.2.0"

• Kubernetes version (use kubectl version):

☁  ~  kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T21:04:39Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-21T23:01:33Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

This is kind cluster

☁  ~  kind get clusters
cluster1
cluster2
hub

• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Clusternet version:
  • Clusternet-agent version (user clusternet-agent --version=json):
  • Clusternet-hub version (user clusternet-hub --version=json):
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

What would you like to be added:

Don't specify the parent cluster url when agent starts, Use a CRD like "Target"、“Parent” to Store the parent url adress.

Why is this needed:

So we can decouple the hard relationship, and give a change to program the structure of Cluster resources, program means we can init/edit as will.

What happened:

when we execute the step "PARENTURL=https://192.168.10.10 REGTOKEN=07401b.f395accd246ae52d envsubst < ./deploy/templates/clusternet_agent_secret.yaml | kubectl apply -f -", the cmd report that "'PARENTURL' is not recognized as an internal or external command" in the file folder "clusternet-main". And when we execute this command in the subfolder "deploy/agent", the cmd report thet "the system cannot find the path specified."

What you expected to happen:

we can create this secret successfully.

How to reproduce it (as minimally and precisely as possible):

do everything in "Get Started" step by step.

Anything else we need to know?:

Environment:

• Clusternet version: v.0.2.0
  • Clusternet-agent version (user clusternet-agent --version=json): v.0.2.0
  • Clusternet-hub version (user clusternet-hub --version=json): v.0.2.0
• Kubernetes version (use kubectl version): v.1.21.3
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

What happened:

when apply an exist deployment, return error

Error from server (NotFound): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"labels\":{\"clusternet-app\":\"multi-cluster-nginx\"},\"name\":\"my-nginx\",\"namespace\":\"foo\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"nginx\"}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"nginx\"}},\"spec\":{\"containers\":[{\"image\":\"nginx:1.14.2\",\"name\":\"nginx\",\"ports\":[{\"containerPort\":80}]}]}}}}\n"},"labels":{"aa":null}},"spec":{"template":{"spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx","ports":[{"containerPort":80}]}]}}}}
to:
Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
Name: "my-nginx", Namespace: "foo"
for: "nginx-deployment-foo.yaml": deployments.shadow "my-nginx" not found

kubectl clusternet apply -f deploy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
  namespace: foo
  labels:
    clusternet-app: multi-cluster-nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
            - containerPort: 80

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Clusternet version:
  • Clusternet-agent version (user clusternet-agent --version=json):
  • Clusternet-hub version (user clusternet-hub --version=json):

{
  "programName": "clusternet-hub",
  "major": "0",
  "minor": "5+",
  "gitVersion": "v0.5.0-11-g6112b08",
  "gitCommit": "6112b08eb21357e669f3f390b2eea56ab729d72e",
  "gitTreeState": "clean",
  "buildDate": "2021-11-03T08:05:21Z",
  "goVersion": "go1.14.15",
  "compiler": "gc",
  "platform": "linux/amd64"
}

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

What happened:

Distribute job application error:

The Job "dxl-job" is invalid:
* spec.template.metadata.labels[controller-uid]: Invalid value: map[string]string{"controller-uid":"d514b1e9-00f5-4e53-9419-a82f90648189", "job-name":"dxl-job", "k8s-app":"dxl-job", "qcloud-app":"dxl-job"}: must be '08e982e3-d34f-4e50-aab9-ce5c2ed37cd1'
* spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"controller-uid":"d514b1e9-00f5-4e53-9419-a82f90648189"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: `selector` not auto-generated

Create a job app, clusternet will render job as a template into the Description spec.raw, this template includes an auto-generated selector, apply this job template to the child cluster will fail with error selector not auto-generated

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Clusternet version:
  • Clusternet-agent version (user clusternet-agent --version=json):
  • Clusternet-hub version (user clusternet-hub --version=json):
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

What happened:

clusternet-agent error log:

W0901 02:45:02.131938       1 status_manager.go:124] cluster status is not ready, will retry later
W0901 02:45:02.649602       1 status_manager.go:124] cluster status is not ready, will retry later
W0901 02:45:05.366810       1 status_manager.go:124] cluster status is not ready, will retry later
I0901 02:45:10.403261       1 reflector.go:255] Listing and watching *v1.Node from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167
E0901 02:45:10.404607       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:clusternet-system:clusternet-agent" cannot list resource "nodes" in API group "" at the cluster scope
W0901 02:45:19.086087       1 status_manager.go:124] cluster status is not ready, will retry later

The clusternet-agent sa need clusterrole permission.

What would you like to be added:

Why is this needed:

What would you like to be added:

Add a new OverrideType ApplyNow.

Why is this needed:

When Localizations/Globalizations are updated, these changes should be applied immediately to objects in child clusters.

What would you like to be added:

Add a short-path URL handler for direct child cluster visiting

Why is this needed:

When visiting child cluster, the server url is needed, such as /apis/proxies.clusternet.io/v1alpha1/sockets/abc-def-ghi/http/<SERVER-URL>.

Actually a short-path url visiting, such as /apis/proxies.clusternet.io/v1alpha1/sockets/<CLUSTER-ID>/direct should be supported.

as title

What happened:

child register failed.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

agentlog:
I0909 02:47:53.941839 1 agent.go:215] try to bootstrap cluster registration if needed
I0909 02:47:53.942325 1 agent.go:407] generate a random string "clusternet-cluster-6dpgk" as cluster name for later use
E0909 02:47:53.958818 1 agent.go:193] failed to create ClusterRegistrationRequest: serializer for text/html; charset=utf-8 doesn't exist
W0909 02:47:53.958841 1 agent.go:194] something went wrong when using existing parent cluster credentials, switch to use bootstrap token instead

Environment:

• Clusternet version:
  • Clusternet-agent version (user clusternet-agent --version=json):v0.3.0
  • Clusternet-hub version (user clusternet-hub --version=json):v0.3.0
• Kubernetes version (use kubectl version):v1.18.2
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

What happened:

Create a test sub, and check the description created.

➜  ~ k get sub app-demo -n default -ojson | jq .spec.feeds
[
  {
    "apiVersion": "apps.clusternet.io/v1alpha1",
    "kind": "HelmChart",
    "name": "mysql",
    "namespace": "default"
  },
  {
    "apiVersion": "v1",
    "kind": "Namespace",
    "name": "foo"
  },
  {
    "apiVersion": "v1",
    "kind": "Service",
    "name": "my-nginx-svc",
    "namespace": "foo"
  },
  {
    "apiVersion": "apps/v1",
    "kind": "Deployment",
    "name": "my-nginx",
    "namespace": "foo"
  }
]
➜  ~ k get description -A
NAMESPACE          NAME               DEPLOYER   STATUS    AGE
clusternet-826s5   app-demo-generic   Generic    Success   77s
clusternet-826s5   app-demo-helm      Helm       Success   77s

Delete all feeds from sub

➜  ~ k edit sub app-demo -n default
subscription.apps.clusternet.io/app-demo edited
➜  ~ k get sub app-demo -n default -ojson | jq .spec.feeds
[]

Check the description, found the description still existing.

➜  ~ k get description -A
NAMESPACE          NAME               DEPLOYER   STATUS    AGE
clusternet-826s5   app-demo-generic   Generic    Success   107s
clusternet-826s5   app-demo-helm      Helm       Success   107s

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Clusternet version:
  • Clusternet-agent version (user clusternet-agent --version=json):
  • Clusternet-hub version (user clusternet-hub --version=json):
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

What would you like to be added:

When the application template is deleted, the application feeds should be removed from the subscription.

Example:
// Delete mysql chart
kubectl delete charts mysql -n default
// mysql feed should be delete

apiVersion: apps.clusternet.io/v1alpha1
kind: Subscription
....
  feeds:
  - apiVersion: apps.clusternet.io/v1alpha1
    kind: HelmChart
    name: mysql
    namespace: default

Why is this needed:

This issue was a part of previous #128

===================
I'm proposing a way :the agent will watch the desc and reconcile that deployment to original state (aka, rollback any change in child cluster and restore to hub cluster config)
hub cluster can override the difference which child cluster made. just like KubeFed.
and moreover , when I accidentally delete the deployment from child cluster , I'm looking forward clusternet can restore it back .

What happened:

➜  ~ kubectl clusternet get deploy -n foo
NAME       CREATED AT
my-nginx   2021-08-17T13:36:31Z
➜  ~ kubectl clusternet get deploy -A
No resources found

What you expected to happen:

➜  ~ kubectl clusternet get deploy -A
NAMESPACE            NAME                     CREATED AT
foo                  my-nginx                 2021-08-17T13:36:31Z

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

➜  ~ kubectl clusternet version
{
  "gitVersion": "0.2.0",
  "gitCommit": "13d72bbc7b2227a6e24319ca5f62ddbb39094ab9",
  "buildDate": "2021-08-11T12:07:34Z",
  "platform": "linux/amd64"
}

• Clusternet version:
  • Clusternet-agent version (user clusternet-agent --version=json):
  • Clusternet-hub version (user clusternet-hub --version=json):
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others: