I k8s setup with: flannel, weave and genie
using the following pod setup:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-weave-2
  annotations:
    cni: "weave,weave"
spec:
  containers:
  - name: nginx
    image: nginx:1.7.9
    ports:
    - containerPort: 80

I get an error message from genie:

Warning  FailedCreatePodSandBox  8m (x2717 over 2h)   kubelet, vm-18-80  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "22471ae954b9986f5a9b2eeee1fe3f52bf8cd4b40f286a3fd6e7d5f6cda0e80e" network for pod "nginx-weave-2": NetworkPlugin cni failed to set up pod "nginx-weave-2_default" network: CNI Genie Add IP internal error: CNI Genie error at addNetwork: could not create veth pair vethwepl22471ae-vethwepg22471ae: file exists, result: %!!(MISSING)s(<nil>)

Seems like veth pair with the same name is already created.
Note that creation of a pod with 2 flannel networks, or with weave and flannel works well on my setup.

Right now our code inside genie package has multi typos. Need to fix this, also make sure we maintain A+ go report standard. Right now, the grade is 'A' goal is to make it 'A+' all the time.

Unit & end to end tests for covering following scenarios:

1. Pod creation with different cni types weave, canal, nocni
2. Check for existing CNS running on the node

Below latest calico installation link need to be updated genie test script
https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/kubeadm/1.7/calico.yaml

In current implementation, In case of multiple Ip address for pod, All Ip addresses can be always shown in annotation only if user select multi ip preferences, not by default

so All Ip addresses can be always shown in annotation in case of multiple ips

I installed k8s with flannel , and then installed weave , but when I used weave as the annotation , encountered the following issue .

8月 23 15:11:47 hp-k8s-4 kubelet[17877]: CNI Genie cniName=weave
8月 23 15:11:47 hp-k8s-4 kubelet[17877]: CNI Genie files =[/etc/cni/net.d/00-genie.conf /etc/cni/net.d/10-flannel.conf /etc/cni/net.d/10-weave.conf]
8月 23 15:11:47 hp-k8s-4 kubelet[17877]: CNI Genie confFromFile ={ k8s-pod-network genie { [] []} 0 hp-k8s-4 kubernetes info {k8s eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZWZhdWx0LXRva2VuLTg0enRmIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRlZmF1bHQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxNmVjODAyYS04MjY4LTExZTctODNjOC0wMDUwNTZiY2JmZDkiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGVmYXVsdCJ9.pBOr1bisfN5nBp9vhMaZVmQfc8G52McK_LKDch64ylhKLJbQ0rVWDxmLUqwvQGKEqJh2ffp3RtpnwFn7to8bjHnsio-5E5L7xOkUDPG5gl6ThUwKXWDM28dCBdmm0Z4L3cNP-f9KYjK-mMT2lo0SIc61UexOiiCMYQc1kPnVGtw8g3iUaMxuVZSX8xzob5PQrIYkKQHlLmECV9bLmCCwbJ8gOi50dNFD3Soy3teWlhs7RI29KQbAEpYMQZfqs-BWe19xL__wytNcStGp7zWSs9ucEzFq9j_6ReIiqo2V9bv84mk39EJrlquuG8qeJGDwD1VwQxIxvyDZmkfeD760JA } {https://10.253.0.1:443 /etc/cni/net.d/genie-kubeconfig } {{{ {[]}}}} { { } { }} http://: romanaSegment}
8月 23 15:11:47 hp-k8s-4 kubelet[17877]: CNI Genie confFromFile ={ k8s-pod-network flannel { [] []} 0 { } { } {{{ {[]}}}} {bridge info { } { }} }
8月 23 15:11:47 hp-k8s-4 kubelet[17877]: CNI Genie confFromFile ={ weave weave-net { [] []} 0 { } { } {{{ {[]}}}} { { } { }} }
8月 23 15:11:47 hp-k8s-4 kubelet[17877]: CNI Genie cniName file found!!!!!! confFromFile.Type =weave-net
8月 23 15:11:47 hp-k8s-4 kubelet[17877]: E0823 07:11:47.934415 17989 docker_manager.go:372] NetworkPlugin cni failed on the status hook for pod 'centos2' - Unexpected command output Device "eth0" does not exist.
8月 23 15:11:47 hp-k8s-4 kubelet[17877]: with error: exit status 1

Do you have any suggestion ?

Hi, there is a Slack Link on the Mead GitHub ReadMe however when I click the link it only allows @Huawei email holders to join, how can non huawei email holders join?

After installed the k8s , the apiserver default listening port is on 6443 instead of 8080 , but when I run make test , I can see errors listed below . And I changed the 8080 to 6443 in cni-genie_k8s_test.go and run it again . But still have the same errors .

CNIGenie
/usr/lib/golang/src/github.com/Huawei-PaaS/CNI-Genie/cni-genie_k8s_test.go:492
Add nocni networking for Pod
/usr/lib/golang/src/github.com/Huawei-PaaS/CNI-Genie/cni-genie_k8s_test.go:472
using cni-genie for configuring nocni CNI
/usr/lib/golang/src/github.com/Huawei-PaaS/CNI-Genie/cni-genie_k8s_test.go:470
should succeed nocni networking for pod [It]
/usr/lib/golang/src/github.com/Huawei-PaaS/CNI-Genie/cni-genie_k8s_test.go:469

  Expected error:
      <*url.Error | 0xc8202ca420>: {
          Op: "Post",
          URL: "http://localhost:8080/api/v1/namespaces/test/pods",
          Err: {
              Op: "dial",
              Net: "tcp",
              Source: nil,
              Addr: {
                  IP: "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01",
                  Port: 8080,
                  Zone: "",
              },
              Err: {
                  Syscall: "getsockopt",
                  Err: 0x6f,
              },
          },
      }
      Post http://localhost:8080/api/v1/namespaces/test/pods: dial tcp [::1]:8080: getsockopt: connection refused
  not to have occurred

  /usr/lib/golang/src/github.com/Huawei-PaaS/CNI-Genie/cni-genie_k8s_test.go:448

I encountered another issue on flannel by leveraging Genie , after the pod running the default route is not existing in the pod , so as to the communication between pod is working but when I want to ping host's ip it is not working . the workaournd is to set the default route by command route add -net 0.0.0.0 ...... , also I checked the website containernetworking/cni#184 , it seems the default 10-flannel.conf should be changed , but it did not work for me .

from the website we can see , sounds like CNI has already added a new parameter "isDefaultGateway" to address this issue . But I tried it on my env , it is not working .
{
"name": "myflannel",
"type": "flannel",
"delegate": {
"bridge": "myflannelbr",
"isDefaultGateway": true
}
}
So I am not sure if Genie can support this config file ?

Genie is working well on my env with K8s 5.7 installed . And I enabled two network solution flannel and calico , both are working well so far . but when I used k8s network policy , I encountered some issues . the detailed steps are listed below .

1. kubectl run nginx --image=nginx --replicas=2 ( calico as the network provider)
   deployment "nginx" created

2. kubectl expose deployment nginx --port=80
   service "nginx" exposed

3. Now I can see the nginx can be accessed .
   $ kubectl run busybox --rm -ti --image=busybox /bin/sh
   Waiting for pod default/busybox-472357175-y0m47 to be running, status is Pending, pod ready: false
   Hit enter for command prompt
   / # wget --spider --timeout=1 nginx
   Connecting to nginx (10.100.0.16:80)
   / #

4. $ kubectl annotate ns default "net.beta.kubernetes.io/network-policy={"ingress": {"isolation": "DefaultDeny"}}"

5. and I run the " kubectl run busybox --rm -ti --image=busybox /bin/sh" again to try to connect to Nginx , but it still can be connected .

BTW , before I run these steps I have enabled - --runtime-config=extensions/v1beta1/networkpolicies=true on apiserver .

https://github.com/Huawei-PaaS/CNI-Genie/blob/master/docs/GettingStarted.md#test-process

I am trying to use a bridge plugin but we do see message Only supported are (Romana, weave, canal, calico, flannel). Any ideas how difficult it is to support additional plugins(https://github.com/containernetworking/plugins/tree/master/plugins/main).

After Genie picks up user's choice of cni plugin from Pod annotations, it should find the corresponding .conf file under /etc/cni/net.d/ for that plugin and parse it to build the cni command that is passed to the cni plugin agent

More specifically, in methods addNetwork() and deleteNetwork(), conf variable, that is passed to ExecAdd() and ExecDel(), should be populated based on the data parsed from the .conf file

This ensures that any new cni plugin that gets installed on the node in run-time can immediately be used by CNI-Genie

More specifically, we need to re-structure the existing documents so that in addition to multi-plugin & multi-NIC, we also cover multiple logical networks, multiple physical networks, network policy.

The specific tasks would be

• Update first page: what is genie, why we created genie, what are the major value propositions (use-cases), genie doesn't change k8s default behaviour, i.e., pods with no annotations (Issue #98 )
• Update getting started: how to add it to any existing cluster, i.e., deployment automation (single-click to download, build, run, helm charts, etc.)
• Include new demo videos

Under what circumstances should CNI-Genie be chosen?

Whenever Logical network delete is done, Entries in admission controller are not getting cleared properly. Due to this, recreation of logical network is giving failing

Right now, the new calico conf file is not parsed correctly.

I installed k8s 1.5.7 with Flannel installed ， and then installed weave on this Env as well . I followed the "GettingStarted.md" to install Genie . And then made the following changes on my env

1. I changed kubelet parameters with the following items added ( --network-plugin-dir=cni --network-plugin-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin) and then restart kubelet service .
2. I checked the following directory

[root@mscp3 net.d]# ls
00-genie.conf 10-calico.conf 10-weave.conf calico-kubeconfig calico-tls genie-kubeconfig
[root@mscp3 bin]# ls
bridge calico calico-ipam dhcp flannel genie host-local ipvlan loopback macvlan portmap ptp sample tuning vlan weave-ipam weave-net weave-plugin-2.0.1
[root@mscp3 bin]#

3. I just run the sample yaml file provided by Genie as following

apiVersion: v1
kind: Pod
metadata:
name: tomcat-weave1
labels:
app: web
annotations:
cni: "weave"
spec:
containers:
- name: key-value-store
image: tomcat:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080

4. After running kubectl create -f ..yaml , I got the ip from Flannel assigned . So it looks the Genie is not working well .

So based on my description , do you have any suggestion how to make it work , and how to debug the issues . thanks .

https://github.com/aws/amazon-vpc-cni-k8s

Even though I deploy a Pod with just a single network my pod should connect to, the following incorrect information is filled-in by Genie in my Pod annotation post deployment.

Describe Pod shows -

Annotations:    cni=calico
                multi-ip-preferences={"multi_entry":2,"ips":{"":{},"ip1":{"ip":"192.168.243.214","interface":"eth0"}}}

Notice the first empty entry in the 'ips' map and the 'multi_entry' value as '2'.

Hi all:
I am using CNI-Genie on arm64 server now(k8s 1.5.4), But there is an error occurs for which I can not resolve now. Could you provide some help for me?

the configuration is as follows:

1. /etc/cni/net.d/00-genie.conf
   {
   "name": "k8s-pod-network",
   "type": "genie",
   "etcd_endpoints": "http://192.168.100.177:2379",
   "log_level": "debug",
   "policy": {
   "type": "k8s",
   "k8s_api_root": "http://192.168.100.177:8080"
   },
   "kubernetes": {
   "kubeconfig": "/etc/cni/net.d/genie-kubeconfig"
   }
   }
2. /etc/cni/net.d/10-flannel.conf:

{
"name": "flannel",
"type": "flannel",
"subnetFile": "/run/flannel/subnet.env",
"dataDir": "/var/lib/cni/flannel",
"delegate": {
"bridge": "kbr0",
"isDefaultGateway": true
}
}
and the flannel network is ok by previous work;
3. the Pod to be deployed:
apiVersion: v1
kind: Pod
metadata:
name: genie-test-pod3
labels:
apps: busybox-genie-test
annotations:
cni: flannel
spec: # specification of the pod's contents
restartPolicy: Never
containers:

• name: genie-test-pod
  image: "arm64v8/busybox"
  stdin: true
  terminationMessagePath: /dev/termination-log
  tty: true

But the following error/events occured when create the pod:
#kubectl describe pod genie-test-pod

......................

Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason M

24m 24m 1 {default-scheduler } Normal Scheduled So fed25-k8sminion2
23m 7m 81 {kubelet fed25-k8sminion2} Warning FailedSync E "SetupNetwork" for "genie-test-pod_default" with SetupNetworkError: "Failed to setup network for pod "genie-test-pod_de4068b)" using network plugins "cni": CNI Genie Add IP internal error: CNI Genie error at addNetwork: no plugin name pr

24m 7m 82 {kubelet fed25-k8sminion2} Warning MissingClusterDNS kubelet does not have Clute Pod using "ClusterFirst" policy. Falling back to DNSDefault policy.
4m 4m 1 {kubelet fed25-k8sminion2} Warning FailedSync Error syncing pod, skippinie-test-pod_default" with SetupNetworkError: "Failed to setup network for pod "genie-test-pod_default(ec26bffc-831e-11eplugins "cni": CNI Genie Add IP internal error: CNI Genie error at addNetwork: no plugin name provided; Skipping pod"
, failed to "KillContainer" for "POD" with KillContainerError: "operation timeout: context deadline exceeded"

When DHCP ipam implementation could not acquire a lease, the following is shown as a result -

Aug 16 21:18:15 localhost kubelet[19237]: CNI Genie addNetwork err *** error calling DHCP.Allocate: no more tries result***
Aug 16 21:18:15 localhost kubelet[19237]: panic: runtime error: invalid memory address or nil pointer dereference
Aug 16 21:18:15 localhost kubelet[19237]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x7c3cb7]
Aug 16 21:18:15 localhost kubelet[19237]: goroutine 1 [running, locked to thread]:
Aug 16 21:18:15 localhost kubelet[19237]: github.com/Huawei-PaaS/CNI-Genie/vendor/github.com/containernetworking/cni/pkg/types/current.NewResultFromResult(0x0, 0x0, 0x3, 0xc42001e1e0, 0xc420028070)
Aug 16 21:18:15 localhost kubelet[19237]: #11/home/salmon/code/src/github.com/Huawei-PaaS/CNI-Genie/vendor/github.com/containernetworking/cni/pkg/types/current/types.go:124 +0x37
Aug 16 21:18:15 localhost kubelet[19237]: github.com/Huawei-PaaS/CNI-Genie/genie.updateRoutes(0x0, 0x0, 0xc42001e1e0, 0x59, 0x60, 0x59)
Aug 16 21:18:15 localhost kubelet[19237]: #11/home/salmon/code/src/github.com/Huawei-PaaS/CNI-Genie/genie/genie-controller.go:796 +0x4d
Aug 16 21:18:15 localhost kubelet[19237]: github.com/Huawei-PaaS/CNI-Genie/genie.mergeWithResult(0x0, 0x0, 0x0, 0x0, 0xc42028b448, 0x2, 0x2, 0x59)
Aug 16 21:18:15 localhost kubelet[19237]: #11/home/salmon/code/src/github.com/Huawei-PaaS/CNI-Genie/genie/genie-controller.go:742 +0x50
Aug 16 21:18:15 localhost kubelet[19237]: github.com/Huawei-PaaS/CNI-Genie/genie.AddPodNetwork(0xc420014010, 0x40, 0xc42001600a, 0x11, 0xc42001201b, 0x4, 0xc420018009, 0x9d, 0xc42001a009, 0x1b, ...)
Aug 16 21:18:15 localhost kubelet[19237]: #11/home/salmon/code/src/github.com/Huawei-PaaS/CNI-Genie/genie/genie-controller.go:134 +0x77b
Aug 16 21:18:15 localhost kubelet[19237]: main.cmdAdd(0xc420256540, 0xc4200c6e98, 0x5)
Aug 16 21:18:15 localhost kubelet[19237]: #11/home/salmon/code/src/github.com/Huawei-PaaS/CNI-Genie/cni-genie.go:42 +0x485
Aug 16 21:18:15 localhost kubelet[19237]: github.com/Huawei-PaaS/CNI-Genie/vendor/github.com/containernetworking/cni/pkg/skel.(*dispatcher).checkVersionAndCall(0xc4202589c0, 0xc420256540, 0x16c5f00, 0xc4200c3da0, 0x109cc20, 0x0, 0x410c88)
Aug 16 21:18:15 localhost kubelet[19237]: #11/home/salmon/code/src/github.com/Huawei-PaaS/CNI-Genie/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:162 +0x19f
Aug 16 21:18:15 localhost kubelet[19237]: github.com/Huawei-PaaS/CNI-Genie/vendor/github.com/containernetworking/cni/pkg/skel.(*dispatcher).pluginMain(0xc4202589c0, 0x109cc20, 0x109cc28, 0x16c5f00, 0xc4200c3da0, 0xf3d700)
Aug 16 21:18:15 localhost kubelet[19237]: #11/home/salmon/code/src/github.com/Huawei-PaaS/CNI-Genie/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:173 +0x2a2
Aug 16 21:18:15 localhost kubelet[19237]: github.com/Huawei-PaaS/CNI-Genie/vendor/github.com/containernetworking/cni/pkg/skel.PluginMainWithError(0x109cc20, 0x109cc28, 0x16c5f00, 0xc4200c3da0, 0x4046a4)
Aug 16 21:18:15 localhost kubelet[19237]: #11/home/salmon/code/src/github.com/Huawei-PaaS/CNI-Genie/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:210 +0xed
Aug 16 21:18:15 localhost kubelet[19237]: github.com/Huawei-PaaS/CNI-Genie/vendor/github.com/containernetworking/cni/pkg/skel.PluginMain(0x109cc20, 0x109cc28, 0x16c5f00, 0xc4200c3da0)
Aug 16 21:18:15 localhost kubelet[19237]: #11/home/salmon/code/src/github.com/Huawei-PaaS/CNI-Genie/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:222 +0x4d

actually only Calico, Flannel, Romana, Weave are supported

I use https://raw.githubusercontent.com/Huawei-PaaS/CNI-Genie/master/conf/1.8/genie.yaml
on k8s 1.8

  annotations:
    cni: ""

when cni is empty, pod can't run and has below error
if cni isn't empyt,it can work

Dec  2 16:51:38 k8s-215 kubelet: E1202 16:51:38.311955    2436 cni.go:301] Error adding network: netplugin failed but error parsing its diagnostic message "": unexpected end of JSON input
Dec  2 16:51:38 k8s-215 kubelet: CNI Genie annot= [map[cni:]]
Dec  2 16:51:38 k8s-215 kubelet: CNI Genie End result= %!s(<nil>)
Dec  2 16:51:38 k8s-215 kubelet: panic: runtime error: invalid memory address or nil pointer dereference
Dec  2 16:51:38 k8s-215 kubelet: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x53bc16]
Dec  2 16:51:38 k8s-215 kubelet: goroutine 1 [running, locked to thread]:

According to this demo https://asciinema.org/a/120340 cAdvisor is used when no plugin was selected:

cni: ""

but current behavior is different. When there are no annotations then default_plugin is used. When there are ANY annotations then OR when there is empty cni annotation then cAdvisor is used.

The end result is that default_plugin is only used when pod has no annotattions.

Is this an expected behavior or is it bug?

Right now, genie only supports k8s version 1.5

For the multiple IP scenario, calico and weave plugin is used in my k8s environment.
And when I set namespace ingress as defaultDeny, only the eth0 link can be denied.

So is it the normal action? And is there a way to set network policy to all IP link (eth0, eth1, ...) ?

Currently for supporting smart selection, genie makes use of cadvisor. But currently genie expects cadvisor to be always running at fixed address (http://127.0.0.1:4194). This can be made dynamic and user can specify path in config file

We are trying to connect some Kubernetes pods to multiple networks with CNI Genie with the following yaml as required:

annotation:
  cni: "network1, network2"

However, many of our pods do not need to connect to multiple networks and therefore do not provide the cni annotation. Is there a way to connect these pods to a default network (say network1) without having to specify a cni annotation?

If not, we request this feature as this is a valid usecase in several scenarios.

CNI Genie should also support parsing of those conf files whose names don't start with '*-'.

This can cause IP addresses to be replaced, instead of being appended, in case of multiple IPs

Whenever network level policy is added for selector subnet(x) with other peer subnet(y), pods beloning to x and y subenet are able to communicate but 2 pods belonging to same x subnet are not able to communicate.
Rule need to be added to allow communication of pods within the selector subnet

The end-to-end solution should look like this:

When pod requests for ips from multiple plugin, then genie should assign multiple veths to the container. However, genie throws error as "container veth name provided (eth0) already exists".