mirantis / cri-dockerd Goto Github PK
View Code? Open in Web Editor NEWdockerd as a compliant Container Runtime Interface for Kubernetes
License: Apache License 2.0
dockerd as a compliant Container Runtime Interface for Kubernetes
License: Apache License 2.0
The API has been upgraded to v1
, which breaks the containerd and docker runtimes.
$ ./crictl info
FATA[0000] getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService
crictl version 1.23.0-18-g0871ebbc
cri-dockerd 0.2.0 (a4d1895)
There is an issue to allow fallback to CRI v1alpha2
:
It is not a strict requirement to update crictl
to the same version of Kubernetes.
It works, when running the latest release (cri-tools v1.23.0
), but not master
.
$ ./crictl-1.23.0 info
{
"status": {
"conditions": [
{
"type": "RuntimeReady",
"status": true,
"reason": "",
"message": ""
},
{
"type": "NetworkReady",
"status": true,
"reason": "",
"message": ""
}
]
}
}
So it is possible to use cri-tools 1.19 or 1.21 or 1.23, even when using k8s 1.24.
$ ls -l cri-dockerd
-rwxr-xr-x 1 root root 52363914 Jun 8 23:11 cri-dockerd
$ ./cri-dockerd --version
cri-dockerd 0.2.1 (HEAD)
Because version.go
wasn't updated:
cri-dockerd/cmd/version/version.go
Lines 4 to 5 in d627d3e
There is an error when I use cri-dockerd as container-runtime.
Kubelet tries to reopen log file, but get the error from container runtime.
E0209 21:28:39.416074 1071258 remote_runtime.go:1134] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID=....
E0209 21:28:39.981058 1071258 container_log_manager.go:244] "Container log doesn't exist, reopen container log failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID=.....
It stuck when I try to run kubernetes metrics-server.
When I run the same helm chart with kubelet build in dockerhim it works fine.
kubectl version:
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", BuildDate:"2022-01-28T07:59:30Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
OS: 5.10.93-flatcar
Also, I find the same error as mine here rancher/rke#2716
Do you have any ideas how to fix this, thanks?
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-dockerd.service
sed: can't read /etc/systemd/system/cri-dockerd.service: No such file or directory
Should it be 'cri-docker.service' not 'cri-dockerd.service'?
The application binary, at least as present in the debian buster amd64 package, reports itself as the wrong version
It seems like currently the "noop" plugin is always used ?
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=
level=info msg="Docker cri networking managed by network plugin kubernetes.io/no-op"
In order to use CNI, one needs to supply additional params:
--network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-cache-dir=/var/lib/cni/cache --cni-conf-dir=/etc/cni/net.d
level=info msg="Docker cri networking managed by network plugin cni"
The previous default values seem to be missing, compared to dockershim ?
BEFORE (1.23)
--network-plugin string The name of the network plugin to be invoked for various events in kubelet/pod lifecycle. This docker-specific flag only works when container-runtime is set to docker. (DEPRECATED: will be removed along with dockershim.)
--network-plugin-mtu int32 The MTU to be passed to the network plugin, to override the default. Set to 0 to use the default 1460 MTU. This docker-specific flag only works when container-runtime is set to docker. (DEPRECATED: will be removed along with dockershim.)
--cni-bin-dir string A comma-separated list of full paths of directories in which to search for CNI plugin binaries. This docker-specific flag only works when container-runtime is set to docker. (default "/opt/cni/bin") (DEPRECATED: will be removed along with dockershim.)
--cni-cache-dir string The full path of the directory in which CNI should store cache files. This docker-specific flag only works when container-runtime is set to docker. (default "/var/lib/cni/cache") (DEPRECATED: will be removed along with dockershim.)
--cni-conf-dir string The full path of the directory in which to search for CNI config files. This docker-specific flag only works when container-runtime is set to docker. (default "/etc/cni/net.d") (DEPRECATED: will be removed along with dockershim.)
AFTER (1.24)
--network-plugin string <Warning: Alpha feature> The name of the network plugin to be invoked for various events in kubelet/pod lifecycle.
--network-plugin-mtu int32 <Warning: Alpha feature> The MTU to be passed to the network plugin, to override the default. Set to 0 to use the default 1460 MTU.
--cni-bin-dir string <Warning: Alpha feature> A comma-separated list of full paths of directories in which to search for CNI plugin binaries.
--cni-cache-dir string <Warning: Alpha feature> The full path of the directory in which CNI should store cache files.
--cni-conf-dir string <Warning: Alpha feature> The full path of the directory in which to search for CNI config files
I just installed Ubuntu 22.04 server on a new VM. The install process offered to "install" "docker", and I took the offer. Later I found out this install was done with snap
. It turns out that snap install docker
, at least on Ubuntu 22.04, does NOT create a usergroup named docker
. But the cri-docker.socket
systemd unit here assumes the existence of such a group.
Not sure, whether it is the right place to ask, moreover it's not an issue seeking information.
I am new to Kubernetes, just trying to install the cluster. ---
seeing an issue while running the commands to install in Linux.
exactly at cd cri-dockerd
I don't see a directory anywhere, how can I move forward?
Or should the issues be opened elsewhere instead ?
https://www.mirantis.com/blog/mirantis-to-take-over-support-of-kubernetes-dockershim-2/
Otherwise, if you’re using the open source Docker Engine, the dockershim project will be available as an open source component, and you will be able to continue to use it with Kubernetes; it will just require a small configuration change, which we will document.
ping @evol262
$ time docker stats --no-stream
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
61cad08c09c0 k8s_metrics-server_metrics-server-8595bd7d4c-5nvnl_kube-system_c4c13514-6af6-460f-b6d4-cadabb1ca23d_0 0.34% 14.67MiB / 5.805GiB 0.25% 1.44MB / 2.42MB 0B / 8.19kB 13
af9d6b482eda k8s_storage-provisioner_storage-provisioner_kube-system_91efee2f-9d1c-4962-8847-fc515d518896_0 0.27% 8.922MiB / 5.805GiB 0.15% 0B / 0B 0B / 0B 9
af46c0c8fa90 k8s_kube-proxy_kube-proxy-qwhqs_kube-system_bae1cac7-f02e-4836-ba60-efdf8ce1bc2b_0 0.08% 9.422MiB / 5.805GiB 0.16% 0B / 0B 0B / 16.4kB 9
f423e70923c8 k8s_POD_kube-proxy-qwhqs_kube-system_bae1cac7-f02e-4836-ba60-efdf8ce1bc2b_0 0.00% 308KiB / 5.805GiB 0.01% 0B / 0B 0B / 0B 1
4f2260ec6dec k8s_coredns_coredns-6d4b75cb6d-dhvgf_kube-system_508a1c0d-547c-45c9-8256-0c6a6bb0008f_0 0.26% 12.04MiB / 170MiB 7.08% 257kB / 248kB 0B / 0B 11
d595622f36bc k8s_POD_coredns-6d4b75cb6d-dhvgf_kube-system_508a1c0d-547c-45c9-8256-0c6a6bb0008f_0 0.00% 232KiB / 5.805GiB 0.00% 257kB / 248kB 0B / 0B 1
1c5f241c831e k8s_POD_storage-provisioner_kube-system_91efee2f-9d1c-4962-8847-fc515d518896_0 0.00% 172KiB / 5.805GiB 0.00% 0B / 0B 0B / 0B 1
e46a35791ca9 k8s_POD_metrics-server-8595bd7d4c-5nvnl_kube-system_c4c13514-6af6-460f-b6d4-cadabb1ca23d_0 0.00% 236KiB / 5.805GiB 0.00% 1.44MB / 2.42MB 0B / 0B 1
427cec7f3a62 k8s_kube-controller-manager_kube-controller-manager-minikube_kube-system_09d2e6db6299ac08c1c74b56109ba3d0_0 3.20% 39.98MiB / 5.805GiB 0.67% 0B / 0B 0B / 0B 16
08c6bf8af5bb k8s_kube-apiserver_kube-apiserver-minikube_kube-system_1b39c2135b30a71bf5fa75156498160b_0 5.78% 280MiB / 5.805GiB 4.71% 0B / 0B 0B / 0B 18
f8902332b034 k8s_etcd_etcd-minikube_kube-system_906edd533192a4db2396a938662a5271_0 1.75% 31.13MiB / 5.805GiB 0.52% 0B / 0B 0B / 54.3MB 12
0afbee693806 k8s_kube-scheduler_kube-scheduler-minikube_kube-system_e731e44e54402bb1350402612c5f28bb_0 0.80% 14.12MiB / 5.805GiB 0.24% 0B / 0B 0B / 0B 11
2fcee2fd796c k8s_POD_kube-controller-manager-minikube_kube-system_09d2e6db6299ac08c1c74b56109ba3d0_0 0.00% 180KiB / 5.805GiB 0.00% 0B / 0B 0B / 0B 1
f032ed29ad23 k8s_POD_etcd-minikube_kube-system_906edd533192a4db2396a938662a5271_0 0.00% 248KiB / 5.805GiB 0.00% 0B / 0B 73.7kB / 0B 1
4ef53131c420 k8s_POD_kube-apiserver-minikube_kube-system_1b39c2135b30a71bf5fa75156498160b_0 0.00% 176KiB / 5.805GiB 0.00% 0B / 0B 0B / 0B 1
5a53b2ae1ae8 k8s_POD_kube-scheduler-minikube_kube-system_e731e44e54402bb1350402612c5f28bb_0 0.00% 176KiB / 5.805GiB 0.00% 0B / 0B 0B / 0B 1
real 0m2.631s
user 0m0.037s
sys 0m0.028s
$ crictl stats
CONTAINER CPU % MEM DISK INODES
08c6bf8af5bbc 7.02 0B 0B 0
0afbee6938067 0.51 0B 0B 0
427cec7f3a622 3.28 0B 0B 0
4f2260ec6decd 0.29 0B 0B 0
61cad08c09c0f 0.43 0B 0B 0
af46c0c8fa906 0.07 0B 2.294kB 0
af9d6b482eda5 0.26 0B 0B 0
f8902332b0346 2.45 0B 0B 0
real 0m32.294s
user 0m0.017s
sys 0m0.025s
There is no memory being reported back, but docker is outputting memory fine. Also, the command takes 32 seconds vs the 2.6 seconds with docker.
Need the new "cri-dockerd" name, too. So that it matches the README
systemctl enable --now cri-dockerd.socket
Since:
Alternatively, the README could be updated to reflect the systemd units ?
That is: keep the "cri-docker" name, and make sure everyone uses that.
systemctl enable --now cri-docker.socket
I am running cri-dockerd on the socket below. I am trying to pull a large image onto the machine with crictl and it always times out after 2 mins . Is there a tunable that i can change in cri-dockerd to fix the current issue ?
I am running cri-dockerd-v0.2.0 with the following options.
cri-dockerd-v0.2.0 --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --container-runtime-endpoint=unix:///var/run/dockershim.sock --network-plugin=cni --image-pull-progress-deadline=5m0s
[root@qct32 ~]# time /usr/bin/crictl -i unix:///var/run/dockershim.sock -r unix:///var/run/dockershim.sock pull docker.io/slightlytyler/large-image-test
FATA[0119] pulling image: rpc error: code = Unknown desc = context deadline exceeded
real 1m59.087s
user 0m0.034s
sys 0m0.031s
I am getting errors trying to build cri-dockerd in support of a new Kubernetes installation, Kubernetes version 1.24. The error messages are
go: downloading google.golang.org/appengine v1.6.5
malformed import path "github.com/Mirantis/cri-dockerd/version.Version=0.2.2": invalid char '='
malformed import path "-X": leading dash
malformed import path "github.com/Mirantis/cri-dockerd/version.PreRelease=": invalid char '='
malformed import path "github.com/Mirantis/cri-dockerd/version.BuildTime=": invalid char '='
malformed import path "github.com/Mirantis/cri-dockerd/version.GitCommit=b872f96": invalid char '='
malformed import path "-o": leading dash
package cri-dockerd is not in GOROOT (/home/tjcw/.go/src/cri-dockerd)
and the command I was trying to run was
$ go get && go build ${CRI_DOCKERD_LDFLAGS} -o cri-dockerd
I am using go version 1.18.3 on a linux/amd64 system (Ubuntu 20.04).
Can anyone help me with what the problem is ?
I set up cri-dockerd on Ubuntu 20.04 alongside latest stable Docker and minikube (using driver 'none'), to enable minikube to work now it uses Kubernetes 1.24.
It does work, but it's furiously logging the following "error" several times per second (with varying hexadecimal values):
Jun 26 07:07:23 a cri-dockerd[941]: time="2022-06-26T07:07:23Z" level=error msg="ContainerStats resp: {0x40008700c0 linux}"
while run k8s in dualstak, only one ip returned.
I see the codes in network/plugins.go using k8s.io/apiserver/pkg/util/feature.DefaultFeatureGate to check is ipv6DualStack enabled, but no args exports, how to enable this featuregate? should upgrade k8s after 1.23
utilfeature.DefaultFeatureGate.Enabled(kubefeatures.IPv6DualStack)
We pursued an implementation of cri-dockerd in kubespray kubernetes-sigs/kubespray#8623 , an ansible based kubernetes deployment tool.
In the linked PR we discovered that the cri-dockerd does not use the configured CNI resulting in incorrect pod addressing.
Failed CI runs:
The configured pod CIDR for the CI is 10.233.64.0/18 but the pods get IP addresses like 172.17.0.6 and 172.17.0.7 which are allocated from the default docker network.
I tried forcing the use of calico with setting the cri-dockerd command line like this:
ExecStart=/usr/local/bin/cri-dockerd --container-runtime-endpoint fd:// --cni-bin-dir=/opt/cni/bin --cni-cache-dir=/var/lib/cni --cni-conf-dir=/etc/cni/net.d --network-plugin=calico --pod-cidr=10.233.64.0/18
But I get the following error when starting cri-dockerd:
Mar 15 09:57:08 localhost cri-dockerd[48989]: time="2022-03-15T09:57:08Z" level=info msg="Using CNI configuration file /etc/cni/net.d/10-calico.conflist"
Mar 15 09:57:08 localhost cri-dockerd[48989]: time="2022-03-15T09:57:08Z" level=fatal msg="didn't find compatible CNI plugin with given settings &{HairpinMode:none NonMasqueradeCIDR:10.0.0.0/8 PluginName:calico PluginBinDirString:/opt/cni/bin PluginBinDirs:[/opt/cni/bin] PluginConfDir:/etc/cni/net.d PluginCacheDir:/var/lib/cni MTU:0}: network plugin \"calico\" not found"
Mar 15 09:57:08 localhost systemd[1]: cri-dockerd.service: Main process exited, code=exited, status=1/FAILURE
Mar 15 09:57:08 localhost systemd[1]: cri-dockerd.service: Failed with result 'exit-code'.
Mar 15 09:57:08 localhost systemd[1]: Failed to start CRI Interface for Docker Application Container Engine.
Mar 15 09:57:10 localhost systemd[1]: cri-dockerd.service: Scheduled restart job, restart counter is at 3.
Mar 15 09:57:10 localhost systemd[1]: Stopped CRI Interface for Docker Application Container Engine.
Mar 15 09:57:10 localhost systemd[1]: cri-dockerd.service: Start request repeated too quickly.
Mar 15 09:57:10 localhost systemd[1]: cri-dockerd.service: Failed with result 'exit-code'.
Mar 15 09:57:10 localhost systemd[1]: Failed to start CRI Interface for Docker Application Container Engine.
The test CNI configuration:
(venv) root@instance-1:~/kubespray# ls -l /opt/cni/bin/
total 163508
-rwxr-xr-x 1 root root 3990548 Mar 15 09:42 bandwidth
-rwsr-xr-x 1 root root 47026188 Mar 15 09:42 calico
-rwsr-xr-x 1 root root 47026188 Mar 15 09:42 calico-ipam
-rwxr-xr-x 1 root root 3357992 Mar 15 09:42 flannel
-rwxr-xr-x 1 root root 3402808 Mar 15 09:42 host-local
-rwsr-xr-x 1 root root 47026188 Mar 15 09:42 install
-rwxr-xr-x 1 root root 3472123 Mar 15 09:42 loopback
-rwxr-xr-x 1 root root 3924908 Mar 15 09:42 portmap
-rw-r--r-- 1 root root 4555575 Mar 15 09:42 tags.txt
-rwxr-xr-x 1 root root 3622648 Mar 15 09:42 tuning
(venv) root@instance-1:~/kubespray# ls -l /etc/cni/net.d/
total 12
-rw-r--r-- 1 root root 709 Mar 15 09:42 10-calico.conflist
-rw-r--r-- 1 root root 715 Mar 15 09:42 calico.conflist.template
-rw------- 1 root root 2824 Mar 15 09:42 calico-kubeconfig
(venv) root@instance-1:~/kubespray# cat /etc/cni/net.d/10-calico.conflist
{
"name": "cni0",
"cniVersion":"0.3.1",
"plugins":[
{
"datastore_type": "kubernetes",
"nodename": "localhost",
"type": "calico",
"log_level": "info",
"log_file_path": "/var/log/calico/cni/cni.log",
"ipam": {
"type": "calico-ipam",
"assign_ipv4": "true",
"ipv4_pools": ["10.233.64.0/18"]
},
"policy": {
"type": "k8s"
},
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
}
},
{
"type":"portmap",
"capabilities": {
"portMappings": true
}
},
{
"type":"bandwidth",
"capabilities": {
"bandwidth": true
}
}
]
}
Hello,
I have upgraded kubernetes from 1.23 to 1.24 and added cri-dockerd between docker and kubernetes. Also upgraded cni network plugins to 1.1.1. I was also using flannel 16.3, this one I did not upgrade.
When i try to delete a pod that i already had i get:
Normal Killing 14m kubelet Stopping container metrics-server
Warning FailedKillPod 4m22s (x50 over 14m) kubelet error killing pod: failed to "KillPodSandbox" for "fc41f3c5-292a-4e31-99c1-db87fc6f4fd5" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"metrics-server-77dc799f5f-rh8dr_kube-system\" network: could not retrieve port mappings: key is not found"
[cloud-user@razvan-ssd bin]$ sudo systemctl status cri-docker
● cri-docker.service - CRI Interface for Docker Application Container Engine
Loaded: loaded (/etc/systemd/system/cri-docker.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2022-05-11 14:49:13 UTC; 11min ago
Docs: https://docs.mirantis.com
Main PID: 1773162 (cri-dockerd)
Tasks: 10
Memory: 21.7M
CGroup: /system.slice/cri-docker.service
└─1773162 /usr/bin/cri-dockerd --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-cache-dir=/var/lib/cni/cache --cni-conf-dir=/etc/cni/net.d --pod-infra-container-image=k8s.gcr.io/pause:3.7
May 11 15:00:14 razvan-ssd.novalocal cri-dockerd[1773162]: time="2022-05-11T15:00:14Z" level=error msg="CNI failed to delete loopback network: could not retrieve port mappings: key is not found"
May 11 15:00:14 razvan-ssd.novalocal cri-dockerd[1773162]: time="2022-05-11T15:00:14Z" level=error msg="Error deleting network when building cni runtime conf: could not retrieve port mappings: key is not found"
Please advise.
Would it be possible to get a new release of cri-dockerd ?
The current 0.2.0 release still has the old socket location
Missing 50c048c
Also missing the .deb and .rpm files, mentioned in https://www.mirantis.com/blog/the-future-of-dockershim-is-cri-dockerd/
As a workaround, we are shipping v0.2.0-11-ga4d1895
(building from source, rather than using vendor binaries)
cri-dockerd 0.2.0 (a4d1895)
Hi!
I'm working on the KEP that will be implemented in 1.25 (next k8s release) to support user namespaces. We are creating an implementation for containerd and CRIO, but it will be nice if dockershim implemented that too.
I think there are some limitations docker needs to fix as a pre-requisite for the implementation. IIUC docker only supports a single ID mappings shared by all containers running in the host. There is not support for multiple ID mappings yet. However, for isolation reasons, we are using a different ID mappings for each pod in Kubernetes, which doesn't overlap with mappings of other pods either. So, we will need to use multiple ID mappings for containers, not just a single mapping shared by all containers as docker currently supports.
Some very old comments on the linked moby issue mention that this limitation might be simpler to solve once containerd 1.0 is used, which is already the case. Do you know if this limitation is indeed "easy" to fix now?
It would be great if you can implement userns support for Kubernetes pods in dockershim :)
rpm -ivh cri-dockerd-0.2.1.20220525024524.cb92d7b-0.el8.x86_64.rpm
error: Failed dependencies:
or is needed by cri-dockerd-3:0.2.1.20220525024524.cb92d7b-0.el8.x86_64
rpm -qpR cri-dockerd-0.2.1.20220525024524.cb92d7b-0.el8.x86_64.rpm
/bin/sh
/bin/sh
/bin/sh
container-selinux >= 2:2.74
containerd.io >= 1.2.2-3
device-mapper-libs >= 1.02.90-1
iptables
libc.so.6()(64bit)
libc.so.6(GLIBC_2.2.5)(64bit)
libcgroup
libpthread.so.0()(64bit)
libpthread.so.0(GLIBC_2.2.5)(64bit)
libpthread.so.0(GLIBC_2.3.2)(64bit)
libseccomp >= 2.3
nftables
or
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(PayloadIsXz) <= 5.2-1
systemd
tar
xz
unable to install rpm on centos-8
hello, in minikube we support docker, containerd and cri-o runtimes. we like to keep supporting docker runtime. I would like to know if there has already been work started for a separate dockershim that we could use in minikube?
Sorry to bother, I have trouble installing.
When I follow README and execute export CRI_DOCKERD_LDFLAGS=-ldflags "-X github.com/Mirantis/cri-dockerd/version.Version=${VERSION} -X github.com/Mirantis/cri-dockerd/version.PreRelease=${PRERELEASE} -X github.com/Mirantis/cri-dockerd/version.BuildTime=${BUILD_DATE} -X github.com/Mirantis/cri-dockerd/version.GitCommit=${REVISION}"
, the output is
-bash: export: `-X github.com/Mirantis/cri-dockerd/version.Version=0.2.0 -X github.com/Mirantis/cri-dockerd/version.PreRelease= -X github.com/Mirantis/cri-dockerd/version.BuildTime= -X github.com/Mirantis/cri-dockerd/version.GitCommit=13b3b70': not a valid identifier
I am using go with version 1.16.10. I would appreciate any help. Thanks!
hello,
i see different values for the socket path in the packaged systemd socket file:
https://github.com/Mirantis/cri-dockerd/blob/master/packaging/systemd/cri-docker.socket#L6
(has ListenStream=%t/cri-docker.sock
)
and the default value in:
cri-dockerd/src/cmd/cri/options/options.go
Lines 44 to 47 in 542e27d
the flag has .../cri-dockerd.sock
(notice the trailing d
). Windows packaging seems to not overload the default npipe:////./pipe/cri-dockerd
we have a pending kubeadm PR for k8s 1.24 where we are hardcoding the cri-dockerd "known" endpoints to the same values as in the cri-dockerd options.go above for Linux/Windows:
kubernetes/kubernetes#107317
i'm assuming that the missing d
is a typo?
are we safe to use npipe:////./pipe/cri-dockerd
for Windows and unix:///var/run/cri-dockerd.sock
for Linux in kubeadm defaults?
There are some typos, both cosmetical:
Docs: https://docs.micrantis.com
And more severe ones, failing the start:
unknown flag: --networkplugin
Metrics data being pulled through cri-dockerd is much slower than querying docker directly.
Details here: rancher/rke#2716
GOARCH=amd64
packaging/static/Makefile-.PHONY: static-linux
packaging/static/Makefile-static-linux:
packaging/static/Makefile- mkdir -p build/linux/cri-dockerd
packaging/static/Makefile: cd $(APP_DIR) && go get && env CGO_ENABLED=$(CGO_ENABLED) GOOS=linux GOARCH=amd64 go build ${CRI_DOCKERD_LDFLAGS} -o cri-dockerd
packaging/static/Makefile- mv $(APP_DIR)/cri-dockerd build/linux/cri-dockerd/cri-dockerd
packaging/static/Makefile- tar -C build/linux -c -z -f build/linux/cri-dockerd-$(VERSION).tgz cri-dockerd
packaging/static/Makefile-
Which means that it won't build for arm64, for instance.
ERROR: architecture for "/usr/bin/cri-dockerd" is "Advanced Micro Devices X86-64", should be "AArch64"
There is also a hardcoded go version, but it doesn't seem to have any affect (it uses go
, from the PATH)
packaging/common.mk:GO_VERSION:=1.16.8
packaging/static/Makefile:GO_VERSION=$(shell grep "ARG GO_VERSION" $(APP_DIR)/dockerfiles/Dockerfile.dev | awk -F'=' '{print $$2}')
So it still uses the correct version, currently 1.18.1
(using the same go compiler as for the Kubernetes)
The debian packages, as-is, are not installable in a default debian or ubuntu environment: they depend on containerd.io
instead of containerd
. I'm not sure if the former is a package name provided by Docker's upstream packaging, to go with docker-ce
, but the docker.io
& related packages that ship with Debian & Ubuntu use the containerd
package name.
I think this could be fixed simply by changing this line:
cri-dockerd/packaging/deb/common/control
Line 29 in 60a25a1
To use Depends: containerd.io (>= 1.2.2-3) | containerd(>= 1.2.2-3),
so that either one will satisfy the requirements
Otherwise it's possible to get issues like this..
cri-dockerd --help
cri-dockerd: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by cri-dockerd)
Enabled like this..
--- a/packaging/static/Makefile
+++ b/packaging/static/Makefile
@@ -23,7 +23,7 @@ static: static-linux cross-mac cross-win cross-arm ## create all static packages
.PHONY: static-linux
static-linux:
mkdir -p build/linux/cri-dockerd
- cd $(APP_DIR) && go get && env GOOS=linux GOARCH=amd64 go build -o cri-dockerd
+ cd $(APP_DIR) && go get && env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o cri-dockerd
mv $(APP_DIR)/cri-dockerd build/linux/cri-dockerd/cri-dockerd
tar -C build/linux -c -z -f build/linux/cri-dockerd-$(STATIC_VERSION).tgz cri-dockerd
@@ -35,14 +35,14 @@ hash_files:
.PHONY: cross-mac
cross-mac:
mkdir -p build/mac/cri-dockerd
- cd $(APP_DIR) && go get && env GOOS=darwin GOARCH=amd64 go build -o cri-dockerd-darwin-amd64
+ cd $(APP_DIR) && go get && env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o cri-dockerd-darwin-amd64
mv $(APP_DIR)/cri-dockerd-darwin-amd64 build/mac/cri-dockerd/cri-dockerd
tar -C build/mac -c -z -f build/mac/cri-dockerd-$(STATIC_VERSION).tgz cri-dockerd
.PHONY: cross-win
cross-win:
mkdir -p build/win/cri-dockerd
- cd $(APP_DIR) && go get && env GOOS=windows GOARCH=amd64 go build -o cri-dockerd-windows-amd64
+ cd $(APP_DIR) && go get && env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o cri-dockerd-windows-amd64
mv $(APP_DIR)/cri-dockerd-windows-amd64 build/win/cri-dockerd/cri-dockerd.exe
if ! grep -sq 'docker\|lxc' /proc/1/cgroup; then \
docker run --rm -v $(CURDIR)/build/win:/v -w /v alpine sh -c 'apk update && apk add zip && zip -r cri-dockerd-$(STATIC_VERSION).zip cri-dockerd'; \
@@ -52,6 +52,6 @@ cross-win:
.PHONY: cross-arm
cross-arm: ## create tgz with linux armhf client only
mkdir -p build/arm/cri-dockerd
- cd $(APP_DIR) && go get && env GOOS=linux GOARCH=arm64 go build -o cri-dockerd-arm64
+ cd $(APP_DIR) && go get && env CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o cri-dockerd-arm64
mv $(APP_DIR)/cri-dockerd-arm64 build/arm/cri-dockerd/cri-dockerd
tar -C build/arm -c -z -f build/arm/cri-dockerd-$(STATIC_VERSION).tgz cri-dockerd
root@Ria-PC:/home/ria# rm -r bin/
root@Ria-PC:/home/ria# mkdir bin
root@Ria-PC:/home/ria# VERSION=$((git describe --abbrev=0 --tags | sed -e 's/v//') || echo
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
root@Ria-PC:/home/ria# go get && go build -ldflags="-X github.com/Mirantis/cri-dockerd/version.Version='$VERSION}' -X github.com/Mirantis/cri-dockerd/version.PreRelease='$PRERELEASE' -X github.com/Mirantis/cri-dockerd/version.BuildTime='$BUILD_DATE' -X github.com/Mirantis/cri-dockerd/version.GitCommit='$REVISION'" -o cri-dockerd
can't load package: package .: no Go files in /home/ria
root@Ria-PC:/home/ria#
Unable to get logs after using cri-dockerd with Kubelet, everything else looks good.
crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock logs f96a85c2686f4
FATA[0000] failed to try resolving symlinks in path "/var/log/pods/kube-system_kube-apiserver-zsm-test-0_cb68ef30a75a5673ba80d69dedf5b217/kube-apiserver/2.log": lstat /var/log/pods/kube-system_kube-apiserver-zsm-test-0_cb68ef30a75a5673ba80d69dedf5b217/kube-apiserver/2.log: no such file or directory
kubectl logs -f -n kube-system kube-apiserver-zsm-test-0
failed to try resolving symlinks in path "/var/log/pods/kube-system_kube-apiserver-zsm-test-0_cb68ef30a75a5673ba80d69dedf5b217/kube-apiserver/2.log": lstat /var/log/pods/kube-system_kube-apiserver-zsm-test-0_cb68ef30a75a5673ba80d69dedf5b217/kube-apiserver/2.log: no such file or directory
mkdir -p /usr/local/bin
install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd
cp -a packaging/systemd/* /etc/systemd/system
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-dockerd.service
sed: can't read /etc/systemd/system/cri-dockerd.service: No such file or directory
systemctl daemon-reload
systemctl enable cri-dockerd.service
systemctl enable --now cri-dockerd.socket
Failed to enable unit: Unit file cri-dockerd.socket does not exist.
So looks like releases would need some kind of regression testing ? (Or manual)
When doing last-minute changes, such as 50c048c
How come there is no 20.04 LTS? 22.04 is still not available in Azure so its a bit early to close up shop?
.deb packages can be built from this directory with the following syntax
make deb
Artifacts will be located in debbuild under the following directory structure: debbuild/$distro-$distro_version/
Specifying a specific distro
make ubuntu
I'm trying to create cri-dockerd deb package as mentioned in the readme but getting the below error.
$make deb
Output truncated
...
dpkg-buildpackage: warning: debian/changelog(l1): version '5:not-0~ubuntu-bionic' is invalid: version number does not start with digit
LINE: cri-docker (5:not-0~ubuntu-bionic) bionic; urgency=low
dpkg-buildpackage: error: version number does not start with digit
dpkg-buildpackage: info: source package cri-docker
dpkg-buildpackage: info: source version unknown
Makefile:70: recipe for target 'ubuntu-bionic' failed
make: *** [ubuntu-bionic] Error 255
OS details:
$pwd
cri-dockerd/packaging/deb
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic
Now that Kubernetes 1.23 is in alpha, it would be nice with a 0.1.0
release and some binaries... (like static and deb/rpm)
https://www.mirantis.com/blog/the-future-of-dockershim-is-cri-dockerd/
There is a Makefile with a few targets.
make deb
ormake rpm
will probably have you covered, and you can install the packages as normal. If you’re using a different distribution,make static
will give you raw binaries you can invoke.
I expect to be able to download these, from the GibHub release "assets"
I am trying the steps in build and install on Ubuntu 18.04 and am running into some issues ...
mkdir bin
Is this command supposed to be run inside a clone of the project?
export CRI_DOCKERD_LDFLAGS=-ldflags "-X github.com/Mirantis/cri-dockerd/version.Version=${VERSION} -X github.com/Mirantis/cri-dockerd/version.PreRelease=${PRERELEASE} -X github.com/Mirantis/cri-dockerd/version.BuildTime=${BUILD_DATE} -X github.com/Mirantis/cri-dockerd/version.GitCommit=${REVISION}"
This command seems to be failing when run in bash
. Does it need some other setting?
export CRI_DOCKERD_LDFLAGS=-ldflags "-X github.com/Mirantis/cri-dockerd/version.Version=${VERSION} -X github.com/Mirantis/cri-dockerd/version.PreRelease=${PRERELEASE} -X github.com/Mirantis/cri-dockerd/version.BuildTime=${BUILD_DATE} -X github.com/Mirantis/cri-dockerd/version.GitCommit=${REVISION}"
bash: export: `-X github.com/Mirantis/cri-dockerd/version.Version=0.2.1 -X github.com/Mirantis/cri-dockerd/version.PreRelease= -X github.com/Mirantis/cri-dockerd/version.BuildTime= -X github.com/Mirantis/cri-dockerd/version.GitCommit=60a25a1': not a valid identifier
go get && go build ${CRI_DOCKERD_LDFLAGS} -o ../bin/cri-dockerd
Based on the -o
flag, the output is supposed to be places in ../bin/cri-dockerd
. Is this the same directory where the project was cloned?
after kubeadm init:
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 5.501531 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-check] Initial timeout of 40s passed.
error execution phase upload-config/kubelet: Error writing Crisocket information for the control-plane node: nodes "xxxx" not found
KEP 2040: Kubelet CRI support
https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2040-kubelet-cri#pinned-images
Requires CRI API 1.23
How do I install this thing using go install
, which is apparently the new "right way"? Thanks! I'll gladly submit a README patch once I get it working.
Wanted to leverage cri-dockerd
as CRI for k8s.
We tested the shim code and noticed kubectl logs doesnt work anymore when docker was configured with journald
as logging driver.
Is there an option I need to enable to make cri-dockerd as proxy and leverage the functionality of docker as is??
Error logs:
-sh-4.2#
kubectl logs test-9ng99 -n test -c test -f
failed to try resolving symlinks in path "/var/log/pods/test_test-9ng99_e5c59673-3af9-4b01-a764-4ebb6adc4313/test/5.log": lstat /var/log/pods/test_test-9ng99_e5c59673-3af9-4b01-a764-4ebb6adc4313/test/5.log: no such file or directory
-sh-4.2#
docker info
-sh-4.2# docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
scan: Docker Scan (Docker Inc., v0.12.0)
Server:
Containers: 139
Running: 59
Paused: 0
Stopped: 80
Images: 734
Server Version: 20.10.9
Storage Driver: btrfs
Build Version: Btrfs v4.9.1
Library Version: 102
Logging Driver: journald
Cgroup Driver: systemd
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
Default Runtime: runc
Init Binary: docker-init
containerd version: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
runc version: v1.1.1-0-g52de29d
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-1160.62.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 48
Total Memory: 125.6GiB
Name: test.local.net
ID: 6IQV:5TXZ:ATQ5:FQK7:HSDQ:OULR:BAAN:CRME:K2GO:GXAB:U4OL:IRT7
Docker Root Dir: /cowdata/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
test.loca.net:5000
127.0.0.0/8
Live Restore Enabled: false
It seems like the socket/service pair, is still starting /var/run/dockershim.sock
?
When adding the missing fd://
for socket-activation, this feature looks missing ?
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=""
failed to listen on "fd://": protocol "fd" not supported
As a workaround, one can ignore systemd and start unix:///var/run/cri-docker.sock
$ sudo cri-dockerd --container-runtime-endpoint unix:///var/run/cri-docker.sock
After the introduction of "src", it is harder to work with the repo with standard go tools.
They sort of assume that go.mod
and the resulting vendor
is at the top of the repo...
Could this be reverted ?
commit 278a1fc
$ cri-dockerd --version
Kubernetes v0.0.0-master+$Format:%h$
It was supposed to show "cri-dockerd", not "Kubernetes".
And whatever the git command was it tried to use, it failed.
(There is a known bug with magic git files in the base version*)
* But cri-dockerd is supposed to output its own version, anyway.
VERSION=0.1.0-dev
make static
probably a copy/paste error, from the docker unit ?
BindsTo=containerd.service
it only needs to wait for and talk to the docker.service
since "containerd.service" doesn't even have to exist...
Hi,
I have deployed k8s 1.24.1 and used docker-cri for the first time here as dockershim is removed.
The nodes went to ready state after cluster initialization without deploying any cni, earlier with dockershim till 1.23.x I used to deploy calico then only nodes go in ready state.
I noticed that docker0 is used as network for Pod's, all Pod's take an IP from docker0. does cri comes with an inbuilt cni ??
also I found that 2 pod's are taking same IP, let's suppose 2 pod's are running on diff worker nodes, they will take IP from respective node docker0 network, hence same IP's.
Also in such a case how can pod's talk that are running on different worker node ? as each pod takes IP from respective docker0 network
Can we by any chance switch to any other cni like calico or flannel or something else ??
In other projects
import "github.com/Mirantis/cri-dockerd/cmd"
github.com/Mirantis/cri-dockerd/cmd: module github.com/Mirantis/cri-dockerd@latest found (v0.2.0), but does not contain package github.com/Mirantis/cri-dockerd/cmd
import "github.com/Mirantis/cri-dockerd/src/cmd"
github.com/Mirantis/cri-dockerd/src/cmd: github.com/Mirantis/cri-dockerd/[email protected]: parsing go.mod:
module declares its path as: github.com/Mirantis/cri-dockerd
but was required as: github.com/Mirantis/cri-dockerd/src
// go.mod
replace github.com/Mirantis/cri-dockerd => github.com/Mirantis/cri-dockerd/src v0.2.0
github.com/Mirantis/[email protected]: reading github.com/Mirantis/cri-dockerd/src/src/go.mod at revision src/v0.2.0: unknown revision src/v0.2.0
Since it is socket-activated, it will be started on-demand (when requested).
systemctl daemon-reload
systemctl enable cri-dockerd.service
systemctl enable --now cri-dockerd.socket
Currently there is some developer documentation on GitHub, and MCR documentation on docs.mirantis.com
But there is no information for Docker Engine users, how install a cri-dockerd
package and configure crictl.
One could use GitHub Pages for this home page, perhaps ?
Ideally it should have a vanity domain like cri-dockerd.io
Like:
When not using git
to build, but a release tarball
From #19 (comment)
Optionally also support SOURCE_DATE_EPOCH.
$ cri-dockerd --version
cri-dockerd 0.2.0 (HEAD)
Apparently the support for supplying the version and commit at build-time never materialized ?
var (
// Version of the product
Version = "0.2.0"
// PreRelease is set during the build
PreRelease = ""
// GitCommit is set during the build
GitCommit = "HEAD"
// BuildTime is set during the build
BuildTime = "<unknown>"
)
https://github.com/Mirantis/cri-dockerd/blob/master/src/version/version.go
While building on vm using the build command :
cd src && go get && go build -o ../bin/cri-dockerd
The logs are:
package github.com/Mirantis/cri-dockerd/cmd: cannot find package "github.com/Mirantis/cri-dockerd/cmd" in any of:
/usr/src/github.com/Mirantis/cri-dockerd/cmd (from $GOROOT)
/home/vagrant/go/src/github.com/Mirantis/cri-dockerd/cmd (from $GOPATH)
package io/fs: unrecognized import path "io/fs" (import path does not begin with hostname)
package go.opentelemetry.io/otel/exporters/otlp/otlpgrpc: cannot find package "go.opentelemetry.io/otel/exporters/otlp/otlpgrpc" in any of:
/usr/src/go.opentelemetry.io/otel/exporters/otlp/otlpgrpc (from $GOROOT)
/home/vagrant/go/src/go.opentelemetry.io/otel/exporters/otlp/otlpgrpc (from $GOPATH)`
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.