rootless-containers / usernetes Goto Github PK
View Code? Open in Web Editor NEWKubernetes without the root privileges
License: Apache License 2.0
Kubernetes without the root privileges
License: Apache License 2.0
drwxr-xr-x travis/travis 0 2018-10-23 04:18 usernetes/bin/crio/
-rwxr-xr-x travis/travis 7765512 2018-10-23 04:18 usernetes/bin/crio/runc
drwxr-xr-x travis/travis 0 2018-10-23 04:16 usernetes/bin/crio/conf/
-rw-r--r-- travis/travis 708 2018-10-23 04:16 usernetes/bin/crio/conf/README.md
-rw-r--r-- travis/travis 54 2018-10-23 04:16 usernetes/bin/crio/conf/99-loopback.conf
-rw-r--r-- travis/travis 294 2018-10-23 04:16 usernetes/bin/crio/conf/10-crio-bridge.conf
-rwxr-xr-x travis/travis 40267088 2018-10-23 04:16 usernetes/bin/crio/crio
drwxr-xr-x travis/travis 0 2018-10-23 04:17 usernetes/bin/crio/plugins/
-rwxr-xr-x travis/travis 11686112 2018-10-23 04:17 usernetes/bin/crio/plugins/dhcp
-rwxr-xr-x travis/travis 5189984 2018-10-23 04:17 usernetes/bin/crio/plugins/bridge
-rwxr-xr-x travis/travis 4745880 2018-10-23 04:17 usernetes/bin/crio/plugins/macvlan
-rwxr-xr-x travis/travis 4240480 2018-10-23 04:17 usernetes/bin/crio/plugins/host-local
-rwxr-xr-x travis/travis 4702000 2018-10-23 04:17 usernetes/bin/crio/plugins/vlan
-rwxr-xr-x travis/travis 4706552 2018-10-23 04:17 usernetes/bin/crio/plugins/ipvlan
-rwxr-xr-x travis/travis 5152856 2018-10-23 04:17 usernetes/bin/crio/plugins/ptp
-rwxr-xr-x travis/travis 4094744 2018-10-23 04:17 usernetes/bin/crio/plugins/host-device
-rwxr-xr-x travis/travis 2927370 2018-10-23 04:17 usernetes/bin/crio/plugins/flannel
-rwxr-xr-x travis/travis 2746953 2018-10-23 04:17 usernetes/bin/crio/plugins/static
-rwxr-xr-x travis/travis 4599416 2018-10-23 04:17 usernetes/bin/crio/plugins/bandwidth
-rwxr-xr-x travis/travis 4632648 2018-10-23 04:17 usernetes/bin/crio/plugins/portmap
-rwxr-xr-x travis/travis 4091368 2018-10-23 04:17 usernetes/bin/crio/plugins/tuning
-rwxr-xr-x travis/travis 4031872 2018-10-23 04:17 usernetes/bin/crio/plugins/loopback
-rwxr-xr-x travis/travis 2716920 2018-10-23 04:17 usernetes/bin/crio/plugins/sample
-rwxr-xr-x travis/travis 55432 2018-10-23 04:16 usernetes/bin/crio/conmon
plugins
and conf
were expected to be located under the usernetes/bin/crio/cni
Weirdly, the issue does not happen for my local artifact (built with docker-archive/engine@bc4c1c2 , DOCKER_BUILDKIT=1)
kube-apiserver should be launched with--authorization-mode=Node,RBAC
https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/1.15.3/docs/08-bootstrapping-kubernetes-controllers.md
This would need additional service for determining slirp4netns CIDR per each of node.
(RootlessKit should determine CIDR using etcd?)
The problem is that vagrant can't be used on any free CI AFAIK 😢
$ task -d build build-crio
...
[build-crio] #7 230.5 ./build.sh: 27: [: windows: unexpected operator
[build-crio] #7 230.5 windows
[build-crio] #7 230.5 can't load package: package github.com/containernetworking/plugins/plugins/main/windows: no Go files in /go/src/github.com/containernetworking/plugins/
gopath/src/github.com/containernetworking/plugins/plugins/main/windows
[build-crio] executor failed running [/bin/sh -c git clone https://github.com/kubernetes-incubator/cri-o.git /go/src/github.com/kubernetes-incubator/cri-o && cd /go/src/gi
thub.com/kubernetes-incubator/cri-o && git checkout ${CRIO_COMMIT} && make BUILDTAGS="exclude_graphdriver_btrfs exclude_graphdriver_devicemapper containers_image_openpgp"
binaries && mkdir -p /crio/cni/plugins/ /crio/cni/conf && cp bin/conmon bin/crio /crio && cp contrib/cni/* /crio/cni/conf && git clone https://github.com/containerne
tworking/plugins /go/src/github.com/containernetworking/plugins && cd /go/src/github.com/containernetworking/plugins && sh ./build.sh -ldflags "-extldflags -static" &&
cp bin/* /crio/cni/plugins && git clone https://github.com/opencontainers/runc.git /go/src/github.com/opencontainers/runc && cd /go/src/github.com/opencontainers/runc &
& git checkout ${RUNC_COMMIT} && make BUILDTAGS="" SHELL=/bin/sh static && cp runc /crio/runc]: exit code: 1
[build-crio] #7 completed: 2018-10-03 11:31:02.974964005 +0000 UTC
[build-crio] #7 duration: 3m50.964761354s
[build-crio] #7 error: "executor failed running [/bin/sh -c git clone https://github.com/kubernetes-incubator/cri-o.git /go/src/github.com/kubernetes-incubator/cri-o
&& cd /go/src/github.com/kubernetes-incubator/cri-o && git checkout ${CRIO_COMMIT} && make BUILDTAGS=\"exclude_graphdriver_btrfs exclude_graphdriver_devicemapper contain
ers_image_openpgp\" binaries && mkdir -p /crio/cni/plugins/ /crio/cni/conf && cp bin/conmon bin/crio /crio && cp contrib/cni/* /crio/cni/conf && git clone https://gi
thub.com/containernetworking/plugins /go/src/github.com/containernetworking/plugins && cd /go/src/github.com/containernetworking/plugins && sh ./build.sh -ldflags \"-ext
ldflags -static\" && cp bin/* /crio/cni/plugins && git clone https://github.com/opencontainers/runc.git /go/src/github.com/opencontainers/runc && cd /go/src/github.com
/opencontainers/runc && git checkout ${RUNC_COMMIT} && make BUILDTAGS=\"\" SHELL=/bin/sh static && cp runc /crio/runc]: exit code: 1"
[build-crio]
@giuseppe Could you please pin CNI commit?
I've found that the kubernetes-sigs/sig-storage-local-static-provisioner attempts to mount /dev from the host in its provisioner container, which causes issues when running the daemonset under u7s. Given that there are no reports of this being broken on their issue tracker, I presumed it was an unexpected failure case for u7s rather than them doing the wrong thing (feel free to correct me on this).
It's fairly trivial to work around the issue by setting the terminationMessagePath
for the container, although it's not clear to me if this would result in termination messages being lost. See the badness.yaml
snippet below and its failure events in describe pod badness
:
$ cat badness.yaml
apiVersion: v1
kind: Pod
metadata:
name: badness
labels:
app: badness
spec:
containers:
- image: busybox
name: badness
command: ["sleep", "infinity"]
volumeMounts:
- name: shared-dev
mountPath: /dev
#terminationMessagePath: /var/run/termination-log
volumes:
- name: shared-dev
hostPath:
path: /dev/
$ kubectl apply -f badness.yaml
pod/badness created
$ kubectl describe pod badness
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> Successfully assigned default/user-storage-provisioner-7sn8s to host
Normal Pulled 2s (x2 over 2s) kubelet, host Container image "quay.io/external_storage/local-volume-provisioner:v2.3.4" already present on machine
Warning Failed 2s kubelet, host Error: container create failed: open `/home/user/.local/share/usernetes/containers/storage/overlay/6d3e5e577d2174a25f2947faa0af0b5e68af13bedb662eaaf59068f173640091/merged/termination-log`: No such file or directory
Warning Failed 2s kubelet, host Error: container create failed: open `/home/user/.local/share/usernetes/containers/storage/overlay/273772f44ad6a6e0d55af8b2d3dadb154c734084d4bc82b7f7e4f0cd7e8bda8e/merged/termination-log`: No such file or directory
Uncommenting the terminationMessagePath
in badness.yaml
makes the pod start happily.
When we open PRs to the kubernetes/kubernetes upstream, maybe we should squash commits. (Signed-Off-By should be kept)
etcd and kube-{apiserver,scheduler,controller-manager} do not require privileges and should be executed as a plain old rootless user without nsentering into RootlessKit's USER+MNT+NET namespaces.
The following Dockerfile does not work
FROM alpine:3.8
RUN apk add --no-cache git build-base autoconf automake
RUN git clone git://repo.or.cz/socat.git /socat
WORKDIR /socat
ARG SOCAT_COMMIT
RUN git pull && git checkout ${SOCAT_COMMIT}
RUN autoconf && ./configure LDFLAGS="-static" && make && \
mkdir -p /out && cp -f socat /out
#8 15.77 gcc -O -D_GNU_SOURCE -Wall -Wno-parentheses -DHAVE_CONFIG_H -I. -I. -c -o xio-socket.o xio-socket.c
#8 15.82 xio-socket.c: In function 'xiocheckpeer':
#8 15.82 xio-socket.c:1713:8: warning: unused variable 'result' [-Wunused-variable]
#8 15.82 int result;
#8 15.82 ^~~~~~
#8 16.02 gcc -O -D_GNU_SOURCE -Wall -Wno-parentheses -DHAVE_CONFIG_H -I. -I. -c -o xio-interface.o xio-interface.c
#8 16.07 gcc -O -D_GNU_SOURCE -Wall -Wno-parentheses -DHAVE_CONFIG_H -I. -I. -c -o xio-listen.o xio-listen.c
#8 16.15 gcc -O -D_GNU_SOURCE -Wall -Wno-parentheses -DHAVE_CONFIG_H -I. -I. -c -o xio-unix.o xio-unix.c
#8 16.26 gcc -O -D_GNU_SOURCE -Wall -Wno-parentheses -DHAVE_CONFIG_H -I. -I. -c -o xio-ip.o xio-ip.c
#8 16.36 gcc -O -D_GNU_SOURCE -Wall -Wno-parentheses -DHAVE_CONFIG_H -I. -I. -c -o xio-ip4.o xio-ip4.c
#8 16.39 In file included from xioopen.h:10:0,
#8 16.39 from xio-ip4.c:11:
#8 16.39 xio-ip4.c: In function 'xioparsenetwork_ip4':
#8 16.39 xio-ip4.c:49:14: error: 'NETDB_INTERNAL' undeclared (first use in this function)
#8 16.39 h_errno == NETDB_INTERNAL ? strerror(errno) :
#8 16.39 ^
#8 16.39 error.h:47:42: note: in definition of macro 'Error2'
#8 16.39 #define Error2(m,a1,a2) msg(E_ERROR,m,a1,a2)
#8 16.39 ^~
#8 16.39 xio-ip4.c:49:14: note: each undeclared identifier is reported only once for each function it appears in
#8 16.39 h_errno == NETDB_INTERNAL ? strerror(errno) :
#8 16.39 ^
#8 16.39 error.h:47:42: note: in definition of macro 'Error2'
#8 16.39 #define Error2(m,a1,a2) msg(E_ERROR,m,a1,a2)
#8 16.39 ^~
#8 16.39 make: *** [<builtin>: xio-ip4.o] Error 1
#8 completed: 2018-11-09 10:03:29.262687054 +0000 UTC
#8 duration: 16.602700111s
#8 error: "executor failed running [/bin/sh -c autoconf && ./configure LDFLAGS=\"-static\" && make && mkdir -p /out && cp -f socat /out]: exit code: 2"
executor failed running [/bin/sh -c autoconf && ./configure LDFLAGS="-static" && make && mkdir -p /out && cp -f socat /out]: exit code: 2
MAINPID
needs to be converted across the PID namespace boundary
Memo
sudo modprobe ip_tables iptable_mangle iptable_nat iptable_filter
sudo prlimit --nofile=:65536 --pid $$
kubectl.sh
needs --preserve-credential
for nsenter
$ ./dockercli.sh run -it --rm alpine
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
8e3ba11ec2a2: Extracting 2.207MB/2.207MB
docker: failed to register layer: ApplyLayer exit status 1 stdout: stderr: lchown /etc/shadow: invalid argument.
usernetes: v20180824.0
vagrant@vagrant-openSUSE-Tumbleweed:~/usernetes> id
uid=1000(vagrant) gid=100(users) groups=100(users),1000(vagrant)
vagrant@vagrant-openSUSE-Tumbleweed:~/usernetes> cat /etc/subuid
vagrant:231072:65536
vagrant@vagrant-openSUSE-Tumbleweed:~/usernetes> cat /etc/subgid
vagrant:231072:65536
vagrant@vagrant-openSUSE-Tumbleweed:~/usernetes> cat /etc/os-release
NAME="openSUSE Tumbleweed"
# VERSION="20180827"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20180827"
PRETTY_NAME="openSUSE Tumbleweed"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:tumbleweed:20180827"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
vagrant@vagrant-openSUSE-Tumbleweed:~/usernetes> uname -a
Linux vagrant-openSUSE-Tumbleweed 4.18.5-1-default #1 SMP PREEMPT Fri Aug 24 12:38:43 UTC 2018 (9e91e29) x86_64 x86_64 x86_64 GNU/Linux
ref: cri-o/cri-o#1729
cc @giuseppe
[kubelet-crio] E0824 07:42:07.848387 13836 pod_workers.go:186] Error syncing pod f2bae4d7-a770-11e8-8182-7a7eddbf465f ("foo-7b675c6d5-trfmv_default(f2bae4d7-a770-11e8-8182-7a7eddbf465f)"), skipping: failed to "CreatePodSandbox" for "foo-7b675c6d5-trfmv_default(f2bae4d7-a770-11e8-8182-7a7eddbf465f)" with CreatePodSandboxError: "CreatePodSandbox for pod \"foo-7b675c6d5-trfmv_default(f2bae4d7-a770-11e8-8182-7a7eddbf465f)\" failed: rpc error: code = Unknown desc = error creating pod sandbox with name \"k8s_foo-7b675c6d5-trfmv_default_f2bae4d7-a770-11e8-8182-7a7eddbf465f_0\": open /etc/containers/policy.json: no such file or directory"
I am observing DNS errors after running docker-compose
under usernetes (specifically default-docker-nokube
) for "a long time". This affects all services (i.e., containers) in my composition. Bringing the composition down and back up resolves the issue - I do not have to restart dockerd
.
For example, after running
./run.sh default-docker-nokube
for several hours (e.g., 19 hours), my DNS fails inside all docker containers.
# python -c "import socket; python -c "import socket; print(socket.gethostbyname('google-public-dns-a.google.com'))"
Traceback (most recent call last):
File "<string>", line 1, in <module>
socket.gaierror: [Errno -3] Temporary failure in name resolution
whereas it works fine outside:
$ python -c "import socket; print(socket.gethostbyname('google-public-dns-a.google.com'))"
8.8.8.8
Here is the context in which I am experiencing this:
$ uname -srvpio
Linux 4.18.0-17-generic #18~18.04.1-Ubuntu SMP Fri Mar 15 15:27:12 UTC 2019 x86_64 x86_64 GNU/Linux
$ docker-compose --version
docker-compose version 1.24.0, build 0aa5906
$ docker --version
Docker version 18.09.3-rc1, build 516e093
https://github.com/AkihiroSuda/balancker could be used for load-balancing with multiple remote Docker hosts
we are currently running each service (etcd, k8s services, runtime) as part of a Taskfile.yml
.
It'd be nice if we support restarting a single service, perhaps not using go-task
for handling the bootstrap of Usernetes.
Perhaps systemctl --user
?
The biggest problem I see is that all the services must be in the same user namespace
Probably using Vagrant
The pods like pause is failing to be pulled:
tePodSandbox for pod \"pod1_default(afaf1c4e-9a2d-11e8-b119-ae906b331d15)\" failed: rpc error: code = Unknown desc = failed pulling image \"k8s.gcr.io/pause:3.1\": Error response from daemon: open /etc/docker/certs.d/k8s.gcr.io: permission denied"
I was able to manually run containers with the dockercli.sh
script and using the vfs driver in fedora.
hyperkube kubelet \
--cert-dir /tmp/usernetes/pki \
--root-dir /tmp/usernetes/kubelet \
--log-dir /tmp/usernetes/kubelet-log \
--volume-plugin-dir /tmp/usernetes/kubelet-plugins-exec \
--docker-endpoint unix://${XDG_RUNTIME_DIR}/docker.sock \
--kubeconfig localhost.kubeconfig \
--anonymous-auth=true \
--authorization-mode=AlwaysAllow \
--fail-swap-on=false \
--feature-gates DevicePlugins=false
[kubelet] Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
[kubelet] Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
[kubelet] Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
[kubelet] Flag --feature-gates has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
[kubelet] I0723 14:03:49.502669 4922 server.go:406] Version: v1.12-usernetes
[kubelet] I0723 14:03:49.502912 4922 plugins.go:97] No cloud provider specified.
[kubelet] I0723 14:03:49.537776 4922 server.go:646] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
[kubelet] I0723 14:03:49.538427 4922 container_manager_linux.go:244] container manager verified user specified cgroup-root exists: []
[kubelet] I0723 14:03:49.538452 4922 container_manager_linux.go:249] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/tmp/usernetes/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true}
[kubelet] I0723 14:03:49.538588 4922 container_manager_linux.go:268] Creating device plugin manager: false
[kubelet] I0723 14:03:49.538627 4922 state_mem.go:36] [cpumanager] initializing new in-memory state store
[kubelet] I0723 14:03:49.538796 4922 state_mem.go:84] [cpumanager] updated default cpuset: ""
[kubelet] I0723 14:03:49.538825 4922 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"
[kubelet] W0723 14:03:49.739246 4922 server.go:719] write /proc/self/oom_score_adj: permission denied
[kubelet] I0723 14:03:49.739482 4922 kubelet.go:297] Watching apiserver
[kubelet] I0723 14:03:49.749619 4922 client.go:75] Connecting to docker on unix:///run/user/1000/docker.sock
[kubelet] I0723 14:03:49.749723 4922 client.go:104] Start docker client with request timeout=2m0s
[kubelet] W0723 14:03:49.752820 4922 docker_service.go:545] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
[kubelet] I0723 14:03:49.752851 4922 docker_service.go:238] Hairpin mode set to "hairpin-veth"
[kubelet] W0723 14:03:49.752962 4922 cni.go:180] Unable to update cni config: No networks found in /etc/cni/net.d
[kubelet] W0723 14:03:49.756544 4922 hostport_manager.go:68] The binary conntrack is not installed, this can cause failures in network connection cleanup.
[kubelet] W0723 14:03:49.758685 4922 plugins.go:186] can't set sysctl net/bridge/bridge-nf-call-iptables: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory
[kubelet] I0723 14:03:49.758755 4922 docker_service.go:253] Docker cri networking managed by kubernetes.io/no-op
[kubelet] I0723 14:03:49.765941 4922 docker_service.go:258] Docker Info: &{ID:IPII:W3CP:WJVF:WA2C:266N:Y7YJ:FD5N:UJX2:MUHX:2LQO:ZEBF:7XJ5 Containers:0 ContainersRunning:0 ContainersPaused:0 ContainersStopped:0 Images:0 Driver:vfs DriverStatus:[] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true IPv4Forwarding:true BridgeNfIptables:false BridgeNfIP6tables:false Debug:false NFd:22 OomKillDisable:true NGoroutines:43 SystemTime:2018-07-23T14:03:49.760094599+05:30 LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:4.17.6-200.fc28.x86_64 OperatingSystem:Fedora 28 (Workstation Edition) OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc420976a80 NCPU:4 MemTotal:16724094976 GenericResources:[] DockerRootDir:/home/frezbo/.local/share/docker HTTPProxy: HTTPSProxy: NoProxy: Name:localhost.localdomain Labels:[] ExperimentalBuild:true ServerVersion:dev ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:docker-runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:d64c661f1d51c48782c9cec8fda7604785f93587 Expected:d64c661f1d51c48782c9cec8fda7604785f93587} RuncCommit:{ID:2c632d1a2de0192c3f18a2542ccb6f30a8719b1f Expected:2c632d1a2de0192c3f18a2542ccb6f30a8719b1f} InitCommit:{ID:fec3683 Expected:fec3683} SecurityOptions:[name=seccomp,profile=default name=rootless]}
[kubelet] I0723 14:03:49.766018 4922 docker_service.go:271] Setting cgroupDriver to cgroupfs
[kubelet] I0723 14:03:49.778557 4922 kuberuntime_manager.go:186] Container runtime docker initialized, version: dev, apiVersion: 1.38.0
[kubelet] I0723 14:03:49.779030 4922 csi_plugin.go:110] kubernetes.io/csi: plugin initializing...
[kubelet] I0723 14:03:49.780158 4922 server.go:984] Started kubelet
[kubelet] E0723 14:03:49.781268 4922 kubelet.go:1223] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
[kubelet] I0723 14:03:49.783766 4922 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
[kubelet] I0723 14:03:49.783864 4922 status_manager.go:152] Starting to sync pod status with apiserver
[kubelet] I0723 14:03:49.783924 4922 kubelet.go:1720] Starting kubelet main sync loop.
[kubelet] I0723 14:03:49.783996 4922 kubelet.go:1737] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]
[kubelet] I0723 14:03:49.784466 4922 server.go:129] Starting to listen on 0.0.0.0:10250
[kubelet] I0723 14:03:49.785396 4922 server.go:303] Adding debug handlers to kubelet server.
[kubelet] I0723 14:03:49.801132 4922 volume_manager.go:247] Starting Kubelet Volume Manager
[kubelet] I0723 14:03:49.801575 4922 desired_state_of_world_populator.go:130] Desired state populator starts to run
[kubelet] I0723 14:03:49.884386 4922 kubelet.go:1737] skipping pod synchronization - [container runtime is down]
[kubelet] I0723 14:03:49.901712 4922 kubelet_node_status.go:268] Setting node annotation to enable volume controller attach/detach
[kubelet] I0723 14:03:49.904748 4922 kubelet_node_status.go:78] Attempting to register node localhost.localdomain
[kubelet] I0723 14:03:49.910958 4922 kubelet_node_status.go:122] Node localhost.localdomain was previously registered
[kubelet] I0723 14:03:49.910993 4922 kubelet_node_status.go:81] Successfully registered node localhost.localdomain
[kubelet] I0723 14:03:49.919312 4922 kubelet_node_status.go:824] Node became not ready: {Type:Ready Status:False LastHeartbeatTime:2018-07-23 14:03:49.919290603 +0530 IST m=+0.606415439 LastTransitionTime:2018-07-23 14:03:49.919290603 +0530 IST m=+0.606415439 Reason:KubeletNotReady Message:container runtime is down}
[kubelet] I0723 14:03:50.090399 4922 kubelet.go:1737] skipping pod synchronization - [container runtime is down]
[kubelet] E0723 14:03:50.170425 4922 container_manager_linux.go:98] Unable to ensure the docker processes run in the desired containers: [errors moving "docker" pid: failed to find pid namespace of init process, errors moving "docker-containerd" pid: failed to find pid namespace of init process, errors moving "docker-containerd" pid: failed to find pid namespace of init process]
[kubelet] I0723 14:03:50.322927 4922 cpu_manager.go:155] [cpumanager] starting with none policy
[kubelet] I0723 14:03:50.322971 4922 cpu_manager.go:156] [cpumanager] reconciling every 10s
[kubelet] I0723 14:03:50.322991 4922 policy_none.go:42] [cpumanager] none policy: Start
[kubelet] F0723 14:03:50.323422 4922 kubelet.go:1295] Failed to start ContainerManager failed to get rootfs info: failed to get device for dir "/tmp/usernetes/kubelet": could not find device with major: 0, minor: 45 in cached partitions map
Seems these two are not available when running under docker-rootlesskit
I think this is causing the kubelet to fail
OS Info
Fedora 28
~ (aws:rean-gov-sd)(kc)$ uname -am
Linux localhost.localdomain 4.17.6-200.fc28.x86_64 #1 SMP Wed Jul 11 20:29:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
~ (aws:rean-gov-sd)(kc)$
bin/hyperkube:
linux-vdso.so.1 (0x00007ffcfd788000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f04cfbc8000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f04cf82a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f04cf60b000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f04cf407000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f04cf016000)
/lib64/ld-linux-x86-64.so.2 (0x00007f04cff56000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f04cedfe000)
During SIG-node meeting today, there was a discussion about using some virtual kubelet provider instead of modifying kubelet.
https://github.com/virtual-kubelet/virtual-kubelet
Maybe we can easily implement virtual kubelet provider with rootless CRI runtimes or even rootless Podman (I suppose implementing Podman provider is much easier), but I'm not sure about drawbacks.
Probably it doesn't work well with Services
?
If we could separate out CgroupManager
to a separate gRPC service that would listen on an UNIX socket, it might be easier to experiment cgroup2 driver as well as "NOP" driver.
@giuseppe WDYT?
Description
Launching usernetes + enable cgroup V2, and create a pod failed.
kubectl describe the pod, show :
Warning FailedCreatePodSandBox 9s (x4 over 30s) kubelet, izj6cgx3q19bmgalpqjydtz Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: sd-bus call: Permission denied: unknown
Steps to reproduce the issue:
Running Ubuntu 20.04
Linux iZj6cgx3q19bmgalpqjydtZ 5.4.0-31-generic #35-Ubuntu SMP Thu May 7 20:20:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Changed grub linux arguments to support the unified hierarchy:
GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1 vga=792 console=tty0 console=ttyS0,115200n8 net.ifnames=0 noibrs"
Install u7s:
./install.sh --cni=flannel --cri=containerd --publish=0.0.0.0:8472:8472/udp --publish=0.0.0.0:2379:2379/tcp --publish=0.0.0.0:6443:6443/tcp --cgroup-manager=systemd
kubectl create -f pod.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: busybox-master
spec:
selector:
matchLabels:
user: test-master
replicas: 1
template:
metadata:
labels:
user: test-master
spec:
containers:
- name: busybox-master
image: busybox
command: ['sleep','100000000']
XDG_RUNTIME_DIR is exist:
paas@iZj6cgx3q19bmgalpqjydtZ:~/usernetes$ echo $XDG_RUNTIME_DIR
/run/user/1000
The current POC is implemented by just putting a bunch of non-robust rsystem.RunningInUserNS()
check to a random set of functions: AkihiroSuda/kubernetes@8318759
We need to reduce these checks and implement pkg/kubelet/cm in the proper way.
Especially, pcm.Exists(existingPodName)
needs to be implemented to return true even when cgroups is not available. (#17)
cc @cyphar
$ docker run -d --name usernetes-node -p 127.0.0.1:8080:8080 -e U7S_ROOTLESSKIT_PORTS=0.0.0.0:8080:8080/tcp --privileged rootlesscontainers/usernetes default-docker
03c3f4df528ff65f8bef7733613ce1443424356a7fadd60b74012499a48742df
$ docker container logs usernetes-node
./boot/kube-proxy.sh
./boot/etcd.sh
./boot/rootlesskit.sh
./boot/dockerd.sh
./boot/kube-scheduler.sh
./boot/kubelet-dockershim.sh
./boot/kube-controller-manager.sh
./boot/kube-apiserver.sh
[rootlesskit] open: No such file or directory
[rootlesskit] [rootlesskit:parent] error: failed to setup network &{binary:slirp4netns mtu:65520 ipnet:<nil> disableHostLoopback:true apiSocketPath: enableSandbox:true enableSeccomp:true}: setting up tap tap0: executing [[nsenter -t 107 -n -m -U --preserve-credentials ip tuntap add name tap0 mode tap] [nsenter -t 107 -n -m -U --preserve-credentials ip link set tap0 up]]: exit status 1
[rootlesskit] [rootlesskit:child ] error: parsing message from fd 3: EOF
[kube-proxy] [INFO] Entering RootlessKit namespaces: ...
[kube-apiserver] [INFO] Entering RootlessKit namespaces: ...
[etcd] [INFO] Entering RootlessKit namespaces: ...
[kube-controller-manager] [INFO] Entering RootlessKit namespaces: ...
[kubelet-dockershim] [INFO] Entering RootlessKit namespaces: ...
[kube-scheduler] [INFO] Entering RootlessKit namespaces: ...
task: Failed to run task "rootlesskit": exit status 1
[dockerd] [INFO] Entering RootlessKit namespaces: ...
$
I.e. equivalent of https://github.com/moby/vpnkit/tree/master/go/cmd/kube-vpnkit-forwarder
RootlessKit will provide implementation-agnostic REST API for exposing a port in the child netns to the parent netns, via an UNIX socket.
e.g. to expose 80/tcp as 8080/tcp in the host:
curl -X PUT -H Content-Type:application/json -d '{"childPort":80}' --unix-socket /run/user/1001/usernetes/rootlesskit/rootlesskit.sock http://v1/parentPorts/tcp/8080
Planned implementations:
builtin
Built-in implementation based on libnetwork proxy
reexec-with-cgroups
Re-exec builtin
with cgroups
vpnkit-native
Uses VPNKit "VMN3T" API
slirp4netns-native
Uses slirp4netns builtin forwarder.
slirp4netns needs to be modified.
exec-privileged-iptables
Executes iptables with some SETUID/SETCAP helper.
Less secure but fast.
kube-vpnkit-forwarder
into kube-rootlesskit-forwader
kube-rootlesskit-forwarder
to Usernetescc @giuseppe
via kubeadm probably #69
the pod status remains Terminating
Reproducible with all CRI runtimes
UPDATE: now crio is dynamically linked with glibc instead of musl libc
$ ldd bin/crio/crio
linux-vdso.so.1 (0x00007ffd1c117000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fdf5c866000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fdf5c662000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdf5c271000)
/lib64/ld-linux-x86-64.so.2 (0x00007fdf5ca85000)
(previously crio was unexpectedly linked with musl libc)
penguin01@suda-ws01:/tmp/usernetes$ find bin -executable -type f | xargs ldd
bin/docker-proxy:
not a dynamic executable
bin/dockerd:
not a dynamic executable
bin/docker:
not a dynamic executable
bin/docker-containerd-shim:
not a dynamic executable
bin/docker-runc:
not a dynamic executable
bin/docker-init:
not a dynamic executable
bin/task:
not a dynamic executable
bin/docker-containerd-ctr:
not a dynamic executable
bin/hyperkube:
linux-vdso.so.1 (0x00007ffcfd788000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f04cfbc8000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f04cf82a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f04cf60b000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f04cf407000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f04cf016000)
/lib64/ld-linux-x86-64.so.2 (0x00007f04cff56000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f04cedfe000)
bin/rootlesskit:
not a dynamic executable
bin/etcdctl:
not a dynamic executable
bin/etcd:
not a dynamic executable
bin/docker-containerd:
not a dynamic executable
bin/crio/runc:
not a dynamic executable
bin/crio/conmon:
statically linked
bin/crio/cni/plugins/static:
linux-vdso.so.1 (0x00007ffe7d7ca000)
libc.musl-x86_64.so.1 => not found
bin/crio/cni/plugins/vlan:
not a dynamic executable
bin/crio/cni/plugins/bandwidth:
not a dynamic executable
bin/crio/cni/plugins/ptp:
not a dynamic executable
bin/crio/cni/plugins/loopback:
not a dynamic executable
bin/crio/cni/plugins/ipvlan:
not a dynamic executable
bin/crio/cni/plugins/host-local:
not a dynamic executable
bin/crio/cni/plugins/dhcp:
not a dynamic executable
bin/crio/cni/plugins/bridge:
not a dynamic executable
bin/crio/cni/plugins/tuning:
not a dynamic executable
bin/crio/cni/plugins/host-device:
not a dynamic executable
bin/crio/cni/plugins/sample:
linux-vdso.so.1 (0x00007fff39ac6000)
libc.musl-x86_64.so.1 => not found
bin/crio/cni/plugins/macvlan:
not a dynamic executable
bin/crio/cni/plugins/flannel:
linux-vdso.so.1 (0x00007ffe96b86000)
libc.musl-x86_64.so.1 => not found
bin/crio/cni/plugins/portmap:
not a dynamic executable
bin/crio/crio:
linux-vdso.so.1 (0x00007fff5e7f1000)
libgpgme.so.11 => /usr/lib/x86_64-linux-gnu/libgpgme.so.11 (0x00007f5fffab4000)
libc.musl-x86_64.so.1 => not found
libassuan.so.0 => /usr/lib/x86_64-linux-gnu/libassuan.so.0 (0x00007f5fff8a1000)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f5fff68c000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5fff29b000)
/lib/ld-musl-x86_64.so.1 => /lib64/ld-linux-x86-64.so.2 (0x00007f5fffcfb000)
bin/slirp4netns:
statically linked
@giuseppe PTAL?
$ ./kubectl.sh exec -it foo-68d78ff5c9-bdzs5 ls
OCI runtime exec failed: exec failed: container_linux.go:336: starting container process caused "process_linux.go:86: executing setns process caused \"exit status 22\"": unknown
command terminated with exit code 126
https://twitter.com/_AkihiroSuda_/status/1135154204731772928
possible backends:
Following the requirements and download steps, I wasn't able to use even dockerd (but actually the fault of rootlesskit) on a OpenVZ container:
[root@s usernetes]# ./run.sh default-docker-nokube
./boot/dockerd.sh
./boot/rootlesskit.sh
[rootlesskit] [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted
task: Failed to run task "rootlesskit": exit status 1
uname:
[root@s usernetes]# uname -a
Linux s 2.6.32-042stab128.2 #1 SMP Thu Mar 22 10:58:36 MSK 2018 x86_64 x86_64 x86_64 GNU/Linux
Template: CentOS 7 Minimal
However, cat /proc/self/exe
does work, and it shows a bunch of garbled characters (well they're assembly anyway)
usernetes/boot/kube-proxy.sh
proxy-mode can iptables or ipvs be supported ?
+ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
localhost.localdomain Ready <none> 13s v1.19.0-usernetes 10.0.2.100 <none> Fedora 31 (Cloud Edition) 5.3.7-301.fc31.x86_64 containerd://1.3.0-429-g1c1a08e7
+ timeout 60 kubectl run --rm -i --image busybox --restart=Never hello echo hello
pod "hello" deleted
pod default/hello terminated (StartError)
failed to create containerd task: failed to mount rootfs component &{fuse3.fuse-overlayfs overlay [workdir=/home/vagrant/.local/share/usernetes/containerd/io.containerd.snapshotter.v1.fuse-overlayfs/snapshots/4/work upperdir=/home/vagrant/.local/share/usernetes/containerd/io.containerd.snapshotter.v1.fuse-overlayfs/snapshots/4/fs lowerdir=/home/vagrant/.local/share/usernetes/containerd/io.containerd.snapshotter.v1.fuse-overlayfs/snapshots/3/fs writeback=0]}: mount helper [mount.fuse3 [overlay /run/user/1000/usernetes/containerd/io.containerd.runtime.v2.task/k8s.io/dcbe12d6759bd896b9c0a65d6f74224652110671371895a27099a1385f4e74a1/rootfs -o workdir=/home/vagrant/.local/share/usernetes/containerd/io.containerd.snapshotter.v1.fuse-overlayfs/snapshots/4/work -o upperdir=/home/vagrant/.local/share/usernetes/containerd/io.containerd.snapshotter.v1.fuse-overlayfs/snapshots/4/fs -o lowerdir=/home/vagrant/.local/share/usernetes/containerd/io.containerd.snapshotter.v1.fuse-overlayfs/snapshots/3/fs -o writeback=0 -t fuse-overlayfs]] failed: "": wait: no child processes: unknown
+ ERROR 'Pod is not ready.'
https://travis-ci.org/github/rootless-containers/usernetes/builds/668025122
2020-08-13T10:43:48.2176040Z + INFO 'Connecting to dnstest-{0,1,2}.dnstest.default.svc.cluster.local'
2020-08-13T10:43:48.2176720Z �[104m�[97m[INFO]�[49m�[39m Connecting to dnstest-{0,1,2}.dnstest.default.svc.cluster.local
2020-08-13T10:43:48.2177370Z + echo -e '\e[104m\e[97m[INFO]\e[49m\e[39m Connecting to dnstest-{0,1,2}.dnstest.default.svc.cluster.local'
2020-08-13T10:43:48.2178050Z + kubectl run -i --rm --image=alpine --restart=Never dnstest-shell -- sh -exc 'for f in $(seq 0 2); do wget -O- http://dnstest-${f}.dnstest.default.svc.cluster.local; done'
2020-08-13T10:43:50.2488250Z + seq 0 2
2020-08-13T10:43:50.2490070Z + wget -O- http://dnstest-0.dnstest.default.svc.cluster.local
2020-08-13T10:43:50.2491290Z Connecting to dnstest-0.dnstest.default.svc.cluster.local (10.88.0.15:80)
2020-08-13T10:43:50.2491580Z writing to stdout
2020-08-13T10:43:50.2492520Z - 100% |********************************| 612 0:00:00 ETA
2020-08-13T10:43:50.2492710Z written to stdout
2020-08-13T10:43:50.2494740Z + wget -O- http://dnstest-1.dnstest.default.svc.cluster.local
2020-08-13T10:43:50.2495120Z <!DOCTYPE html>
2020-08-13T10:43:50.2495370Z <html>
2020-08-13T10:43:50.2495580Z <head>
2020-08-13T10:43:50.2495810Z <title>Welcome to nginx!</title>
2020-08-13T10:43:50.2496050Z <style>
2020-08-13T10:43:50.2496200Z body {
2020-08-13T10:43:50.2496420Z width: 35em;
2020-08-13T10:43:50.2496740Z margin: 0 auto;
2020-08-13T10:43:50.2497740Z font-family: Tahoma, Verdana, Arial, sans-serif;
2020-08-13T10:43:50.2498130Z }
2020-08-13T10:43:50.2498370Z </style>
2020-08-13T10:43:50.2498600Z </head>
2020-08-13T10:43:50.2498820Z <body>
2020-08-13T10:43:50.2498990Z <h1>Welcome to nginx!</h1>
2020-08-13T10:43:50.2499250Z <p>If you see this page, the nginx web server is successfully installed and
2020-08-13T10:43:50.2499500Z working. Further configuration is required.</p>
2020-08-13T10:43:50.2499670Z
2020-08-13T10:43:50.2499860Z <p>For online documentation and support please refer to
2020-08-13T10:43:50.2500170Z <a href="http://nginx.org/">nginx.org</a>.<br/>
2020-08-13T10:43:50.2500560Z Commercial support is available at
2020-08-13T10:43:50.2500850Z <a href="http://nginx.com/">nginx.com</a>.</p>
2020-08-13T10:43:50.2501170Z
2020-08-13T10:43:50.2501360Z <p><em>Thank you for using nginx.</em></p>
2020-08-13T10:43:50.2501580Z </body>
2020-08-13T10:43:50.2501810Z </html>
2020-08-13T10:43:50.2502700Z wget: bad address 'dnstest-1.dnstest.default.svc.cluster.local'
2020-08-13T10:43:50.2878570Z pod default/dnstest-shell terminated (Error)
2020-08-13T10:43:50.2879110Z pod "dnstest-shell" deleted
https://github.com/rootless-containers/usernetes/runs/979933303 (v20200813.0)
The failure isn't reproducible on my laptop
without DOCKER_BUILDKIT=1
, bin/cni/*
files are unexpectedly generated under bin
docker pull fails pulling kube-cross image. Most other images work though.
docker pull k8s.gcr.io/kube-cross:v1.11.2-1
v1.11.2-1: Pulling from kube-cross
bc9ab73e5b14: Already exists
193a6306c92a: Already exists
e5c3f8c317dc: Already exists
a587a86c9dcb: Already exists
1bc310ac474b: Already exists
87ab348d90cc: Already exists
786bc4873ebc: Already exists
251302927e9c: Pull complete
5651f8073a75: Pull complete
8a85eca14b20: Pull complete
ac20f6dc4bc0: Pull complete
683f835d2e99: Pull complete
bf4a87ea5477: Pull complete
0954fe016605: Extracting [==================================================>] 10.69MB/10.69MB
failed to register layer: ApplyLayer exit status 1 stdout: stderr: lchown /usr/local/src/etcd/etcd-v3.2.24-linux-amd64: invalid argument
docker info:
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 72
Server Version: dev
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9f2e07b1fc1342d1c48fe4d7bbb94cb6d1bf278b
runc version: a00bf0190895aa465a5fbed0268888e2c8ddfe85
init version: fec3683
Security Options:
seccomp
Profile: default
rootless
Kernel Version: 4.18.16-300.fc29.x86_64
Operating System: Fedora 29 (Workstation Edition)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.39GiB
Name: host
ID: 4C7X:657N:XBEM:NGW3:76BW:OW55:EEQE:QCZ7:W6XE:FU3Y:Y44M:3WK4
Docker Root Dir: /home/naadir/.local/share/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Kubernetes (https://github.com/rootless-containers/usernetes/tree/master/src/patches/kubernetes)
CRI-O (https://github.com/rootless-containers/usernetes/tree/master/src/patches/crio)
requires rootless-containers/rootlesskit#32
Hey, great project! I tried to get the helm nfs server provisioner running and ran into some roadblocks on a GCE centOS 8 machine.
First, just starting the helm chart with this config: (run through envsubst)
replicaCount: 1
image:
repository: quay.io/kubernetes_incubator/nfs-provisioner
tag: v2.2.1-k8s1.12
pullPolicy: IfNotPresent
service:
type: ClusterIP
nfsPort: 2049
mountdPort: 20048
rpcbindPort: 51413
externalIPs: []
persistence:
enabled: true
storageClass: "-"
accessMode: ReadWriteOnce
size: "100Gi"
storageClass:
create: true
defaultClass: false
name: nfs
allowVolumeExpansion: true
parameters: {}
mountOptions:
- vers=4.1
- noatime
reclaimPolicy: Retain
rbac:
create: true
serviceAccountName: default
resources:
{}
nodeSelector:
kubernetes.io/hostname: ${nodename}
tolerations: []
affinity: {}
Afterwards, the necessary PV is created and a PVC is bound to nfs:
apiVersion: v1
kind: PersistentVolume
metadata:
name: "${nfspvcname}"
spec:
capacity:
storage: "100Gi"
accessModes:
- ReadWriteOnce
hostPath:
path: "${mainstoragepath}"
claimRef:
namespace: default
name: "${nfspvcname}"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: default-pvc
spec:
storageClassName: "nfs"
accessModes:
- ReadWriteMany
resources:
requests:
storage: "90Gi"
$nfspvcname
is set to the PVC created by NFS.
Now the pod for nfs crashes continously:
kubectl describe pod nfs-release-nfs-server-provisioner-0
[INFO] Entering RootlessKit namespaces: OK
Name: nfs-release-nfs-server-provisioner-0
Namespace: default
Node: centos/10.0.2.100
Start Time: Fri, 03 Jan 2020 17:49:16 +0000
Labels: app=nfs-server-provisioner
chart=nfs-server-provisioner-0.3.2
controller-revision-hash=nfs-release-nfs-server-provisioner-79c9977558
heritage=Helm
release=nfs-release
statefulset.kubernetes.io/pod-name=nfs-release-nfs-server-provisioner-0
Annotations: <none>
Status: Running
IP: 10.88.0.4
IPs:
IP: 10.88.0.4
Controlled By: StatefulSet/nfs-release-nfs-server-provisioner
Containers:
nfs-server-provisioner:
Container ID: docker://8ce423f7c0df95d08a4c49531b0fd59d6a8e8d97afd9e7756a99a38e51b9736f
Image: quay.io/kubernetes_incubator/nfs-provisioner:v2.2.1-k8s1.12
Image ID: docker-pullable://quay.io/kubernetes_incubator/nfs-provisioner@sha256:f0f0d9d39f8aac4a2f39a1b0b602baa993bca0f22c982f208ca9d7a0d2b2399f
Ports: 2049/TCP, 20048/TCP, 111/TCP, 111/UDP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/UDP
Args:
-provisioner=cluster.local/nfs-release-nfs-server-provisioner
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Fri, 03 Jan 2020 17:56:16 +0000
Finished: Fri, 03 Jan 2020 17:56:23 +0000
Ready: False
Restart Count: 6
Environment:
POD_IP: (v1:status.podIP)
SERVICE_NAME: nfs-release-nfs-server-provisioner
POD_NAMESPACE: default (v1:metadata.namespace)
Mounts:
/export from data (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-nfs-release-nfs-server-provisioner-0
ReadOnly: false
QoS Class: BestEffort
Node-Selectors: kubernetes.io/hostname=centos
Tolerations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> default-scheduler error while running "VolumeBinding" filter plugin for pod "nfs-release-nfs-server-provisioner-0": pod has unbound immediate PersistentVolumeClaims
Warning FailedScheduling <unknown> default-scheduler error while running "VolumeBinding" filter plugin for pod "nfs-release-nfs-server-provisioner-0": pod has unbound immediate PersistentVolumeClaims
Normal Scheduled <unknown> default-scheduler Successfully assigned default/nfs-release-nfs-server-provisioner-0 to centos
Normal Pulling 11m kubelet, centos Pulling image "quay.io/kubernetes_incubator/nfs-provisioner:v2.2.1-k8s1.12"
Normal Pulled 10m kubelet, centos Successfully pulled image "quay.io/kubernetes_incubator/nfs-provisioner:v2.2.1-k8s1.12"
Normal Pulled 10m (x2 over 10m) kubelet, centos Container image "quay.io/kubernetes_incubator/nfs-provisioner:v2.2.1-k8s1.12" already present on machine
Normal Created 10m (x3 over 10m) kubelet, centos Created container nfs-server-provisioner
Normal Started 10m (x3 over 10m) kubelet, centos Started container nfs-server-provisioner
Warning BackOff 9m47s (x3 over 10m) kubelet, centos Back-off restarting failed container
Warning MissingClusterDNS 50s (x57 over 11m) kubelet, centos pod: "nfs-release-nfs-server-provisioner-0_default(ba5bd6f1-c4af-4f78-93f2-c9a48d126ba9)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.
Since the error seems to be related to missing DNS services I tried to setup kube dns via https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/12-dns-addon.md
I had to change the IP from 10.32.0.10 to 10.0.0.10, but the dns pod also fails:
kubectl describe pod coredns-68567cdb47-78x67 --namespace kube-system
[INFO] Entering RootlessKit namespaces: OK
Name: coredns-68567cdb47-78x67
Namespace: kube-system
Priority Class Name: system-cluster-critical
Node: centos/10.0.2.100
Start Time: Fri, 03 Jan 2020 17:48:54 +0000
Labels: k8s-app=kube-dns
pod-template-hash=68567cdb47
Annotations: <none>
Status: Running
IP: 10.88.0.3
IPs:
IP: 10.88.0.3
Controlled By: ReplicaSet/coredns-68567cdb47
Containers:
coredns:
Container ID: docker://387906805acc0fff0f1bbf1e392e886e77c09363bbcce61720db4f316862aaa7
Image: coredns/coredns:1.6.2
Image ID: docker-pullable://coredns/coredns@sha256:12eb885b8685b1b13a04ecf5c23bc809c2e57917252fd7b0be9e9c00644e8ee5
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 03 Jan 2020 17:59:48 +0000
Finished: Fri, 03 Jan 2020 17:59:48 +0000
Ready: False
Restart Count: 7
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/coredns from config-volume (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
QoS Class: Burstable
Node-Selectors: beta.kubernetes.io/os=linux
Tolerations: CriticalAddonsOnly
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned kube-system/coredns-68567cdb47-78x67 to centos
Normal Pulling 14m kubelet, centos Pulling image "coredns/coredns:1.6.2"
Normal Pulled 14m kubelet, centos Successfully pulled image "coredns/coredns:1.6.2"
Normal Created 13m (x5 over 14m) kubelet, centos Created container coredns
Normal Started 13m (x5 over 14m) kubelet, centos Started container coredns
Normal Pulled 13m (x4 over 14m) kubelet, centos Container image "coredns/coredns:1.6.2" already present on machine
Warning BackOff 4m51s (x50 over 14m) kubelet, centos Back-off restarting failed container
Stderr of run.sh:
[kubelet-dockershim] E0103 18:03:48.735562 114 container_manager_linux.go:477] cpu and memory cgroup hierarchy not unified. cpu: /, memory: /user.slice/user-1000.slice/session-1.scope
[kubelet-dockershim] E0103 18:03:48.867398 114 container_manager_linux.go:101] Unable to ensure the docker processes run in the desired containers: errors moving "dockerd" pid: failed to apply oom score -999 to PID 85: write /proc/85/oom_score_adj: permission denied
...
[dockerd] time="2020-01-03T18:04:56.269919903Z" level=info msg="shim containerd-shim started" address=/containerd-shim/6ba264bcbcc738a4686c6b6bbc36cd4c96cbd3a5ff04b2a14b4064f48779d088.sock debug=false pid=7150
[dockerd] time="2020-01-03T18:04:56.903241134Z" level=info msg="shim reaped" id=d023513cf1411c90f83bf1a30d2d55a3e3cde300d59dea05f2ecafea011be2e1
[dockerd] time="2020-01-03T18:04:56.913507529Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[dockerd] time="2020-01-03T18:05:05.238881369Z" level=info msg="shim containerd-shim started" address=/containerd-shim/1a5da1ec06ca697778b35df31d55a6f5befe987f6a82b138b3def79ccab895eb.sock debug=false pid=7269
[dockerd] time="2020-01-03T18:05:05.601143797Z" level=info msg="shim reaped" id=e6eebcd70064eed6d293f505e98c9e6a3d2a682213a281cdfdebecf245b2f3fa
[dockerd] time="2020-01-03T18:05:05.611475434Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
...
[kubelet-dockershim] E0103 18:05:06.218907 114 pod_workers.go:191] Error syncing pod 10011dcf-1fcc-4f90-9692-c27225bfb393 ("coredns-68567cdb47-78x67_kube-system(10011dcf-1fcc-4f90-9692-c27225bfb393)"), skipping: failed to "StartContainer" for "coredns" with CrashLoopBackOff: "back-off 5m0s restarting failed container=coredns pod=coredns-68567cdb47-78x67_kube-system(10011dcf-1fcc-4f90-9692-c27225bfb393)"
[kubelet-dockershim] E0103 18:05:06.595575 114 pod_workers.go:191] Error syncing pod 9ff213d8-36e9-448c-9675-8f344be436fc ("coredns-68567cdb47-xvxnv_kube-system(9ff213d8-36e9-448c-9675-8f344be436fc)"), skipping: failed to "StartContainer" for "coredns" with CrashLoopBackOff: "back-off 5m0s restarting failed container=coredns pod=coredns-68567cdb47-xvxnv_kube-system(9ff213d8-36e9-448c-9675-8f344be436fc)"
[kubelet-dockershim] E0103 18:06:14.151994 114 pod_workers.go:191] Error syncing pod ba5bd6f1-c4af-4f78-93f2-c9a48d126ba9 ("nfs-release-nfs-server-provisioner-0_default(ba5bd6f1-c4af-4f78-93f2-c9a48d126ba9)"), skipping: failed to "StartContainer" for "nfs-server-provisioner" with CrashLoopBackOff: "back-off 5m0s restarting failed container=nfs-server-provisioner pod=nfs-release-nfs-server-provisioner-0_default(ba5bd6f1-c4af-4f78-93f2-c9a48d126ba9)"
dockershim might not be going to support cgroup v2: kubernetes/enhancements#1370 (comment)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.