Apiserver startup takes 50+ seconds about cluster-openshift-apiserver-operator HOT 6 CLOSED

[root@osiris-master-0 ~]# systemctl status kubelet | more
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-12-04 15:38:07 UTC; 1h 53min ago
  Process: 2841 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
 Main PID: 2861 (hyperkube)
   Memory: 174.0M
   CGroup: /system.slice/kubelet.service
           └─2861 /usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --rotate-certificates --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-run
time-endpoint=/var/run/crio/crio.sock --allow-privileged --node-labels=node-role.kubernetes.io/master --minimum-container-ttl-duration=6m0s --client-ca-file=/etc/kubernetes/ca.crt --cloud-provider= --anonymous-auth=false --register-with-t
aints=node-role.kubernetes.io/master=:NoSchedule

Dec 04 17:31:14 osiris-master-0 hyperkube[2861]: I1204 17:31:14.666162    2861 kuberuntime_manager.go:771] Back-off 5m0s restarting failed container=operator pod=openshift-cluster-kube-apiserver-operator-758cc84fc-c8gdv_openshift-cluster-
kube-apiserver-operator(3ffff38b-f7e0-11e8-81a8-e275210e488d)
Dec 04 17:31:14 osiris-master-0 hyperkube[2861]: E1204 17:31:14.666187    2861 pod_workers.go:186] Error syncing pod 3ffff38b-f7e0-11e8-81a8-e275210e488d ("openshift-cluster-kube-apiserver-operator-758cc84fc-c8gdv_openshift-cluster-kube-a
piserver-operator(3ffff38b-f7e0-11e8-81a8-e275210e488d)"), skipping: failed to "StartContainer" for "operator" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=operator pod=openshift-cluster-kube-apiserver-operator-758cc8
4fc-c8gdv_openshift-cluster-kube-apiserver-operator(3ffff38b-f7e0-11e8-81a8-e275210e488d)"
Dec 04 17:31:14 osiris-master-0 hyperkube[2861]: I1204 17:31:14.866160    2861 kuberuntime_manager.go:517] Container {Name:openshift-apiserver Image:registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-04-145453@sha256:776be01eae9e
a2cb2213dc9f5747c5b0347d54bfca162533023a5f06f9f65051 Command:[hypershift openshift-apiserver] Args:[--config=/var/run/configmaps/config/config.yaml -v=2] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:8443 Protocol:TCP HostIP:}] EnvFr
om:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:config ReadOnly:false MountPath:/var/run/configmaps/config SubPath: MountPropagation:<nil>} {Name:client-ca ReadOnly:false MountPath:/var/run/configmaps/client-ca Su
bPath: MountPropagation:<nil>} {Name:etcd-serving-ca ReadOnly:false MountPath:/var/run/configmaps/etcd-serving-ca SubPath: MountPropagation:<nil>} {Name:etcd-client ReadOnly:false MountPath:/var/run/secrets/etcd-client SubPath: MountPropa
gation:<nil>} {Name:serving-cert ReadOnly:false MountPath:/var/run/secrets/serving-cert SubPath: MountPropagation:<nil>} {Name:openshift-apiserver-sa-token-49684 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPat
h: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:healthz,Port:8443,Host:,Scheme:HTTPS,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:30,TimeoutSeconds:1,PeriodSec
onds:10,SuccessThreshold:1,FailureThreshold:3,} ReadinessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:healthz,Port:8443,Host:,Scheme:HTTPS,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,Period
Seconds:10,SuccessThreshold:1,FailureThreshold:10,} Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but Res
tartPolicy says that we should restart it.
Dec 04 17:31:14 osiris-master-0 hyperkube[2861]: I1204 17:31:14.868166    2861 kuberuntime_manager.go:761] checking backoff for container "openshift-apiserver" in pod "apiserver-q46rr_openshift-apiserver(c5193131-f7e1-11e8-8e33-e275210e48
8d)"
Dec 04 17:31:14 osiris-master-0 hyperkube[2861]: I1204 17:31:14.868695    2861 kuberuntime_manager.go:771] Back-off 5m0s restarting failed container=openshift-apiserver pod=apiserver-q46rr_openshift-apiserver(c5193131-f7e1-11e8-8e33-e2752
10e488d)
Dec 04 17:31:14 osiris-master-0 hyperkube[2861]: E1204 17:31:14.869029    2861 pod_workers.go:186] Error syncing pod c5193131-f7e1-11e8-8e33-e275210e488d ("apiserver-q46rr_openshift-apiserver(c5193131-f7e1-11e8-8e33-e275210e488d)"), skipp
ing: failed to "StartContainer" for "openshift-apiserver" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=openshift-apiserver pod=apiserver-q46rr_openshift-apiserver(c5193131-f7e1-11e8-8e33-e275210e488d)"
Dec 04 17:31:15 osiris-master-0 hyperkube[2861]: E1204 17:31:15.066411    2861 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://osiris-api.verbum.local:6443/api/v1/services?limit=500&
resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 04 17:31:15 osiris-master-0 hyperkube[2861]: W1204 17:31:15.266843    2861 status_manager.go:482] Failed to get status for pod "openshift-kube-apiserver-osiris-master-0_openshift-kube-apiserver(6c03908aa6103a13eee00d128cf51753)": Get 
https://osiris-api.verbum.local:6443/api/v1/namespaces/openshift-kube-apiserver/pods/openshift-kube-apiserver-osiris-master-0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 04 17:31:15 osiris-master-0 hyperkube[2861]: E1204 17:31:15.466767    2861 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://osiris-api.verbum.local:6443/api/v1/pods?fieldSelec
tor=spec.nodeName%3Dosiris-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 04 17:31:15 osiris-master-0 hyperkube[2861]: E1204 17:31:15.666087    2861 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://osiris-api.verbum.local:6443/api/v1/nodes?fieldSelector=me
tadata.name%3Dosiris-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused

$ crictl logs 7914d68f7eb20
...
F1204 17:42:17.528286       1 storage_decorator.go:57] Unable to create storage backend: config (&{etcd3 openshift.io [https://osiris-etcd-0.verbum.local:2379] /etc/kubernetes/static-pod-resources/secrets/etcd-client/tls.key /etc/kubernetes/static-pod-resources/secrets/etcd-client/tls.crt /etc/kubernetes/static-pod-resources/configmaps/etcd-serving-ca/ca-bundle.crt true false 1000 0xc439d35b90 <nil> 5m0s 1m0s}), err (context deadline exceeded)
E1204 17:42:18.146902       1 authentication.go:62] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Get https://127.0.0.1:6443/api/v1/namespaces/openshift-controller-manager/serviceaccounts/openshift-controller-manager-sa: dial tcp 127.0.0.1:6443: connect: connection refused, token lookup failed, token lookup failed]]]
E1204 17:42:18.180144       1 authentication.go:62] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Get https://127.0.0.1:6443/api/v1/namespaces/openshift-core-operators/serviceaccounts/openshift-service-cert-signer-operator: dial tcp 127.0.0.1:6443: connect: connection refused, token lookup failed, token lookup failed]]]
W1204 17:42:18.180926       1 clientconn.go:696] Failed to dial osiris-etcd-0.verbum.local:2379: grpc: the connection is closing; please retry.
I1204 17:42:18.264689       1 trace.go:76] Trace[835951314]: "Create /api/v1/namespaces/openshift-console/events" (started: 2018-12-04 17:42:01.834290415 +0000 UTC m=+81.220315663) (total time: 16.430272705s):
Trace[835951314]: [16.430272705s] [16.41572408s] END
W1204 17:42:18.291590       1 clientconn.go:696] Failed to dial osiris-etcd-0.verbum.local:2379: connection error: desc = "transport: authentication handshake failed: context canceled"; please retry.
E1204 17:42:18.904196       1 controller.go:160] StorageError: key not found, Code: 1, Key: /openshift.io/masterleases/192.168.126.11, ResourceVersion: 0, AdditionalErrorMsg: 
E1204 17:42:18.951875       1 watch.go:233] unable to encode watch object: client disconnected (&streaming.encoder{writer:(*framer.lengthDelimitedFrameWriter)(0xc43221f9c0), encoder:(*versioning.codec)(0xc4321e9710), buf:(*bytes.Buffer)(0xc4227fbb90)})