Comments (6)
from cluster-openshift-apiserver-operator.
I am not sure if this is the same issue; coming here after seeing the issues filed against the installer. My apiserver seems to often fall over after my laptop suspends.
[root@osiris-master-0 ~]# systemctl status kubelet | more
● kubelet.service - Kubernetes Kubelet
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2018-12-04 15:38:07 UTC; 1h 53min ago
Process: 2841 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
Main PID: 2861 (hyperkube)
Memory: 174.0M
CGroup: /system.slice/kubelet.service
└─2861 /usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --rotate-certificates --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-run
time-endpoint=/var/run/crio/crio.sock --allow-privileged --node-labels=node-role.kubernetes.io/master --minimum-container-ttl-duration=6m0s --client-ca-file=/etc/kubernetes/ca.crt --cloud-provider= --anonymous-auth=false --register-with-t
aints=node-role.kubernetes.io/master=:NoSchedule
Dec 04 17:31:14 osiris-master-0 hyperkube[2861]: I1204 17:31:14.666162 2861 kuberuntime_manager.go:771] Back-off 5m0s restarting failed container=operator pod=openshift-cluster-kube-apiserver-operator-758cc84fc-c8gdv_openshift-cluster-
kube-apiserver-operator(3ffff38b-f7e0-11e8-81a8-e275210e488d)
Dec 04 17:31:14 osiris-master-0 hyperkube[2861]: E1204 17:31:14.666187 2861 pod_workers.go:186] Error syncing pod 3ffff38b-f7e0-11e8-81a8-e275210e488d ("openshift-cluster-kube-apiserver-operator-758cc84fc-c8gdv_openshift-cluster-kube-a
piserver-operator(3ffff38b-f7e0-11e8-81a8-e275210e488d)"), skipping: failed to "StartContainer" for "operator" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=operator pod=openshift-cluster-kube-apiserver-operator-758cc8
4fc-c8gdv_openshift-cluster-kube-apiserver-operator(3ffff38b-f7e0-11e8-81a8-e275210e488d)"
Dec 04 17:31:14 osiris-master-0 hyperkube[2861]: I1204 17:31:14.866160 2861 kuberuntime_manager.go:517] Container {Name:openshift-apiserver Image:registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-04-145453@sha256:776be01eae9e
a2cb2213dc9f5747c5b0347d54bfca162533023a5f06f9f65051 Command:[hypershift openshift-apiserver] Args:[--config=/var/run/configmaps/config/config.yaml -v=2] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:8443 Protocol:TCP HostIP:}] EnvFr
om:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:config ReadOnly:false MountPath:/var/run/configmaps/config SubPath: MountPropagation:<nil>} {Name:client-ca ReadOnly:false MountPath:/var/run/configmaps/client-ca Su
bPath: MountPropagation:<nil>} {Name:etcd-serving-ca ReadOnly:false MountPath:/var/run/configmaps/etcd-serving-ca SubPath: MountPropagation:<nil>} {Name:etcd-client ReadOnly:false MountPath:/var/run/secrets/etcd-client SubPath: MountPropa
gation:<nil>} {Name:serving-cert ReadOnly:false MountPath:/var/run/secrets/serving-cert SubPath: MountPropagation:<nil>} {Name:openshift-apiserver-sa-token-49684 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPat
h: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:healthz,Port:8443,Host:,Scheme:HTTPS,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:30,TimeoutSeconds:1,PeriodSec
onds:10,SuccessThreshold:1,FailureThreshold:3,} ReadinessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:healthz,Port:8443,Host:,Scheme:HTTPS,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,Period
Seconds:10,SuccessThreshold:1,FailureThreshold:10,} Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but Res
tartPolicy says that we should restart it.
Dec 04 17:31:14 osiris-master-0 hyperkube[2861]: I1204 17:31:14.868166 2861 kuberuntime_manager.go:761] checking backoff for container "openshift-apiserver" in pod "apiserver-q46rr_openshift-apiserver(c5193131-f7e1-11e8-8e33-e275210e48
8d)"
Dec 04 17:31:14 osiris-master-0 hyperkube[2861]: I1204 17:31:14.868695 2861 kuberuntime_manager.go:771] Back-off 5m0s restarting failed container=openshift-apiserver pod=apiserver-q46rr_openshift-apiserver(c5193131-f7e1-11e8-8e33-e2752
10e488d)
Dec 04 17:31:14 osiris-master-0 hyperkube[2861]: E1204 17:31:14.869029 2861 pod_workers.go:186] Error syncing pod c5193131-f7e1-11e8-8e33-e275210e488d ("apiserver-q46rr_openshift-apiserver(c5193131-f7e1-11e8-8e33-e275210e488d)"), skipp
ing: failed to "StartContainer" for "openshift-apiserver" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=openshift-apiserver pod=apiserver-q46rr_openshift-apiserver(c5193131-f7e1-11e8-8e33-e275210e488d)"
Dec 04 17:31:15 osiris-master-0 hyperkube[2861]: E1204 17:31:15.066411 2861 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://osiris-api.verbum.local:6443/api/v1/services?limit=500&
resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 04 17:31:15 osiris-master-0 hyperkube[2861]: W1204 17:31:15.266843 2861 status_manager.go:482] Failed to get status for pod "openshift-kube-apiserver-osiris-master-0_openshift-kube-apiserver(6c03908aa6103a13eee00d128cf51753)": Get
https://osiris-api.verbum.local:6443/api/v1/namespaces/openshift-kube-apiserver/pods/openshift-kube-apiserver-osiris-master-0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 04 17:31:15 osiris-master-0 hyperkube[2861]: E1204 17:31:15.466767 2861 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://osiris-api.verbum.local:6443/api/v1/pods?fieldSelec
tor=spec.nodeName%3Dosiris-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 04 17:31:15 osiris-master-0 hyperkube[2861]: E1204 17:31:15.666087 2861 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://osiris-api.verbum.local:6443/api/v1/nodes?fieldSelector=me
tadata.name%3Dosiris-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Which I guess isn't very useful; looks like the logs from the previous exited apiserver container have already been lost. Ah wait, here are some logs:
$ crictl logs 7914d68f7eb20
...
F1204 17:42:17.528286 1 storage_decorator.go:57] Unable to create storage backend: config (&{etcd3 openshift.io [https://osiris-etcd-0.verbum.local:2379] /etc/kubernetes/static-pod-resources/secrets/etcd-client/tls.key /etc/kubernetes/static-pod-resources/secrets/etcd-client/tls.crt /etc/kubernetes/static-pod-resources/configmaps/etcd-serving-ca/ca-bundle.crt true false 1000 0xc439d35b90 <nil> 5m0s 1m0s}), err (context deadline exceeded)
E1204 17:42:18.146902 1 authentication.go:62] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Get https://127.0.0.1:6443/api/v1/namespaces/openshift-controller-manager/serviceaccounts/openshift-controller-manager-sa: dial tcp 127.0.0.1:6443: connect: connection refused, token lookup failed, token lookup failed]]]
E1204 17:42:18.180144 1 authentication.go:62] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, [Get https://127.0.0.1:6443/api/v1/namespaces/openshift-core-operators/serviceaccounts/openshift-service-cert-signer-operator: dial tcp 127.0.0.1:6443: connect: connection refused, token lookup failed, token lookup failed]]]
W1204 17:42:18.180926 1 clientconn.go:696] Failed to dial osiris-etcd-0.verbum.local:2379: grpc: the connection is closing; please retry.
I1204 17:42:18.264689 1 trace.go:76] Trace[835951314]: "Create /api/v1/namespaces/openshift-console/events" (started: 2018-12-04 17:42:01.834290415 +0000 UTC m=+81.220315663) (total time: 16.430272705s):
Trace[835951314]: [16.430272705s] [16.41572408s] END
W1204 17:42:18.291590 1 clientconn.go:696] Failed to dial osiris-etcd-0.verbum.local:2379: connection error: desc = "transport: authentication handshake failed: context canceled"; please retry.
E1204 17:42:18.904196 1 controller.go:160] StorageError: key not found, Code: 1, Key: /openshift.io/masterleases/192.168.126.11, ResourceVersion: 0, AdditionalErrorMsg:
E1204 17:42:18.951875 1 watch.go:233] unable to encode watch object: client disconnected (&streaming.encoder{writer:(*framer.lengthDelimitedFrameWriter)(0xc43221f9c0), encoder:(*versioning.codec)(0xc4321e9710), buf:(*bytes.Buffer)(0xc4227fbb90)})
My etcd-member
seems healthy offhand.
from cluster-openshift-apiserver-operator.
Issues go stale after 90d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle stale
.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen
.
If this issue is safe to close now please do so with /close
.
/lifecycle stale
from cluster-openshift-apiserver-operator.
Stale issues rot after 30d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle rotten
.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen
.
If this issue is safe to close now please do so with /close
.
/lifecycle rotten
/remove-lifecycle stale
from cluster-openshift-apiserver-operator.
Rotten issues close after 30d of inactivity.
Reopen the issue by commenting /reopen
.
Mark the issue as fresh by commenting /remove-lifecycle rotten
.
Exclude this issue from closing again by commenting /lifecycle frozen
.
/close
from cluster-openshift-apiserver-operator.
@openshift-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue by commenting
/reopen
.
Mark the issue as fresh by commenting/remove-lifecycle rotten
.
Exclude this issue from closing again by commenting/lifecycle frozen
./close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
from cluster-openshift-apiserver-operator.
Related Issues (14)
- operator does not upgrade apiserver daemonset HOT 2
- Future Release Branches Frozen For Merging | branch:release-4.5 branch:release-4.4
- Future Release Branches Frozen For Merging | branch:release-4.18 branch:release-4.19 HOT 2
- update imageregister defaultRoute will make openshift api restart. HOT 7
- Will openshift-apiserver-operator keep openshift-apiserver unchanged HOT 6
- How to configure custom Audit Policy? HOT 2
- error synchronizing: clusterrole.rbac.authorization.k8s.io "aggregated-apiserver-clusterrole" not found HOT 9
- Return error on RunOperator context done HOT 5
- Access to a privileged container allows for breakout to the underlying host HOT 5
- Future Release Branches Frozen For Merging | branch:release-4.16 branch:release-4.17
- Operator fails with 'Generation: expected: 3, actual: 5' HOT 11
- Move TestingEventRecorder to a subpackage under spot where the in memory one is HOT 1
- Need invariant checker for all e2e tests HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cluster-openshift-apiserver-operator.