openshift-metal3 / dev-scripts Goto Github PK
View Code? Open in Web Editor NEWScripts to automate development/test setup for openshift integration with https://github.com/metal3-io/
License: Apache License 2.0
Scripts to automate development/test setup for openshift integration with https://github.com/metal3-io/
License: Apache License 2.0
Running make
failed because of this:
level=debug msg="Still waiting for the Kubernetes API: Get https://api.ostest.test.metalkube.org:6443/version?timeout=32s: dial tcp 192.168.111.5:6443: connect: connection refused"
level=debug msg="Still waiting for the Kubernetes API: Get https://api.ostest.test.metalkube.org:6443/version?timeout=32s: dial tcp 192.168.111.5:6443: connect: connection refused"
level=debug msg="Still waiting for the Kubernetes API: Get https://api.ostest.test.metalkube.org:6443/version?timeout=32s: dial tcp 192.168.111.5:6443: connect: connection refused"
I have ssh-ed to the bootstrap node and found this in the logs:
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: Flag --runtime-request-timeout has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: Flag --allow-privileged has been deprecated, will be removed in a future version
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: Flag --serialize-image-pulls has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.169640 3349 flags.go:33] FLAG: --address="0.0.0.0"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.169967 3349 flags.go:33] FLAG: --allow-privileged="true"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.169978 3349 flags.go:33] FLAG: --allowed-unsafe-sysctls="[]"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.169996 3349 flags.go:33] FLAG: --alsologtostderr="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170006 3349 flags.go:33] FLAG: --anonymous-auth="true"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170014 3349 flags.go:33] FLAG: --application-metrics-count-limit="100"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170035 3349 flags.go:33] FLAG: --authentication-token-webhook="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170044 3349 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170055 3349 flags.go:33] FLAG: --authorization-mode="AlwaysAllow"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170065 3349 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170073 3349 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170081 3349 flags.go:33] FLAG: --azure-container-registry-config=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170089 3349 flags.go:33] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170128 3349 flags.go:33] FLAG: --bootstrap-checkpoint-path=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170137 3349 flags.go:33] FLAG: --bootstrap-kubeconfig=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170144 3349 flags.go:33] FLAG: --cert-dir="/var/lib/kubelet/pki"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170153 3349 flags.go:33] FLAG: --cgroup-driver="systemd"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170160 3349 flags.go:33] FLAG: --cgroup-root=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170168 3349 flags.go:33] FLAG: --cgroups-per-qos="true"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170176 3349 flags.go:33] FLAG: --chaos-chance="0"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170187 3349 flags.go:33] FLAG: --client-ca-file=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170195 3349 flags.go:33] FLAG: --cloud-config=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170202 3349 flags.go:33] FLAG: --cloud-provider=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170210 3349 flags.go:33] FLAG: --cluster-dns="[]"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170225 3349 flags.go:33] FLAG: --cluster-domain="cluster.local"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170233 3349 flags.go:33] FLAG: --cni-bin-dir="/opt/cni/bin"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170241 3349 flags.go:33] FLAG: --cni-conf-dir="/etc/cni/net.d"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170249 3349 flags.go:33] FLAG: --config=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170257 3349 flags.go:33] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170268 3349 flags.go:33] FLAG: --container-log-max-files="5"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170278 3349 flags.go:33] FLAG: --container-log-max-size="10Mi"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170286 3349 flags.go:33] FLAG: --container-runtime="remote"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170293 3349 flags.go:33] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170315 3349 flags.go:33] FLAG: --containerd="unix:///var/run/containerd.sock"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170324 3349 flags.go:33] FLAG: --containerized="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170332 3349 flags.go:33] FLAG: --contention-profiling="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170340 3349 flags.go:33] FLAG: --cpu-cfs-quota="true"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170347 3349 flags.go:33] FLAG: --cpu-cfs-quota-period="100ms"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170355 3349 flags.go:33] FLAG: --cpu-manager-policy="none"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170363 3349 flags.go:33] FLAG: --cpu-manager-reconcile-period="10s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170371 3349 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170379 3349 flags.go:33] FLAG: --docker-endpoint="unix:///var/run/docker.sock"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170387 3349 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170409 3349 flags.go:33] FLAG: --docker-only="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170417 3349 flags.go:33] FLAG: --docker-root="/var/lib/docker"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170426 3349 flags.go:33] FLAG: --docker-tls="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170434 3349 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170441 3349 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170449 3349 flags.go:33] FLAG: --docker-tls-key="key.pem"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170456 3349 flags.go:33] FLAG: --dynamic-config-dir=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170466 3349 flags.go:33] FLAG: --enable-controller-attach-detach="true"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170473 3349 flags.go:33] FLAG: --enable-debugging-handlers="true"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170481 3349 flags.go:33] FLAG: --enable-load-reader="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170488 3349 flags.go:33] FLAG: --enable-server="true"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170496 3349 flags.go:33] FLAG: --enforce-node-allocatable="[pods]"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170512 3349 flags.go:33] FLAG: --event-burst="10"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170520 3349 flags.go:33] FLAG: --event-qps="5"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170527 3349 flags.go:33] FLAG: --event-storage-age-limit="default=0"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170536 3349 flags.go:33] FLAG: --event-storage-event-limit="default=0"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170543 3349 flags.go:33] FLAG: --eviction-hard="imagefs.available<15%,memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170566 3349 flags.go:33] FLAG: --eviction-max-pod-grace-period="0"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170576 3349 flags.go:33] FLAG: --eviction-minimum-reclaim=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170586 3349 flags.go:33] FLAG: --eviction-pressure-transition-period="5m0s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170594 3349 flags.go:33] FLAG: --eviction-soft=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170601 3349 flags.go:33] FLAG: --eviction-soft-grace-period=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170609 3349 flags.go:33] FLAG: --exit-on-lock-contention="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170616 3349 flags.go:33] FLAG: --experimental-allocatable-ignore-eviction="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170623 3349 flags.go:33] FLAG: --experimental-bootstrap-kubeconfig=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170630 3349 flags.go:33] FLAG: --experimental-check-node-capabilities-before-mount="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170637 3349 flags.go:33] FLAG: --experimental-dockershim="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170644 3349 flags.go:33] FLAG: --experimental-dockershim-root-directory="/var/lib/dockershim"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170652 3349 flags.go:33] FLAG: --experimental-fail-swap-on="true"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170659 3349 flags.go:33] FLAG: --experimental-kernel-memcg-notification="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170666 3349 flags.go:33] FLAG: --experimental-mounter-path=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170673 3349 flags.go:33] FLAG: --fail-swap-on="true"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170680 3349 flags.go:33] FLAG: --feature-gates=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170690 3349 flags.go:33] FLAG: --file-check-frequency="20s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170700 3349 flags.go:33] FLAG: --global-housekeeping-interval="1m0s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170708 3349 flags.go:33] FLAG: --google-json-key=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170715 3349 flags.go:33] FLAG: --hairpin-mode="promiscuous-bridge"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170723 3349 flags.go:33] FLAG: --healthz-bind-address="127.0.0.1"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170730 3349 flags.go:33] FLAG: --healthz-port="10248"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170738 3349 flags.go:33] FLAG: --help="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170745 3349 flags.go:33] FLAG: --host-ipc-sources="[*]"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170758 3349 flags.go:33] FLAG: --host-network-sources="[*]"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170771 3349 flags.go:33] FLAG: --host-pid-sources="[*]"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170782 3349 flags.go:33] FLAG: --hostname-override=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170790 3349 flags.go:33] FLAG: --housekeeping-interval="10s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170798 3349 flags.go:33] FLAG: --http-check-frequency="20s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170805 3349 flags.go:33] FLAG: --image-gc-high-threshold="85"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170812 3349 flags.go:33] FLAG: --image-gc-low-threshold="80"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170819 3349 flags.go:33] FLAG: --image-pull-progress-deadline="1m0s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170827 3349 flags.go:33] FLAG: --image-service-endpoint=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170836 3349 flags.go:33] FLAG: --iptables-drop-bit="15"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170844 3349 flags.go:33] FLAG: --iptables-masquerade-bit="14"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170851 3349 flags.go:33] FLAG: --keep-terminated-pod-volumes="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170858 3349 flags.go:33] FLAG: --kube-api-burst="10"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170865 3349 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170873 3349 flags.go:33] FLAG: --kube-api-qps="5"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170880 3349 flags.go:33] FLAG: --kube-reserved=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170888 3349 flags.go:33] FLAG: --kube-reserved-cgroup=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170895 3349 flags.go:33] FLAG: --kubeconfig=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170902 3349 flags.go:33] FLAG: --kubelet-cgroups=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170910 3349 flags.go:33] FLAG: --lock-file=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170917 3349 flags.go:33] FLAG: --log-backtrace-at=":0"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170925 3349 flags.go:33] FLAG: --log-cadvisor-usage="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170933 3349 flags.go:33] FLAG: --log-dir=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170939 3349 flags.go:33] FLAG: --log-flush-frequency="5s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170947 3349 flags.go:33] FLAG: --logtostderr="true"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170956 3349 flags.go:33] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170965 3349 flags.go:33] FLAG: --make-iptables-util-chains="true"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170972 3349 flags.go:33] FLAG: --manifest-url=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170979 3349 flags.go:33] FLAG: --manifest-url-header=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170990 3349 flags.go:33] FLAG: --master-service-namespace="default"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.170998 3349 flags.go:33] FLAG: --max-open-files="1000000"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171008 3349 flags.go:33] FLAG: --max-pods="110"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171015 3349 flags.go:33] FLAG: --maximum-dead-containers="-1"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171023 3349 flags.go:33] FLAG: --maximum-dead-containers-per-container="1"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171030 3349 flags.go:33] FLAG: --minimum-container-ttl-duration="6m0s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171037 3349 flags.go:33] FLAG: --minimum-image-ttl-duration="2m0s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171045 3349 flags.go:33] FLAG: --network-plugin=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171051 3349 flags.go:33] FLAG: --network-plugin-mtu="0"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171058 3349 flags.go:33] FLAG: --node-ip=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171065 3349 flags.go:33] FLAG: --node-labels=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171075 3349 flags.go:33] FLAG: --node-status-max-images="50"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171084 3349 flags.go:33] FLAG: --node-status-update-frequency="10s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171108 3349 flags.go:33] FLAG: --non-masquerade-cidr="10.0.0.0/8"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171119 3349 flags.go:33] FLAG: --oom-score-adj="-999"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171127 3349 flags.go:33] FLAG: --pod-cidr=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171133 3349 flags.go:33] FLAG: --pod-infra-container-image="k8s.gcr.io/pause:3.1"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171141 3349 flags.go:33] FLAG: --pod-manifest-path="/etc/kubernetes/manifests"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171149 3349 flags.go:33] FLAG: --pod-max-pids="-1"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171157 3349 flags.go:33] FLAG: --pods-per-core="0"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171164 3349 flags.go:33] FLAG: --port="10250"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171172 3349 flags.go:33] FLAG: --protect-kernel-defaults="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171180 3349 flags.go:33] FLAG: --provider-id=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171188 3349 flags.go:33] FLAG: --qos-reserved=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171195 3349 flags.go:33] FLAG: --read-only-port="10255"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171203 3349 flags.go:33] FLAG: --really-crash-for-testing="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171210 3349 flags.go:33] FLAG: --redirect-container-streaming="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171217 3349 flags.go:33] FLAG: --register-node="true"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171225 3349 flags.go:33] FLAG: --register-schedulable="true"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171233 3349 flags.go:33] FLAG: --register-with-taints=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171242 3349 flags.go:33] FLAG: --registry-burst="10"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171249 3349 flags.go:33] FLAG: --registry-qps="5"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171256 3349 flags.go:33] FLAG: --resolv-conf="/etc/resolv.conf"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171264 3349 flags.go:33] FLAG: --root-dir="/var/lib/kubelet"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171271 3349 flags.go:33] FLAG: --rotate-certificates="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171278 3349 flags.go:33] FLAG: --rotate-server-certificates="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171285 3349 flags.go:33] FLAG: --runonce="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171292 3349 flags.go:33] FLAG: --runtime-cgroups=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171314 3349 flags.go:33] FLAG: --runtime-request-timeout="10m0s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171322 3349 flags.go:33] FLAG: --seccomp-profile-root="/var/lib/kubelet/seccomp"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171330 3349 flags.go:33] FLAG: --serialize-image-pulls="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171337 3349 flags.go:33] FLAG: --stderrthreshold="2"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171345 3349 flags.go:33] FLAG: --storage-driver-buffer-duration="1m0s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171352 3349 flags.go:33] FLAG: --storage-driver-db="cadvisor"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171362 3349 flags.go:33] FLAG: --storage-driver-host="localhost:8086"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171369 3349 flags.go:33] FLAG: --storage-driver-password="root"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171376 3349 flags.go:33] FLAG: --storage-driver-secure="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171384 3349 flags.go:33] FLAG: --storage-driver-table="stats"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171391 3349 flags.go:33] FLAG: --storage-driver-user="root"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171398 3349 flags.go:33] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171405 3349 flags.go:33] FLAG: --sync-frequency="1m0s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171413 3349 flags.go:33] FLAG: --system-cgroups=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171420 3349 flags.go:33] FLAG: --system-reserved=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171427 3349 flags.go:33] FLAG: --system-reserved-cgroup=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171436 3349 flags.go:33] FLAG: --tls-cert-file=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171443 3349 flags.go:33] FLAG: --tls-cipher-suites="[]"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171460 3349 flags.go:33] FLAG: --tls-min-version=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171467 3349 flags.go:33] FLAG: --tls-private-key-file=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171474 3349 flags.go:33] FLAG: --v="2"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171482 3349 flags.go:33] FLAG: --version="false"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171495 3349 flags.go:33] FLAG: --vmodule=""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171502 3349 flags.go:33] FLAG: --volume-plugin-dir="/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171510 3349 flags.go:33] FLAG: --volume-stats-agg-period="1m0s"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171551 3349 feature_gate.go:206] feature gates: &{map[]}
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.171613 3349 feature_gate.go:206] feature gates: &{map[]}
Apr 04 09:09:32 localhost.localdomain systemd[1]: Started Kubernetes systemd probe.
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.684901 3349 mount_linux.go:179] Detected OS with systemd
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.685110 3349 server.go:415] Version: v1.12.4+2a194a0f02
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.685195 3349 feature_gate.go:206] feature gates: &{map[]}
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.685276 3349 feature_gate.go:206] feature gates: &{map[]}
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.685472 3349 plugins.go:99] No cloud provider specified.
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.685490 3349 server.go:531] No cloud provider specified: "" from the config file: ""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: W0404 09:09:32.685632 3349 server.go:554] standalone mode, no API client
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.687831 3349 manager.go:155] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct/system.slice/kubelet.service"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.692798 3349 fs.go:142] Filesystem UUIDs: map[125df5c3-f244-4ce5-ae4e-562e2ccc3b3a:/dev/vda2 dd7e4c4e-77dc-44de-8559-e1c8f77541e8:/dev/vda1]
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.692819 3349 fs.go:143] Filesystem partitions: map[tmpfs:{mountpoint:/dev/shm major:0 minor:19 fsType:tmpfs blockSize:0} /dev/vda2:{mountpoint:/var major:252 minor:2 fsType:xfs blockSize:0} /dev/vda1:{mountpoint:/boot major:252 minor:1 fsType:xfs blockSize:0}]
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.698649 3349 manager.go:229] Machine: {NumCores:4 CpuFrequency:1999999 MemoryCapacity:4142034944 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:c49cdfb46f024e0a99cf775dadd55056 SystemUUID:C49CDFB4-6F02-4E0A-99CF-775DADD55056 BootID:bcf913e9-f83f-48f2-a067-4892b26be5b8 Filesystems:[{Device:tmpfs DeviceMajor:0 DeviceMinor:19 Capacity:2071015424 Type:vfs Inodes:505619 HasI
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.699868 3349 manager.go:235] Version: {KernelVersion:3.10.0-957.5.1.el7.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 400.7.20190306.0 DockerVersion:Unknown DockerAPIVersion:Unknown CadvisorVersion: CadvisorRevision:}
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: W0404 09:09:32.699966 3349 server.go:472] No api server defined - no events will be sent to API server.
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.699980 3349 server.go:634] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.700431 3349 container_manager_linux.go:247] container manager verified user specified cgroup-root exists: []
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.700452 3349 container_manager_linux.go:252] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.700574 3349 container_manager_linux.go:271] Creating device plugin manager: true
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.700595 3349 manager.go:108] Creating Device Plugin manager at /var/lib/kubelet/device-plugins/kubelet.sock
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.700783 3349 state_mem.go:36] [cpumanager] initializing new in-memory state store
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.702985 3349 server.go:989] Using root directory: /var/lib/kubelet
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.703480 3349 kubelet.go:279] Adding pod path: /etc/kubernetes/manifests
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.703934 3349 file.go:68] Watching path "/etc/kubernetes/manifests"
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: W0404 09:09:32.710117 3349 util_unix.go:75] Using "/var/run/crio/crio.sock" as endpoint is deprecated, please consider using full url format "unix:///var/run/crio/crio.sock".
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.711264 3349 resolver_conn_wrapper.go:64] dialing to target with scheme: ""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.711305 3349 resolver_conn_wrapper.go:70] could not get resolver for scheme: ""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: W0404 09:09:32.711328 3349 util_unix.go:75] Using "/var/run/crio/crio.sock" as endpoint is deprecated, please consider using full url format "unix:///var/run/crio/crio.sock".
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.711364 3349 resolver_conn_wrapper.go:64] dialing to target with scheme: ""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: I0404 09:09:32.711376 3349 resolver_conn_wrapper.go:70] could not get resolver for scheme: ""
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: W0404 09:09:32.712239 3349 clientconn.go:944] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial unix /var/run/crio/crio.sock: connect: no such file or directory"; Reconnecting to {/var/run/crio/crio.sock 0 <nil>}
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: W0404 09:09:32.712568 3349 clientconn.go:944] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial unix /var/run/crio/crio.sock: connect: no such file or directory"; Reconnecting to {/var/run/crio/crio.sock 0 <nil>}
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: E0404 09:09:32.712714 3349 remote_runtime.go:72] Version from runtime service failed: rpc error: code = Unavailable desc = grpc: the connection is unavailable
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: E0404 09:09:32.712787 3349 kuberuntime_manager.go:183] Get runtime version failed: rpc error: code = Unavailable desc = grpc: the connection is unavailable
Apr 04 09:09:32 localhost.localdomain hyperkube[3349]: F0404 09:09:32.712806 3349 server.go:262] failed to run Kubelet: failed to create kubelet: rpc error: code = Unavailable desc = grpc: the connection is unavailable
Apr 04 09:09:32 localhost.localdomain systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Apr 04 09:09:32 localhost.localdomain systemd[1]: Failed to start Kubernetes Kubelet.
Apr 04 09:09:32 localhost.localdomain systemd[1]: Unit kubelet.service entered failed state.
Apr 04 09:09:32 localhost.localdomain systemd[1]: kubelet.service failed.
Apr 04 09:09:32 localhost.localdomain systemd[1]: Starting Preprocess NFS configuration...
Apr 04 09:09:32 localhost.localdomain systemd[1]: Started Bootstrap a Kubernetes cluster.
Apr 04 09:09:32 localhost.localdomain systemd[1]: Started Bootstrap an OpenShift cluster.
[core@master-1 ~]$ source /etc/kubernetes/static-pod-resources/clusterrc
-bash: /etc/kubernetes/static-pod-resources/clusterrc: No such file or directory
[core@master-1 ~]$ source /etc/kubernetes/static-pod-resources/mdns/clusterrc
[core@master-1 ~]$
[centos@provisionhost-0 dev-scripts]$ ./10_deploy_rook.sh | awk '{ print strftime("%Y-%m-%d %H:%M:%S |"), $0; fflush(); }'
+ source logging.sh
+++ dirname ./10_deploy_rook.sh
++ LOGDIR=./logs
++ '[' '!' -d ./logs ']'
+++ basename ./10_deploy_rook.sh .sh
+++ date +%F-%H%M%S
++ LOGFILE=./logs/10_deploy_rook-2019-04-13-175424.log
++ echo 'Logging to ./logs/10_deploy_rook-2019-04-13-175424.log'
++ exec
2019-04-13 17:54:24 | Logging to ./logs/10_deploy_rook-2019-04-13-175424.log
+++ tee ./logs/10_deploy_rook-2019-04-13-175424.log
2019-04-13 17:54:24 | + source common.sh
2019-04-13 17:54:24 | ++++ dirname common.sh
2019-04-13 17:54:24 | +++ cd .
2019-04-13 17:54:24 | +++ pwd
2019-04-13 17:54:24 | ++ SCRIPTDIR=/home/centos/dev-scripts
2019-04-13 17:54:24 | +++ whoami
2019-04-13 17:54:24 | ++ USER=centos
2019-04-13 17:54:24 | ++ '[' -z '' ']'
2019-04-13 17:54:24 | ++ '[' -f /home/centos/dev-scripts/config_centos.sh ']'
2019-04-13 17:54:24 | ++ echo 'Using CONFIG /home/centos/dev-scripts/config_centos.sh'
2019-04-13 17:54:24 | Using CONFIG /home/centos/dev-scripts/config_centos.sh
2019-04-13 17:54:24 | ++ CONFIG=/home/centos/dev-scripts/config_centos.sh
2019-04-13 17:54:24 | ++ source /home/centos/dev-scripts/config_centos.sh
2019-04-13 17:54:24 | +++ set +x
2019-04-13 17:54:24 | +++ BOOTSTRAP_SSH_READY=2500
2019-04-13 17:54:24 | +++ NODES_PLATFORM=baremetal
2019-04-13 17:54:24 | +++ INT_IF=eth0
2019-04-13 17:54:24 | +++ PRO_IF=eth1
2019-04-13 17:54:24 | +++ ROOT_DISK=/dev/vda
2019-04-13 17:54:24 | +++ NODES_FILE=/home/centos/instackenv.json
2019-04-13 17:54:24 | +++ MANAGE_BR_BRIDGE=n
2019-04-13 17:54:24 | ++ ADDN_DNS=
2019-04-13 17:54:24 | ++ EXT_IF=
2019-04-13 17:54:24 | ++ PRO_IF=eth1
2019-04-13 17:54:24 | ++ MANAGE_BR_BRIDGE=n
2019-04-13 17:54:24 | ++ MANAGE_PRO_BRIDGE=y
2019-04-13 17:54:24 | ++ MANAGE_INT_BRIDGE=y
2019-04-13 17:54:24 | ++ INT_IF=eth0
2019-04-13 17:54:24 | ++ ROOT_DISK=/dev/vda
2019-04-13 17:54:24 | ++ KAFKA_NAMESPACE=strimzi
2019-04-13 17:54:24 | ++ KAFKA_CLUSTERNAME=strimzi
2019-04-13 17:54:24 | ++ KAFKA_PVC_SIZE=10
2019-04-13 17:54:24 | ++ KAFKA_PRODUCER_TIMER=100
2019-04-13 17:54:24 | ++ KAFKA_PRODUCER_TOPIC=strimzi-topic
2019-04-13 17:54:24 | ++ FILESYSTEM=/
2019-04-13 17:54:24 | ++ WORKING_DIR=/opt/dev-scripts
2019-04-13 17:54:24 | ++ NODES_FILE=/home/centos/instackenv.json
2019-04-13 17:54:24 | ++ NODES_PLATFORM=baremetal
2019-04-13 17:54:24 | ++ MASTER_NODES_FILE=ocp/master_nodes.json
2019-04-13 17:54:24 | ++ export RHCOS_IMAGE_URL=https://releases-rhcos.svc.ci.openshift.org/storage/releases/ootpa/
2019-04-13 17:54:24 | ++ RHCOS_IMAGE_URL=https://releases-rhcos.svc.ci.openshift.org/storage/releases/ootpa/
2019-04-13 17:54:24 | +++ jq -r '.builds[1]'
2019-04-13 17:54:24 | +++ curl https://releases-rhcos.svc.ci.openshift.org/storage/releases/ootpa//builds.json
2019-04-13 17:54:24 | % Total % Received % Xferd Average Speed Time Time Time Current
2019-04-13 17:54:24 | Dload Upload Total Spent Left Speed
100 737 100 737 0 0 1032 0 --:--:-- --:--:-- --:--:-- 1033
2019-04-13 17:54:25 | ++ export RHCOS_LATEST=410.8.20190412.2
2019-04-13 17:54:25 | ++ RHCOS_LATEST=410.8.20190412.2
2019-04-13 17:54:25 | ++ export RHCOS_IMAGE_VERSION=410.8.20190412.2
2019-04-13 17:54:25 | ++ RHCOS_IMAGE_VERSION=410.8.20190412.2
2019-04-13 17:54:25 | +++ curl https://releases-rhcos.svc.ci.openshift.org/storage/releases/ootpa//410.8.20190412.2/meta.json
2019-04-13 17:54:25 | +++ jq -r .images.openstack.path
2019-04-13 17:54:25 | % Total % Received % Xferd Average Speed Time Time Time Current
2019-04-13 17:54:25 | Dload Upload Total Spent Left Speed
100 4778 100 4778 0 0 10485 0 --:--:-- --:--:-- --:--:-- 10478
2019-04-13 17:54:25 | ++ export RHCOS_IMAGE_FILENAME_OPENSTACK_GZ=rhcos-410.8.20190412.2-openstack.qcow2
2019-04-13 17:54:25 | ++ RHCOS_IMAGE_FILENAME_OPENSTACK_GZ=rhcos-410.8.20190412.2-openstack.qcow2
2019-04-13 17:54:25 | +++ echo rhcos-410.8.20190412.2-openstack.qcow2
2019-04-13 17:54:25 | +++ sed -e 's/-openstack.*//'
2019-04-13 17:54:25 | ++ export RHCOS_IMAGE_NAME=rhcos-410.8.20190412.2
2019-04-13 17:54:25 | ++ RHCOS_IMAGE_NAME=rhcos-410.8.20190412.2
2019-04-13 17:54:25 | ++ export RHCOS_IMAGE_FILENAME_OPENSTACK=rhcos-410.8.20190412.2-openstack.qcow2
2019-04-13 17:54:25 | ++ RHCOS_IMAGE_FILENAME_OPENSTACK=rhcos-410.8.20190412.2-openstack.qcow2
2019-04-13 17:54:25 | ++ export RHCOS_IMAGE_FILENAME_COMPRESSED=rhcos-410.8.20190412.2-compressed.qcow2
2019-04-13 17:54:25 | ++ RHCOS_IMAGE_FILENAME_COMPRESSED=rhcos-410.8.20190412.2-compressed.qcow2
2019-04-13 17:54:25 | ++ export RHCOS_IMAGE_FILENAME_LATEST=rhcos-ootpa-latest.qcow2
2019-04-13 17:54:25 | ++ RHCOS_IMAGE_FILENAME_LATEST=rhcos-ootpa-latest.qcow2
2019-04-13 17:54:25 | ++ export IRONIC_IMAGE=quay.io/metalkube/metalkube-ironic
2019-04-13 17:54:25 | ++ IRONIC_IMAGE=quay.io/metalkube/metalkube-ironic
2019-04-13 17:54:25 | ++ export IRONIC_INSPECTOR_IMAGE=quay.io/metalkube/metalkube-ironic-inspector
2019-04-13 17:54:25 | ++ IRONIC_INSPECTOR_IMAGE=quay.io/metalkube/metalkube-ironic-inspector
2019-04-13 17:54:25 | ++ export IRONIC_DATA_DIR=/opt/dev-scripts/ironic
2019-04-13 17:54:25 | ++ IRONIC_DATA_DIR=/opt/dev-scripts/ironic
2019-04-13 17:54:25 | ++ export KUBECONFIG=/home/centos/dev-scripts/ocp/auth/kubeconfig
2019-04-13 17:54:25 | ++ KUBECONFIG=/home/centos/dev-scripts/ocp/auth/kubeconfig
2019-04-13 17:54:25 | ++ export 'SSH=ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=5'
2019-04-13 17:54:25 | ++ SSH='ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=5'
2019-04-13 17:54:25 | ++ export LIBVIRT_DEFAULT_URI=qemu:///system
2019-04-13 17:54:25 | ++ LIBVIRT_DEFAULT_URI=qemu:///system
2019-04-13 17:54:25 | ++ '[' centos '!=' root -a /run/user/1000 == /run/user/0 ']'
2019-04-13 17:54:25 | ++ sudo -n uptime
2019-04-13 17:54:25 | +++ awk -F= '/^ID=/ { print $2 }' /etc/os-release
2019-04-13 17:54:25 | +++ tr -d '"'
2019-04-13 17:54:25 | ++ [[ ! centos =~ ^(centos|rhel)$ ]]
2019-04-13 17:54:25 | +++ awk -F= '/^VERSION_ID=/ { print $2 }' /etc/os-release
2019-04-13 17:54:25 | +++ tr -d '"'
2019-04-13 17:54:25 | +++ cut -f1 -d.
2019-04-13 17:54:25 | ++ [[ 7 -ne 7 ]]
2019-04-13 17:54:25 | +++ df / --output=fstype
2019-04-13 17:54:25 | +++ grep -v Type
2019-04-13 17:54:25 | ++ FSTYPE=xfs
2019-04-13 17:54:25 | ++ case ${FSTYPE} in
2019-04-13 17:54:25 | +++ xfs_info /
2019-04-13 17:54:25 | +++ grep -q ftype=1
2019-04-13 17:54:25 | ++ [[ -n '' ]]
2019-04-13 17:54:25 | ++ '[' 477 = 0 ']'
2019-04-13 17:54:25 | ++ '[' '!' -d /opt/dev-scripts ']'
2019-04-13 17:54:25 | + figlet 'Deploying rook'
2019-04-13 17:54:25 | + lolcat
2019-04-13 17:54:25 | ____ _ _ _
2019-04-13 17:54:25 | | _ \ ___ _ __ | | ___ _ _(_)_ __ __ _ _ __ ___ ___ | | __
2019-04-13 17:54:25 | | | | |/ _ \ '_ \| |/ _ \| | | | | '_ \ / _` | | '__/ _ \ / _ \| |/ /
2019-04-13 17:54:25 | | |_| | __/ |_) | | (_) | |_| | | | | | (_| | | | | (_) | (_) | <
2019-04-13 17:54:25 | |____/ \___| .__/|_|\___/ \__, |_|_| |_|\__, | |_| \___/ \___/|_|\_\
2019-04-13 17:54:25 | |_| |___/ |___/
2019-04-13 17:54:25 | ++ go env
2019-04-13 17:54:25 | + eval 'GOARCH="amd64"
2019-04-13 17:54:25 | GOBIN=""
2019-04-13 17:54:25 | GOCACHE="/home/centos/.cache/go-build"
2019-04-13 17:54:25 | GOEXE=""
2019-04-13 17:54:25 | GOFLAGS=""
2019-04-13 17:54:25 | GOHOSTARCH="amd64"
2019-04-13 17:54:25 | GOHOSTOS="linux"
2019-04-13 17:54:25 | GOOS="linux"
2019-04-13 17:54:25 | GOPATH="/home/centos/go"
2019-04-13 17:54:25 | GOPROXY=""
2019-04-13 17:54:25 | GORACE=""
2019-04-13 17:54:25 | GOROOT="/usr/lib/golang"
2019-04-13 17:54:25 | GOTMPDIR=""
2019-04-13 17:54:25 | GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
2019-04-13 17:54:25 | GCCGO="gccgo"
2019-04-13 17:54:25 | CC="gcc"
2019-04-13 17:54:25 | CXX="g++"
2019-04-13 17:54:25 | CGO_ENABLED="1"
2019-04-13 17:54:25 | GOMOD=""
2019-04-13 17:54:25 | CGO_CFLAGS="-g -O2"
2019-04-13 17:54:25 | CGO_CPPFLAGS=""
2019-04-13 17:54:25 | CGO_CXXFLAGS="-g -O2"
2019-04-13 17:54:25 | CGO_FFLAGS="-g -O2"
2019-04-13 17:54:25 | CGO_LDFLAGS="-g -O2"
2019-04-13 17:54:25 | PKG_CONFIG="pkg-config"
2019-04-13 17:54:25 | GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build160591485=/tmp/go-build -gno-record-gcc-switches"'
2019-04-13 17:54:25 | ++ GOARCH=amd64
2019-04-13 17:54:25 | ++ GOBIN=
2019-04-13 17:54:25 | ++ GOCACHE=/home/centos/.cache/go-build
2019-04-13 17:54:25 | ++ GOEXE=
2019-04-13 17:54:25 | ++ GOFLAGS=
2019-04-13 17:54:25 | ++ GOHOSTARCH=amd64
2019-04-13 17:54:25 | ++ GOHOSTOS=linux
2019-04-13 17:54:25 | ++ GOOS=linux
2019-04-13 17:54:25 | ++ GOPATH=/home/centos/go
2019-04-13 17:54:25 | ++ GOPROXY=
2019-04-13 17:54:25 | ++ GORACE=
2019-04-13 17:54:25 | ++ GOROOT=/usr/lib/golang
2019-04-13 17:54:25 | ++ GOTMPDIR=
2019-04-13 17:54:25 | ++ GOTOOLDIR=/usr/lib/golang/pkg/tool/linux_amd64
2019-04-13 17:54:25 | ++ GCCGO=gccgo
2019-04-13 17:54:25 | ++ CC=gcc
2019-04-13 17:54:25 | ++ CXX=g++
2019-04-13 17:54:25 | ++ CGO_ENABLED=1
2019-04-13 17:54:25 | ++ GOMOD=
2019-04-13 17:54:25 | ++ CGO_CFLAGS='-g -O2'
2019-04-13 17:54:25 | ++ CGO_CPPFLAGS=
2019-04-13 17:54:25 | ++ CGO_CXXFLAGS='-g -O2'
2019-04-13 17:54:25 | ++ CGO_FFLAGS='-g -O2'
2019-04-13 17:54:25 | ++ CGO_LDFLAGS='-g -O2'
2019-04-13 17:54:25 | ++ PKG_CONFIG=pkg-config
2019-04-13 17:54:25 | ++ GOGCCFLAGS='-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build160591485=/tmp/go-build -gno-record-gcc-switches'
2019-04-13 17:54:25 | + export ROOKPATH=/home/centos/go/src/github.com/rook/rook
2019-04-13 17:54:25 | + ROOKPATH=/home/centos/go/src/github.com/rook/rook
2019-04-13 17:54:25 | + cd /home/centos/go/src/github.com/rook/rook/cluster/examples/kubernetes/ceph
2019-04-13 17:54:25 | + oc create -f common.yaml
2019-04-13 17:54:26 | namespace/rook-ceph created
2019-04-13 17:54:26 | customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created
2019-04-13 17:54:26 | customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created
2019-04-13 17:54:26 | customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created
2019-04-13 17:54:26 | customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created
2019-04-13 17:54:26 | customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created
2019-04-13 17:54:26 | customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created
2019-04-13 17:54:26 | customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created
2019-04-13 17:54:26 | clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
2019-04-13 17:54:26 | role.rbac.authorization.k8s.io/rook-ceph-system created
2019-04-13 17:54:26 | clusterrole.rbac.authorization.k8s.io/rook-ceph-global created
2019-04-13 17:54:26 | clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
2019-04-13 17:54:26 | serviceaccount/rook-ceph-system created
2019-04-13 17:54:26 | rolebinding.rbac.authorization.k8s.io/rook-ceph-system created
2019-04-13 17:54:26 | clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created
2019-04-13 17:54:26 | serviceaccount/rook-ceph-osd created
2019-04-13 17:54:26 | serviceaccount/rook-ceph-mgr created
2019-04-13 17:54:26 | role.rbac.authorization.k8s.io/rook-ceph-osd created
2019-04-13 17:54:26 | clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created
2019-04-13 17:54:26 | role.rbac.authorization.k8s.io/rook-ceph-mgr created
2019-04-13 17:54:26 | rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
2019-04-13 17:54:26 | rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created
2019-04-13 17:54:26 | rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created
2019-04-13 17:54:26 | rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created
2019-04-13 17:54:26 | clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
2019-04-13 17:54:26 | + sed '/FLEXVOLUME_DIR_PATH/!b;n;c\ value: "\/etc/kubernetes\/kubelet-plugins\/volume\/exec"' operator-openshift.yaml
2019-04-13 17:54:26 | + sed -i 's/# - name: FLEXVOLUME_DIR_PATH/- name: FLEXVOLUME_DIR_PATH/' operator-openshift-modified.yaml
2019-04-13 17:54:26 | + oc create -f operator-openshift-modified.yaml
2019-04-13 17:54:26 | securitycontextconstraints.security.openshift.io/rook-ceph created
2019-04-13 17:54:26 | deployment.apps/rook-ceph-operator created
2019-04-13 17:54:26 | + oc wait --for condition=ready pod -l app=rook-ceph-operator -n rook-ceph --timeout=120s
2019-04-13 17:54:26 | error: no matching resources found
apiVersion: metalkube.org/v1alpha1
kind: BareMetalHost
metadata:
name: openshift-master-1
spec:
online: true
bmc:
address: ipmi://a.b.c.d:
credentialsName: openshift-master-1-bmc-secret
bootMACAddress: 00:11:22:33:44:55
machineRef:
name: ostest-master-1
namespace: openshift-machine-api
After running make clean
I ran through the scripts again, and ran into
+ sudo usermod -a -G libvirt test
+ virsh pool-uuid default
+ virsh pool-define /dev/stdin
error: Failed to define pool from /dev/stdin
error: operation failed: Storage source conflict with pool: 'images'
To get around this, I had to destroy and undefine the left over images pool.
After installation that passed, there is too many pods it too many different states like (I particularly dislike "Terminating" and "OOMKilled" :)).
$ oc --config ocp/auth/kubeconfig get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-member-master-0 1/1 Running 0 17h
kube-system etcd-member-master-1 1/1 Running 0 17h
kube-system etcd-member-master-2 1/1 Running 0 17h
openshift-apiserver-operator openshift-apiserver-operator-7b8c99bb8b-7dnlk 1/1 Running 1 17h
openshift-apiserver apiserver-jmjrs 1/1 Running 0 17h
openshift-apiserver apiserver-mzkx6 1/1 Running 0 17h
openshift-apiserver apiserver-p2c7x 1/1 Running 0 17h
openshift-authentication-operator openshift-authentication-operator-bb8775754-9s258 1/1 Running 0 17h
openshift-authentication openshift-authentication-677f9f678-2djk7 1/1 Running 0 17h
openshift-authentication openshift-authentication-677f9f678-b6p4p 1/1 Running 0 17h
openshift-cloud-credential-operator cloud-credential-operator-5cf49888b5-ccktg 1/1 Running 0 17h
openshift-cluster-machine-approver machine-approver-6cf997dbcc-g25cx 1/1 Running 0 17h
openshift-cluster-node-tuning-operator cluster-node-tuning-operator-cb6dd6bcb-2c72p 1/1 Running 0 17h
openshift-cluster-node-tuning-operator tuned-6djpt 1/1 Running 0 17h
openshift-cluster-node-tuning-operator tuned-dx6sl 1/1 Running 0 17h
openshift-cluster-node-tuning-operator tuned-zwkng 1/1 Running 0 17h
openshift-cluster-samples-operator cluster-samples-operator-749c4b7dc7-zc5hz 1/1 Running 0 17h
openshift-cluster-storage-operator cluster-storage-operator-7d7fcb7b56-xzhfm 1/1 Running 0 17h
openshift-cluster-version cluster-version-operator-56c74d99b9-4qxhf 1/1 Running 0 17h
openshift-console-operator console-operator-589ddb9775-tll9h 1/1 Running 0 17h
openshift-console console-595b47967-nqqkz 1/1 Terminating 1 17h
openshift-console console-6d8db4c7df-b5gwh 1/1 Running 1 17h
openshift-console console-6d8db4c7df-dqnn7 0/1 Terminating 0 17h
openshift-console console-6d8db4c7df-vlkrd 1/1 Running 0 17h
openshift-console downloads-7748c8d856-2bhqw 1/1 Running 0 17h
openshift-console downloads-7748c8d856-wd56w 1/1 Running 0 17h
openshift-controller-manager-operator openshift-controller-manager-operator-5f78855946-xxj86 1/1 Running 1 17h
openshift-controller-manager controller-manager-fj5hk 1/1 Running 0 17h
openshift-controller-manager controller-manager-m5vhr 1/1 Running 0 17h
openshift-controller-manager controller-manager-tpdkm 1/1 Running 0 17h
openshift-dns-operator dns-operator-6f9d679b9c-fqj8s 1/1 Running 0 17h
openshift-dns dns-default-9t87q 2/2 Running 0 17h
openshift-dns dns-default-d942l 2/2 Running 0 17h
openshift-dns dns-default-pb227 2/2 Running 0 17h
openshift-image-registry cluster-image-registry-operator-86885f6c8d-4csbf 1/1 Running 0 17h
openshift-image-registry cluster-image-registry-operator-86885f6c8d-vvpzn 1/1 Terminating 0 17h
openshift-ingress-operator ingress-operator-7f8dcf7bb9-hvk9r 1/1 Running 0 17h
openshift-ingress router-default-55f4fcfd66-774t6 1/1 Running 0 17h
openshift-ingress router-default-55f4fcfd66-k4gz2 0/1 Pending 0 17h
openshift-ingress router-default-55f4fcfd66-phlq7 1/1 Terminating 0 17h
openshift-ingress router-default-55f4fcfd66-z555w 1/1 Running 0 17h
openshift-kube-apiserver-operator kube-apiserver-operator-6976f454fb-r5tkj 1/1 Running 1 17h
openshift-kube-apiserver installer-1-master-0 0/1 Completed 0 17h
openshift-kube-apiserver installer-1-master-1 0/1 Completed 0 17h
openshift-kube-apiserver installer-2-master-0 0/1 OOMKilled 0 17h
openshift-kube-apiserver installer-2-master-1 0/1 Completed 0 17h
openshift-kube-apiserver installer-2-master-2 0/1 Completed 0 17h
openshift-kube-apiserver installer-3-master-0 0/1 Completed 0 17h
openshift-kube-apiserver installer-3-master-1 0/1 Completed 0 17h
openshift-kube-apiserver installer-4-master-0 0/1 Completed 0 17h
openshift-kube-apiserver installer-4-master-1 0/1 Completed 0 17h
openshift-kube-apiserver installer-4-master-2 0/1 Completed 0 17h
openshift-kube-apiserver installer-5-master-0 0/1 OOMKilled 0 17h
openshift-kube-apiserver installer-6-master-0 0/1 Completed 0 17h
openshift-kube-apiserver installer-6-master-1 1/1 Running 0 17h
openshift-kube-apiserver kube-apiserver-master-0 2/2 Running 0 17h
openshift-kube-apiserver kube-apiserver-master-1 0/2 Init:0/2 0 17h
openshift-kube-apiserver kube-apiserver-master-2 2/2 Running 0 17h
openshift-kube-apiserver revision-pruner-1-master-0 0/1 Completed 0 17h
openshift-kube-apiserver revision-pruner-1-master-1 0/1 Completed 0 17h
openshift-kube-apiserver revision-pruner-2-master-0 0/1 Completed 0 17h
openshift-kube-apiserver revision-pruner-2-master-1 0/1 Completed 0 17h
openshift-kube-apiserver revision-pruner-2-master-2 0/1 Completed 0 17h
openshift-kube-apiserver revision-pruner-3-master-0 0/1 Completed 0 17h
openshift-kube-apiserver revision-pruner-3-master-1 0/1 Completed 0 17h
openshift-kube-apiserver revision-pruner-4-master-0 0/1 Completed 0 17h
openshift-kube-apiserver revision-pruner-4-master-1 0/1 Completed 0 17h
openshift-kube-apiserver revision-pruner-4-master-2 0/1 Completed 0 17h
openshift-kube-apiserver revision-pruner-5-master-0 0/1 Completed 0 17h
openshift-kube-apiserver revision-pruner-6-master-0 0/1 Completed 0 17h
openshift-kube-controller-manager-operator kube-controller-manager-operator-7566b748b8-7wk4l 1/1 Running 1 17h
openshift-kube-controller-manager installer-1-master-0 0/1 Completed 0 17h
openshift-kube-controller-manager installer-2-master-0 0/1 Completed 0 17h
openshift-kube-controller-manager installer-3-master-0 0/1 Completed 0 17h
openshift-kube-controller-manager installer-4-master-0 0/1 OOMKilled 0 17h
openshift-kube-controller-manager installer-5-master-0 0/1 OOMKilled 0 17h
openshift-kube-controller-manager installer-5-master-1 0/1 Completed 0 17h
openshift-kube-controller-manager installer-5-master-2 0/1 Completed 0 17h
openshift-kube-controller-manager installer-6-master-0 0/1 OOMKilled 0 17h
openshift-kube-controller-manager installer-6-master-1 0/1 Completed 0 17h
openshift-kube-controller-manager installer-6-master-2 0/1 OOMKilled 0 17h
openshift-kube-controller-manager kube-controller-manager-master-0 1/1 Running 2 17h
openshift-kube-controller-manager kube-controller-manager-master-1 1/1 Running 3 17h
openshift-kube-controller-manager kube-controller-manager-master-2 1/1 Running 0 17h
openshift-kube-controller-manager revision-pruner-1-master-0 0/1 Completed 0 17h
openshift-kube-controller-manager revision-pruner-2-master-0 0/1 Completed 0 17h
openshift-kube-controller-manager revision-pruner-3-master-0 0/1 Completed 0 17h
openshift-kube-controller-manager revision-pruner-4-master-0 0/1 Completed 0 17h
openshift-kube-controller-manager revision-pruner-5-master-0 0/1 Completed 0 17h
openshift-kube-controller-manager revision-pruner-5-master-1 0/1 Completed 0 17h
openshift-kube-controller-manager revision-pruner-5-master-2 0/1 Completed 0 17h
openshift-kube-controller-manager revision-pruner-6-master-0 0/1 Completed 0 17h
openshift-kube-controller-manager revision-pruner-6-master-1 0/1 Completed 0 17h
openshift-kube-controller-manager revision-pruner-6-master-2 0/1 Completed 0 17h
openshift-kube-scheduler-operator openshift-kube-scheduler-operator-cd7fd87ff-7jhlk 1/1 Running 1 17h
openshift-kube-scheduler installer-1-master-0 0/1 Completed 0 17h
openshift-kube-scheduler installer-1-master-1 0/1 Completed 0 17h
openshift-kube-scheduler installer-1-master-2 0/1 Completed 0 17h
openshift-kube-scheduler installer-2-master-0 0/1 Completed 0 17h
openshift-kube-scheduler installer-2-master-1 0/1 Completed 0 17h
openshift-kube-scheduler installer-3-master-0 0/1 Completed 0 17h
openshift-kube-scheduler installer-3-master-1 0/1 Completed 0 17h
openshift-kube-scheduler installer-3-master-2 0/1 Completed 0 17h
openshift-kube-scheduler installer-4-master-0 0/1 Completed 0 17h
openshift-kube-scheduler installer-4-master-1 0/1 Completed 0 17h
openshift-kube-scheduler installer-4-master-2 0/1 Completed 0 17h
openshift-kube-scheduler openshift-kube-scheduler-master-0 1/1 Running 0 17h
openshift-kube-scheduler openshift-kube-scheduler-master-1 1/1 Running 3 17h
openshift-kube-scheduler openshift-kube-scheduler-master-2 1/1 Running 3 17h
openshift-kube-scheduler revision-pruner-1-master-0 0/1 Completed 0 17h
openshift-kube-scheduler revision-pruner-1-master-1 0/1 Completed 0 17h
openshift-kube-scheduler revision-pruner-1-master-2 0/1 Completed 0 17h
openshift-kube-scheduler revision-pruner-2-master-0 0/1 Completed 0 17h
openshift-kube-scheduler revision-pruner-3-master-0 0/1 Completed 0 17h
openshift-kube-scheduler revision-pruner-3-master-1 0/1 Completed 0 17h
openshift-kube-scheduler revision-pruner-3-master-2 0/1 Completed 0 17h
openshift-kube-scheduler revision-pruner-4-master-0 0/1 Completed 0 17h
openshift-kube-scheduler revision-pruner-4-master-1 0/1 Completed 0 17h
openshift-kube-scheduler revision-pruner-4-master-2 0/1 Completed 0 17h
openshift-machine-api cluster-autoscaler-operator-6dd695cc7d-v2rx6 1/1 Running 1 17h
openshift-machine-api clusterapi-manager-controllers-5cf47db544-vg7dr 4/4 Running 0 17h
openshift-machine-api machine-api-operator-bd5f59899-z46pj 1/1 Running 1 17h
openshift-machine-api metalkube-baremetal-operator-8f9c66c86-mpq5t 0/1 Terminating 0 17h
openshift-machine-api metalkube-baremetal-operator-8f9c66c86-scgzw 1/1 Running 0 17h
openshift-machine-config-operator machine-config-operator-767fcfcf74-5w5mv 1/1 Running 1 17h
openshift-marketplace certified-operators-5997774686-hkg4h 1/1 Running 0 17h
openshift-marketplace community-operators-7fbb7c4588-bbklz 1/1 Running 0 17h
openshift-marketplace marketplace-operator-78d556c764-l2zgv 1/1 Running 0 17h
openshift-marketplace redhat-operators-6b4f995b78-5csgz 1/1 Running 0 17h
openshift-monitoring alertmanager-main-0 0/3 Pending 0 17h
openshift-monitoring alertmanager-main-1 3/3 Running 0 17h
openshift-monitoring cluster-monitoring-operator-5c68c9d967-th5qk 1/1 Running 0 17h
openshift-monitoring grafana-74876d8b8d-4qwqf 0/2 Preempting 0 17h
openshift-monitoring grafana-74876d8b8d-6h8zm 2/2 Running 0 17h
openshift-monitoring kube-state-metrics-56d947b89d-dfk66 3/3 Terminating 0 17h
openshift-monitoring kube-state-metrics-56d947b89d-l6jlz 3/3 Running 0 17h
openshift-monitoring node-exporter-4jjkc 2/2 Running 0 17h
openshift-monitoring node-exporter-jsrzm 2/2 Running 0 17h
openshift-monitoring node-exporter-kv4bb 2/2 Running 0 17h
openshift-monitoring prometheus-adapter-c76d6596f-82vjk 1/1 Running 0 17h
openshift-monitoring prometheus-adapter-c76d6596f-lcnfv 1/1 Running 0 17h
openshift-monitoring prometheus-adapter-c76d6596f-q5kwq 1/1 Terminating 0 17h
openshift-monitoring prometheus-k8s-0 0/6 Pending 0 17h
openshift-monitoring prometheus-k8s-1 6/6 Terminating 1 17h
openshift-monitoring prometheus-operator-6ff74c9976-rmqvq 1/1 Running 0 17h
openshift-monitoring prometheus-operator-6ff74c9976-x52g6 1/1 Terminating 1 17h
openshift-monitoring telemeter-client-6579d7cf8-q7mkk 3/3 Running 0 17h
openshift-multus multus-8j6kb 1/1 Running 0 17h
openshift-multus multus-98kcc 1/1 Running 0 17h
openshift-multus multus-nzm94 1/1 Running 0 17h
openshift-network-operator network-operator-56b8ccdcbb-xd9q5 1/1 Running 0 17h
openshift-operator-lifecycle-manager catalog-operator-6865f8bd88-hwqth 1/1 Running 0 17h
openshift-operator-lifecycle-manager olm-operator-fccbd8798-9x77k 1/1 Running 0 17h
openshift-operator-lifecycle-manager olm-operators-zhkk2 1/1 Running 0 17h
openshift-operator-lifecycle-manager packageserver-544b89c886-8ldtr 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-54569d7c8f-8j9sv 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-5486cc45fd-nqwvt 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-556df8d845-gcn7w 0/1 Terminating 0 16h
openshift-operator-lifecycle-manager packageserver-55b74df5bb-m2cqx 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-5647ff4c9b-z78cm 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-5668b7bd5-ssvrf 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-5687977596-db522 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-56ccf9988b-f5rzq 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-576dddbcf6-f4fk8 0/1 Terminating 0 16h
openshift-operator-lifecycle-manager packageserver-58d68cfc8b-lpz5m 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-58f796cb5-zcfk8 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-596f49576c-zj282 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-599489cfbf-5tng4 0/1 Terminating 0 13h
openshift-operator-lifecycle-manager packageserver-5bc68c459b-mlpsg 1/1 Running 0 13h
openshift-operator-lifecycle-manager packageserver-5bc68c459b-mqxvq 1/1 Running 0 13h
openshift-operator-lifecycle-manager packageserver-5c548f4fb5-h6tpf 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-5c6c4f598b-xh7hv 0/1 Terminating 0 13h
openshift-operator-lifecycle-manager packageserver-5cc868cff6-kz2gb 0/1 Terminating 0 17h
openshift-operator-lifecycle-manager packageserver-5cdfd89cf9-r7bqn 0/1 Terminating 0 16h
openshift-operator-lifecycle-manager packageserver-5df46449fb-58pdm 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-5dfcd9fb56-fgmlc 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-6484b96c7-lxbnq 1/1 Terminating 0 17h
openshift-operator-lifecycle-manager packageserver-64cbcff7cb-chrv9 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-64ff8c5b89-vnpj4 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-658d685bdf-t4j7v 0/1 Terminating 0 16h
openshift-operator-lifecycle-manager packageserver-65fb76c755-qgh47 0/1 Terminating 0 17h
openshift-operator-lifecycle-manager packageserver-66845969c7-qg7ks 0/1 Terminating 0 13h
openshift-operator-lifecycle-manager packageserver-669b56f59-xz8xb 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-6798cc6565-49dlp 0/1 Terminating 0 16h
openshift-operator-lifecycle-manager packageserver-68475586b6-tx6wm 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-687768b948-q9l9t 0/1 Terminating 0 17h
openshift-operator-lifecycle-manager packageserver-68bdb47898-lh6bm 0/1 Terminating 0 16h
openshift-operator-lifecycle-manager packageserver-68cf9fd6bf-dlbn9 0/1 Terminating 0 17h
openshift-operator-lifecycle-manager packageserver-69488c7665-zbvnt 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-6988754f9c-t7rp2 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-699795f5c6-22lxq 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-6b59bdb4d7-2mgjc 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-6d6548469d-k7jth 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-6d8446c695-z5n4h 0/1 Terminating 0 16h
openshift-operator-lifecycle-manager packageserver-6d89c547c9-m8bk8 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-6f769fdc56-nqc8g 0/1 Terminating 0 13h
openshift-operator-lifecycle-manager packageserver-74cd8b5574-cg8gp 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-76648b7798-tpfsn 0/1 Terminating 0 17h
openshift-operator-lifecycle-manager packageserver-76d4b8dbc4-frmd5 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-77678bbdc-s6pd8 0/1 Terminating 0 16h
openshift-operator-lifecycle-manager packageserver-7869f5c94f-vj6xn 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-78d7685975-jwg4k 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-79d9f749bd-28h7x 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-79ff46ccc5-kwpwg 0/1 Terminating 0 13h
openshift-operator-lifecycle-manager packageserver-7b5bbc9db6-xhpdb 0/1 Terminating 0 16h
openshift-operator-lifecycle-manager packageserver-7bbf6b9f9c-klrst 0/1 Terminating 0 13h
openshift-operator-lifecycle-manager packageserver-7bdf97c5d5-nzbfz 0/1 Terminating 0 16h
openshift-operator-lifecycle-manager packageserver-7c58db8f46-kh9db 0/1 Terminating 0 16h
openshift-operator-lifecycle-manager packageserver-7c64c778c7-gzlsd 0/1 Terminating 0 16h
openshift-operator-lifecycle-manager packageserver-7cf476b5b4-xpnm5 0/1 Terminating 0 16h
openshift-operator-lifecycle-manager packageserver-7dcc548b54-vvt6j 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-845c6bbf44-ppzmv 0/1 Terminating 0 16h
openshift-operator-lifecycle-manager packageserver-84b6f97575-s962t 0/1 Terminating 0 16h
openshift-operator-lifecycle-manager packageserver-84c9cbb78c-bjskn 0/1 Terminating 0 13h
openshift-operator-lifecycle-manager packageserver-85985d844c-prpfw 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-867448d9-f8cfq 0/1 Terminating 0 16h
openshift-operator-lifecycle-manager packageserver-8b74bd6d8-f8vk9 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-b48c575c8-257jl 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-b89585667-srqhn 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-c4dd86cc9-tppd6 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-c78858db-blkc8 0/1 Terminating 0 14h
openshift-operator-lifecycle-manager packageserver-c9f694678-fn8cd 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-cc84784b6-nqnwr 0/1 Terminating 0 15h
openshift-operator-lifecycle-manager packageserver-cf694654d-c724r 0/1 Terminating 0 17h
openshift-sdn ovs-7p2qf 1/1 Running 0 17h
openshift-sdn ovs-k8qbx 1/1 Running 0 17h
openshift-sdn ovs-z7wwv 1/1 Running 0 17h
openshift-sdn sdn-7xlxr 1/1 Running 0 17h
openshift-sdn sdn-controller-jfmj5 1/1 Running 0 17h
openshift-sdn sdn-controller-nfmkf 1/1 Running 0 17h
openshift-sdn sdn-controller-sl2g4 1/1 Running 0 17h
openshift-sdn sdn-sjskt 1/1 Running 2 17h
openshift-sdn sdn-t8bc8 1/1 Running 1 17h
openshift-service-ca-operator openshift-service-ca-operator-7957dd76c9-p7425 1/1 Running 0 17h
openshift-service-ca apiservice-cabundle-injector-6589bb696b-phqmw 1/1 Running 0 17h
openshift-service-ca configmap-cabundle-injector-787f7f684b-z78kn 1/1 Running 0 17h
openshift-service-ca service-serving-cert-signer-58f9487f4f-5stwv 1/1 Running 0 17h
openshift-service-catalog-apiserver-operator openshift-service-catalog-apiserver-operator-84d5b596c7-wghlc 1/1 Running 1 17h
openshift-service-catalog-controller-manager-operator openshift-service-catalog-controller-manager-operator-77cf59cj8 1/1 Running 1 17h
Using:
commit 489d790e1819e2e1ce489bbac87a4d5bbd5ac47e
Merge: c9bbf54 98ba9fb
Author: Steven Hardy <[email protected]>
Date: Wed Apr 10 15:22:10 2019 +0100
Merge pull request #312 from zaneb/fix_certs
Run fix_certs script automatically
Virtual environment:
"level=fatal msg=\"failed to initialize the cluster: Cluster operator machine-config is reporting a failure: Failed to resync 4.0.0-0.alpha-2019-04-10-154442 because: [timed out waiting for the condition during waitForControllerConfigToBeCompleted: controllerconfig is not completed: ControllerConfig has not completed: completed(false) running(false) failing(true), pool master has not progressed to latest configuration: configuration for pool master is empty, retrying]: timed out waiting for the condition\"
make
fails with following error for me:
$ make
[...]
Logging to ./logs/04_setup_ironic-2019-03-24-174739.log
+++ exec
++++ tee ./logs/04_setup_ironic-2019-03-24-174739.log
++ '[' '!' -f redhat-coreos-maipo-47.284-qemu.qcow2 ']'
++ mkdir -p /opt/dev-scripts/ironic/html/images
++ '[' -d images ']'
++ pushd /opt/dev-scripts/ironic/html/images
/opt/dev-scripts/ironic/html/images ~/dev-scripts
++ '[' '!' -f redhat-coreos-maipo-47.284-openstack.qcow2 ']'
++ curl --insecure --compressed -L -o redhat-coreos-maipo-47.284-openstack.qcow2 https://releases-rhcos.svc.ci.openshift.org/storage/releases/maipo//47.284/redhat-coreos-maipo-47.284-openstack.qcow2.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 161 100 161 0 0 325 0 --:--:-- --:--:-- --:--:-- 326
100 662M 100 662M 0 0 4257k 0 0:02:39 0:02:39 --:--:-- 8318k
++ '[' '!' -f ironic-python-agent.initramfs ']'
++ tar -xf -
++ curl --insecure --compressed -L https://images.rdoproject.org/master/rdo_trunk/54c5a6de8ce5b9cfae83632a7d81000721d56071_786d88d2/ironic-python-agent.tar
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 288 100 288 0 0 539 0 --:--:-- --:--:-- --:--:-- 539
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
make: *** [ironic] Error 2
Currently we ship ironic with an unbounded number of API workers. This could become an issue on the platform. We should consider bounding ironic to 12 workers (similar to how we do things in TripleO today [however, not with ironic]).
Check for RHEL 7 fail with RHEL 7.6 errors - several errors - 1 example
++ sudo -n uptime
+++ awk -F= '/^ID=/ { print $2 }' /etc/os-release
+++ tr -d '"'
++ [[ ! rhel =~ ^(centos|rhel)$ ]]
+++ awk -F= '/^VERSION_ID=/ { print $2 }' /etc/os-release
+++ tr -d '"'
++ [[ 7.6 -ne 7 ]]
common.sh: line 84: [[: 7.6: syntax error: invalid arithmetic operator (error token is ".6")
+++ df / --output=fstype
+++ grep -v Type
++ FSTYPE=xfs
[core@master-1 ~]$ ONE_CIDR="$(ip addr show to "$SUBNET_CIDR" | \
grep -Po 'inet \K[\d.]+/[\d.]+' | \ grep -v "${DNS_VIP}/$PREFIX" | \ grep -v "${API_VIP}/$PREFIX" | \ sort | xargs | cut -f1 -d' ')"
Error: any valid prefix is expected rather than "".
I set
WORKING_DIR=${WORKING_DIR:-"/opt/dev-scripts"}
NODES_FILE=${NODES_FILE:-"/home/test/rook.json"}
NODES_PLATFORM=${NODES_PLATFORM:-"baremetal"}
MASTER_NODES_FILE=${MASTER_NODES_FILE:-"/home/test/rook.json"}
TASK [include_role : parts/ovs] *********************************************************************************************************************************************************************************************************
task path: /home/test/dev-scripts/tripleo-quickstart-config/roles/environment/setup/tasks/main.yml:6
fatal: [localhost]: FAILED! => {
"msg": "The conditional check 'networks|selectattr('virtualport_type', 'defined')|map(attribute='name')|list|length > 0' failed. The error was: error while evaluating conditional (networks|selectattr('virtualport_type', 'defined')|map(attribute='name')|list|length > 0): 'networks' is undefined\n\nThe error appears to have been in '/home/test/dev-scripts/tripleo-quickstart-config/roles/environment/setup/tasks/main.yml': line 6, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n # Install OVS dependencies\n - name: Install OVS dependencies\n ^ here\n"
}
to retry, use: --limit @/home/test/dev-scripts/tripleo-quickstart-config/metalkube-setup-playbook.retry
PLAY RECAP ******************************************************************************************************************************************************************************************************************************
localhost : ok=12 changed=2 unreachable=0 failed=1
tripleo-repos warnings
Attempted a baremetal deploy using some qct D51B-1u servers.
I was able to install RHEL 7.6 one these servers after getting the failures attempt to install CentOS on this server fails.
errors from ./01_install_requirements.sh
tripleo-repos states rhel7.6 is unsupported
Installation failed:
$ make
[...]
ironic_node_v1.openshift-master-2: Still creating... (6m20s elapsed)
ironic_node_v1.openshift-master-0: Still creating... (6m20s elapsed)
2019-03-27T18:19:29.662-0400 [DEBUG] plugin.terraform-provider-ironic: 2019/03/27 18:19:29 [DEBUG] Node current state is 'active'
2019-03-27T18:19:29.662-0400 [DEBUG] plugin.terraform-provider-ironic: 2019/03/27 18:19:29 [DEBUG] Node 8f0943d0-839c-45da-a548-c57e69501fdf is 'active', we are done.
ironic_node_v1.openshift-master-2: Creation complete after 6m25s (ID: 8f0943d0-839c-45da-a548-c57e69501fdf)
2019-03-27T18:19:29.878-0400 [DEBUG] plugin.terraform-provider-ironic: 2019/03/27 18:19:29 [DEBUG] Node current state is 'active'
2019-03-27T18:19:29.878-0400 [DEBUG] plugin.terraform-provider-ironic: 2019/03/27 18:19:29 [DEBUG] Node b8e484a7-bee9-49f3-a203-a3f06a474b74 is 'active', we are done.
ironic_node_v1.openshift-master-0: Creation complete after 6m25s (ID: b8e484a7-bee9-49f3-a203-a3f06a474b74)
2019/03/27 18:19:30 [DEBUG] plugin: waiting for all plugin processes to complete...
Error: Error applying plan:
1 error(s) occurred:
* ironic_node_v1.openshift-master-1: 1 error(s) occurred:
* ironic_node_v1.openshift-master-1: Internal Server Error
Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
2019-03-27T18:19:30.101-0400 [DEBUG] plugin.terraform-provider-ironic: 2019/03/27 18:19:30 [ERR] plugin: plugin server: accept unix /tmp/plugin204986381: use of closed network connection
2019-03-27T18:19:30.102-0400 [DEBUG] plugin: plugin process exited: path=/home/kni/.terraform.d/plugins/terraform-provider-ironic
make: *** [ocp_run] Error 1
Some random commands as I basically do not know what I'm doing :-)
[kni@intel-canoepass-09 dev-scripts]$ oc --config /home/kni/dev-scripts/ocp/auth/kubeconfig get nodes
error: the server doesn't have a resource type "nodes"
[kni@intel-canoepass-09 dev-scripts]$ export OS_TOKEN=fake-token
[kni@intel-canoepass-09 dev-scripts]$ export OS_URL=http://localhost:6385/
[kni@intel-canoepass-09 dev-scripts]$ openstack baremetal node list
+--------------------------------------+--------------------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------------------+---------------+-------------+--------------------+-------------+
| c9c0a601-86a7-41ad-bf84-96c429b93fd9 | openshift-master-1 | None | power on | active | False |
| b8e484a7-bee9-49f3-a203-a3f06a474b74 | openshift-master-0 | None | power on | active | False |
| 8f0943d0-839c-45da-a548-c57e69501fdf | openshift-master-2 | None | power on | active | False |
+--------------------------------------+--------------------+---------------+-------------+--------------------+-------------+
[kni@intel-canoepass-09 dev-scripts]$ openstack baremetal node show c9c0a601-86a7-41ad-bf84-96c429b93fd9
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| allocation_uuid | None |
| automated_clean | None |
| bios_interface | no-bios |
| boot_interface | ipxe |
| chassis_uuid | None |
| clean_step | {} |
| conductor | intel-canoepass-09.khw1.lab.eng.bos.redhat.com |
| conductor_group | |
| console_enabled | False |
| console_interface | no-console |
| created_at | 2019-03-27T22:13:04.839047+00:00 |
| deploy_interface | direct |
| deploy_step | {} |
| description | None |
| driver | ipmi |
| driver_info | {u'ipmi_port': u'6231', u'ipmi_username': u'admin', u'deploy_kernel': u'http://172.22.0.1/images/ironic-python-agent.kernel', u'ipmi_address': u'192.168.111.1', u'deploy_ramdisk': u'http://172.22.0.1/images/ironic-python-agent.initramfs', u'ipmi_password': u'******'} |
| driver_internal_info | {u'deploy_boot_mode': u'bios', u'is_whole_disk_image': True, u'root_uuid_or_disk_id': u'0xbaef8236', u'agent_url': u'http://172.22.0.78:9999', u'deploy_steps': None, u'agent_version': u'3.7.0.dev3', u'agent_last_heartbeat': u'2019-03-27T22:18:08.408711'} |
| extra | {} |
| fault | None |
| inspect_interface | inspector |
| inspection_finished_at | None |
| inspection_started_at | None |
| instance_info | {u'root_gb': u'25', u'image_source': u'http://172.22.0.1/images/redhat-coreos-maipo-latest.qcow2', u'image_type': u'whole-disk-image', u'root_device': u'/dev/vda', u'image_checksum': u'308f00a5cb04c5aaf0f15073dabe335f', u'image_url': u'******', u'configdrive': u'******'} |
| instance_uuid | None |
| last_error | None |
| maintenance | False |
| maintenance_reason | None |
| management_interface | ipmitool |
| name | openshift-master-1 |
| network_interface | noop |
| owner | None |
| power_interface | ipmitool |
| power_state | power on |
| properties | {} |
| protected | False |
| protected_reason | None |
| provision_state | active |
| provision_updated_at | 2019-03-27T22:19:20.972264+00:00 |
| raid_config | {} |
| raid_interface | no-raid |
| rescue_interface | no-rescue |
| reservation | None |
| resource_class | None |
| storage_interface | noop |
| target_power_state | None |
| target_provision_state | None |
| target_raid_config | {} |
| traits | [] |
| updated_at | 2019-03-27T22:19:21.011055+00:00 |
| uuid | c9c0a601-86a7-41ad-bf84-96c429b93fd9 |
| vendor_interface | ipmitool |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[kni@intel-canoepass-09 dev-scripts]$ openstack baremetal node show c9c0a601-86a7-41ad-bf84-96c429b93fd9 -f value -c last_error
None
after restarting the nodes, on a virtual setup, ocp no longer works, the api endpoint doesnt respond though the corresponding vip is running on one of the master nodes.
Proposing this after discussion with @mcornea and after debugging issues caused by OOM on a host with 32GB RAM. Would 16GB be more reasonable default allowing higher installation success rate?
currently, nodes are deployed with 2vcpus and 6Gb of ram, which causes pods to crashloop or not launch after initial bootstrapping.
we should increase those values to maybe 8 cpus and at least 8gb of ram
In 10_deploy_rook.sh
there is no way to provision rook with other YAML files. This is problematic for CI systems where certain options might be needed to verify certain behavior.
Although it is helpful to have some sort of "vanilla" setup, it is important to have some way to define a path for these YAML files so that rook can be configured differently.
Since the cluster DNS is not something that is necessary from the outside, if we just put it in ignitions, it will take one thing of the prereqs list.
I'm not sure if this is related to RHCOS version (OSTREE_VERSION=47.284) or if this can be fixed in the dev-scripts, but in my environment (3 masters, 4 nics, eth0 = provisioning network, eth1 = baremetal network, eth2 & eth3 = unused) ironic python agent and fedora get the proper nic order, but RHCOS uses eth2 as eth0 and eth3 as eth1 so the installation fails.
As a workaround, modifying the 04 script to use eth2 and eth3 instead seems to work.
api-haproxy.service unit enters infinite restart loop
[root@master-2 ~]# systemctl status api-haproxy.service
โ api-haproxy.service - Load Balance OpenShift API using HAProxy
Loaded: loaded (/etc/systemd/system/api-haproxy.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-04-10 16:11:02 UTC; 1ms ago
Main PID: 19361 (bash)
Tasks: 0 (limit: 26213)
Memory: 0B
CPU: 189us
CGroup: /system.slice/api-haproxy.service
โโ19361 bash /usr/local/bin/haproxy.sh
Apr 10 16:11:02 master-2 systemd[1]: Started Load Balance OpenShift API using HAProxy.
[root@master-2 ~]# systemctl status api-haproxy.service
โ api-haproxy.service - Load Balance OpenShift API using HAProxy
Loaded: loaded (/etc/systemd/system/api-haproxy.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Wed 2019-04-10 16:11:03 UTC; 427ms ago
Process: 19361 ExecStart=/usr/local/bin/haproxy.sh (code=exited, status=125)
Main PID: 19361 (code=exited, status=125)
CPU: 624ms
Apr 10 16:11:03 master-2 systemd[1]: api-haproxy.service: Consumed 624ms CPU time
From journal:
Apr 10 16:11:02 master-2 sudo[19412]: root : TTY=unknown ; PWD=/etc/haproxy ; USER=root ; COMMAND=/bin/mkdir --parents /etc/haproxy
Apr 10 16:11:02 master-2 sudo[19412]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 10 16:11:02 master-2 sudo[19412]: pam_unix(sudo:session): session closed for user root
Apr 10 16:11:02 master-2 sudo[19427]: root : TTY=unknown ; PWD=/etc/haproxy ; USER=root ; COMMAND=/bin/tee /etc/haproxy/haproxy.cfg
Apr 10 16:11:02 master-2 sudo[19427]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 10 16:11:02 master-2 haproxy.sh[19361]: defaults
Apr 10 16:11:02 master-2 haproxy.sh[19361]: mode tcp
Apr 10 16:11:02 master-2 haproxy.sh[19361]: log global
Apr 10 16:11:02 master-2 haproxy.sh[19361]: option dontlognull
Apr 10 16:11:02 master-2 haproxy.sh[19361]: retries 3
Apr 10 16:11:02 master-2 haproxy.sh[19361]: timeout http-request 10s
Apr 10 16:11:02 master-2 haproxy.sh[19361]: timeout queue 1m
Apr 10 16:11:02 master-2 haproxy.sh[19361]: timeout connect 10s
Apr 10 16:11:02 master-2 haproxy.sh[19361]: timeout client 86400s
Apr 10 16:11:02 master-2 haproxy.sh[19361]: timeout server 86400s
Apr 10 16:11:02 master-2 haproxy.sh[19361]: timeout tunnel 86400s
Apr 10 16:11:02 master-2 haproxy.sh[19361]: frontend main
Apr 10 16:11:02 master-2 haproxy.sh[19361]: bind :7443
Apr 10 16:11:02 master-2 haproxy.sh[19361]: default_backend masters
Apr 10 16:11:02 master-2 haproxy.sh[19361]: backend masters
Apr 10 16:11:02 master-2 haproxy.sh[19361]: option httpchk GET /healthz HTTP/1.0
Apr 10 16:11:02 master-2 haproxy.sh[19361]: option log-health-checks
Apr 10 16:11:02 master-2 haproxy.sh[19361]: balance roundrobin
Apr 10 16:11:02 master-2 sudo[19427]: pam_unix(sudo:session): session closed for user root
Apr 10 16:11:03 master-2 sudo[19511]: root : TTY=unknown ; PWD=/etc/haproxy ; USER=root ; COMMAND=/bin/podman ps -a --format {{.Names}}
Apr 10 16:11:03 master-2 sudo[19511]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 10 16:11:03 master-2 sudo[19511]: pam_unix(sudo:session): session closed for user root
Apr 10 16:11:03 master-2 sudo[19557]: root : TTY=unknown ; PWD=/etc/haproxy ; USER=root ; COMMAND=/bin/podman start api-haproxy
Apr 10 16:11:03 master-2 sudo[19557]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 10 16:11:03 master-2 haproxy.sh[19361]: unable to start container "api-haproxy": unable to determine if "/var/lib/containers/storage/overlay-containers/3da81e2bb93889c088b4a85a9e862693d059077d8e992089ddf7393f3c49df42/userdata/shm" is mounted: failed to canonicalise path for /var/lib/containers/storage/overlay-containers/3da81e2bb93889c088b4a85a9e862693d059077d8e992089ddf7393f3c49df42/userdata/shm: lstat /var/lib/containers/storage/overlay-containers/3da81e2bb93889c088b4a85a9e862693d059077d8e992089ddf7393f3c49df42: no such file or directory
Apr 10 16:11:03 master-2 sudo[19557]: pam_unix(sudo:session): session closed for user root
Apr 10 16:11:03 master-2 systemd[1]: api-haproxy.service: Main process exited, code=exited, status=125/n/a
Apr 10 16:11:03 master-2 systemd[1]: api-haproxy.service: Failed with result 'exit-code'.
Apr 10 16:11:03 master-2 systemd[1]: api-haproxy.service: Consumed 624ms CPU time
Apr 10 16:11:08 master-2 systemd[1]: api-haproxy.service: Service RestartSec=5s expired, scheduling restart.
Apr 10 16:11:08 master-2 systemd[1]: api-haproxy.service: Scheduled restart job, restart counter is at 575.
Apr 10 16:11:08 master-2 systemd[1]: Stopped Load Balance OpenShift API using HAProxy.
Apr 10 16:11:08 master-2 systemd[1]: api-haproxy.service: Consumed 624ms CPU time
Apr 10 16:11:08 master-2 systemd[1]: Started Load Balance OpenShift API using HAProxy.
Apr 10 16:11:09 master-2 sudo[19792]: root : TTY=unknown ; PWD=/etc/haproxy ; USER=root ; COMMAND=/bin/mkdir --parents /etc/haproxy
Apr 10 16:11:09 master-2 sudo[19792]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 10 16:11:09 master-2 sudo[19792]: pam_unix(sudo:session): session closed for user root
Apr 10 16:11:09 master-2 sudo[19807]: root : TTY=unknown ; PWD=/etc/haproxy ; USER=root ; COMMAND=/bin/tee /etc/haproxy/haproxy.cfg
Apr 10 16:11:09 master-2 sudo[19807]: pam_unix(sudo:session): session opened for user root by (uid=0)
See also openshift-metal3/kni-installer#37
In dev-scripts, rather than using the latest rhcos image, we should use the same one build that is hard-coded in the kni-installer repository:
hack/build.sh:5:RHCOS_BUILD_NAME="${RHCOS_BUILD_NAME:-400.7.20190306.0}"
When openshift-metal3/kni-installer#36 lands, that also means changing the channel from maipo to ootpa
Permission denied
Logging to ./logs/04_setup_ironic-2019-03-05-150932.log
+++ exec
++++ tee ./logs/04_setup_ironic-2019-03-05-150932.log
++ '[' '!' -f redhat-coreos-maipo-47.284-qemu.qcow2 ']'
++ mkdir -p /opt/dev-scripts/ironic/html/images
mkdir: cannot create directory โ/opt/dev-scripts/ironicโ: Permission denied
When applying the kubevirt
manifests from manifests/
directory, oc
fails
to post the resources with the following errors:
[root@zeus07 manifests]# oc --config ../ocp/auth/kubeconfig apply -f 110_cnv_kubevirt_op.yaml
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
customresourcedefinition.apiextensions.k8s.io/kubevirts.kubevirt.io configured
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
clusterrole.rbac.authorization.k8s.io/kubevirt.io:operator configured
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
clusterrolebinding.rbac.authorization.k8s.io/kubevirt-operator configured
Error from server (Forbidden): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"labels\":{\"kubevirt.io\":\"\"},\"name\":\"kubevirt\",\"namespace\":\"\"}}\n"},"namespace":""}}
to:
Resource: "/v1, Resource=namespaces", GroupVersionKind: "/v1, Kind=Namespace"
Name: "kubevirt", Namespace: ""
Object: &{map["kind":"Namespace" "apiVersion":"v1" "metadata":map["name":"kubevirt" "selfLink":"/api/v1/namespaces/kubevirt" "uid":"cfc4fdce-41bb-11e9-a9c0-52fdfc072182" "resourceVersion":"904" "creationTimestamp":"2019-03-08T16:04:11Z" "labels":map["kubevirt.io":""]] "spec":map["finalizers":["kubernetes"]] "status":map["phase":"Active"]]}
for: "110_cnv_kubevirt_op.yaml": namespaces "kubevirt" is forbidden: caches not synchronized
Error from server (Forbidden): error when creating "110_cnv_kubevirt_op.yaml": serviceaccounts "kubevirt-operator" is forbidden: caches not synchronized
Error from server (Forbidden): error when creating "110_cnv_kubevirt_op.yaml": deployments.apps "virt-operator" is forbidden: caches not synchronized
When we switch to kube-system
namespace then it works fine.
The same error happens when manifests are applied by openshift-install
itself, as in #127.
I plan to post a PR that switches to kube-system
to make progress on
deployment. This issue will be used to track investigation on reasons behind
the failure and to revert back to kubevirt
namespace once we have the root
cause.
Attempted a baremetal deploy using some qct D51B-1u servers.
I was able to install RHEL 7.6 one these servers after getting the failures attempt to install CentOS on this server fails.
errors from ./01_install_requirements.sh
jq used but not installed - several failures 1 example
+++ jq -r '.builds[0]'
common.sh: line 42: jq: command not found
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 6067 100 6067 0 0 10056 0 --:--:-- --:--:-- --:--:-- 10044
curl: (23) Failed writing body (0 != 6067)
If there is a networking hiccup with the setup, this loop could cause the scripts to wait forever[1].
+ host -t SRV _etcd-server-ssl._tcp.ostest.test.metalkube.org api.ostest.test.metalkube.org
+ echo -n .
.+ sleep 1
+ host -t SRV _etcd-server-ssl._tcp.ostest.test.metalkube.org api.ostest.test.metalkube.org
+ echo -n .
.+ sleep 1
+ host -t SRV _etcd-server-ssl._tcp.ostest.test.metalkube.org api.ostest.test.metalkube.org
+ echo -n .
.+ sleep 1
[1] https://github.com/openshift-metalkube/dev-scripts/blob/master/utils.sh#L162
While executing dev-scripts I observed Error during fetching of latest RHCOS images,
level=debug msg=Fetching RHCOS metadata from "https://releases-rhcos.svc.ci.openshift.org/storage/releases/maipo/400.7.20190306.0/meta.json\""
level=fatal msg="failed to fetch Bootstrap Ignition Config: failed to fetch dependency of "Bootstrap Ignition Config": failed to fetch dependency of "Master Machines": failed to generate asset "Image": failed to fetch RHCOS metadata: incorrect HTTP response (503 Service Unavailable)"
make: *** [ocp_run] Error 1
looks like its due to extra escaping character in URL
"https://releases-rhcos.svc.ci.openshift.org/storage/releases/maipo/400.7.20190306.0/meta.json\""
It looks like the cluster deployed with the dev-scripts is not registered in cloud.openshift.com.
By some talks in the #forum-monitoring channel, it seems that in order to the cluster to be registered, the monitoring operator is failing. That is the component responsible for sending the metrics to telemetry, so if it is failing no metrics will be sent, and the cluster will not be registered.
From the events in the "openshift-monitoring" it looks like prometheus is not deployed as there are no nodes available (insufficient cpu)
[root@f13-h02-000-1029u dev-scripts]# oc -n rook-ceph exec rook-ceph-tools-76c7d559b6-95l8p -- ceph -s
cluster:
id: 2661923a-db3a-405a-b07a-ceb6e33d86ce
health: HEALTH_WARN
367 slow ops, oldest one blocked for 2288 sec, mon.c has slow ops
clock skew detected on mon.b, mon.c
services:
mon: 3 daemons, quorum a,b,c
mgr: a(active)
mds: myfs-1/1/1 up {0=myfs-b=up:active}, 1 up:standby-replay
osd: 3 osds: 3 up, 3 in
data:
pools: 3 pools, 300 pgs
objects: 22 objects, 2.2 KiB
usage: 3.0 GiB used, 2.6 TiB / 2.6 TiB avail
pgs: 300 active+clean
io:
client: 1.2 KiB/s rd, 2 op/s rd, 0 op/s wr
Which seems to be due to times being off
debug 2019-04-15 17:26:12.093 7f10f49ff700 -1 mon.c@2(peon).paxos(paxos updating c 1005..1751) lease_expire from mon.0 172.30.19.151:6789/0 is 123.467528 seconds in the past; mons are probably laggy (or possibly clocks are too skewed)
[root@f13-h02-000-1029u dev-scripts]# for n in 20 21 22; do
> ssh [email protected].$n date;
> done
Mon Apr 15 17:47:01 UTC 2019
Mon Apr 15 17:47:00 UTC 2019
Mon Apr 15 17:49:10 UTC 2019
Which this seems similar to issue #354
curl -s https://releases-rhcos.svc.ci.openshift.org/storage/releases/ootpa/builds.json | jq -r ".builds[0]"
410.8.20190412.1
curl -s https://releases-rhcos.svc.ci.openshift.org/storage/releases/ootpa/410.8.20190412.1/meta.json | jq -r .images
{
"qemu": {
"path": "rhcos-410.8.20190412.1-qemu.qcow2",
"sha256": "b2b80605be197d988816df347cfe76c652db556fc47830df7c74c66e2ac97d28",
"size": "765151729",
"uncompressed-sha256": "1a9394ed8383cebb41228e49e4b3b009d7e6a6cc0045061ce0549d3ed860ccc0",
"uncompressed-size": "2165243904"
}
}
curl -s https://releases-rhcos.svc.ci.openshift.org/storage/releases/ootpa/410.8.20190412.1/meta.json | jq -r .images.openstack
null
Affected code: https://github.com/openshift-metalkube/dev-scripts/blob/master/common.sh#L54
Something recently introduced is causing deployments to fail.
Apr 03 18:16:08 localhost.localdomain systemd[1]: Started Bootstrap a Kubernetes cluster.
Apr 03 18:16:27 localhost.localdomain bootkube.sh[11222]: Rendering MCO manifests...
Apr 03 18:16:28 localhost.localdomain bootkube.sh[11222]: I0403 18:16:28.454325 1 bootstrap.go:84] Version: 4.0.0-alpha.0-154-gc1ba218c-dirty
Apr 03 18:16:28 localhost.localdomain bootkube.sh[11222]: F0403 18:16:28.456056 1 bootstrap.go:104] error rendering bootstrap manifests: open /assets/tls/etcd-metric-ca-bundle.crt: no such file or directory
Apr 03 18:16:28 localhost.localdomain systemd[1]: bootkube.service: main process exited, code=exited, status=255/n/a
Apr 03 18:16:28 localhost.localdomain systemd[1]: Unit bootkube.service entered failed state.
Apr 03 18:16:28 localhost.localdomain systemd[1]: bootkube.service failed.
The kubevirt based ui is not deployed with the dev-scripts. I'm not sure what would be the procedure (maybe add more stuff to the manifests?) but I think it worth opening the issue to ensure this can be added to the deployment (it looks like the ui is this one https://github.com/kubevirt/web-ui-operator)
After 24 of a runnign cluster, nodes (which are tainted masters) getting into NotReady status.
SSH in to the node and systemctl status kubelet showing this:
kubelet.service - Kubernetes Kubelet
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2019-04-14 09:33:24 UTC; 22h ago
Process: 1409 ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state (code=exited, status=0/SUCCESS)
Process: 1407 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
Main PID: 1411 (hyperkube)
Tasks: 57 (limit: 26213)
Memory: 246.6M
CPU: 4h 37min 3.310s
CGroup: /system.slice/kubelet.service
โโ1411 /usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --rotate-certificates --kubeconfig=/var/lib/kubelet/kubeconfig --container>
Apr 15 08:18:24 master-1 hyperkube[1411]: E0415 08:18:24.206447 1411 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Unauthorized
Apr 15 08:18:24 master-1 hyperkube[1411]: E0415 08:18:24.404757 1411 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized
Apr 15 08:18:24 master-1 hyperkube[1411]: W0415 08:18:24.701927 1411 status_manager.go:485] Failed to get status for pod "openshift-authentication-operator-768788f4f7-t85x6_openshift-authentication-operat>
Apr 15 08:18:25 master-1 hyperkube[1411]: E0415 08:18:25.001594 1411 webhook.go:106] Failed to make webhook authenticator request: Unauthorized
Apr 15 08:18:25 master-1 hyperkube[1411]: E0415 08:18:25.001634 1411 server.go:245] Unable to authenticate the request due to an error: Unauthorized
Apr 15 08:18:25 master-1 hyperkube[1411]: E0415 08:18:25.202356 1411 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Unauthorized
Apr 15 08:18:25 master-1 hyperkube[1411]: E0415 08:18:25.402716 1411 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Unauthorized
Apr 15 08:18:25 master-1 hyperkube[1411]: E0415 08:18:25.602266 1411 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized
Apr 15 08:18:25 master-1 hyperkube[1411]: E0415 08:18:25.802347 1411 webhook.go:106] Failed to make webhook authenticator request: Unauthorized
Apr 15 08:18:25 master-1 hyperkube[1411]: E0415 08:18:25.802576 1411 server.go:245] Unable to authenticate the request due to an error: Unauthorized
~
README says for baremetal testing a NODES_FILE variable pointing to a json file with nodes' details should be provided.
It would be nice to have a sample json file with the required content.
On a clean CoreOS 7.6 system, following README.md instructions, make
fails because of missing directory:
[tester@... dev-scripts]$ make
./01_install_requirements.sh
+ source common.sh
++++ dirname common.sh
+++ cd .
+++ pwd
++ SCRIPTDIR=/home/tester/dev-scripts
+++ whoami
++ USER=tester
++ '[' -z '' ']'
++ '[' -f /home/tester/dev-scripts/config_tester.sh ']'
++ echo 'Using CONFIG /home/tester/dev-scripts/config_tester.sh'
Using CONFIG /home/tester/dev-scripts/config_tester.sh
++ CONFIG=/home/tester/dev-scripts/config_tester.sh
++ source /home/tester/dev-scripts/config_tester.sh
+++ set +x
++ ADDN_DNS=
++ EXT_IF=
++ PRO_IF=
++ INT_IF=
++ ROOT_DISK=/dev/vda
++ FILESYSTEM=/
++ WORKING_DIR=/opt/dev-scripts
++ NODES_FILE=/opt/dev-scripts/ironic_nodes.json
++ NODES_PLATFORM=libvirt
++ MASTER_NODES_FILE=ocp/master_nodes.json
++ export RHCOS_IMAGE_URL=https://releases-rhcos.svc.ci.openshift.org/storage/releases/maipo/
++ RHCOS_IMAGE_URL=https://releases-rhcos.svc.ci.openshift.org/storage/releases/maipo/
++ export RHCOS_IMAGE_VERSION=47.284
++ RHCOS_IMAGE_VERSION=47.284
++ export RHCOS_IMAGE_NAME=redhat-coreos-maipo-47.284
++ RHCOS_IMAGE_NAME=redhat-coreos-maipo-47.284
++ export RHCOS_IMAGE_FILENAME=redhat-coreos-maipo-47.284-qemu.qcow2
++ RHCOS_IMAGE_FILENAME=redhat-coreos-maipo-47.284-qemu.qcow2
++ export RHCOS_IMAGE_FILENAME_OPENSTACK=redhat-coreos-maipo-47.284-openstack.qcow2
++ RHCOS_IMAGE_FILENAME_OPENSTACK=redhat-coreos-maipo-47.284-openstack.qcow2
++ export RHCOS_IMAGE_FILENAME_DUALDHCP=redhat-coreos-maipo-47.284-dualdhcp.qcow2
++ RHCOS_IMAGE_FILENAME_DUALDHCP=redhat-coreos-maipo-47.284-dualdhcp.qcow2
++ export RHCOS_IMAGE_FILENAME_LATEST=redhat-coreos-maipo-latest.qcow2
++ RHCOS_IMAGE_FILENAME_LATEST=redhat-coreos-maipo-latest.qcow2
++ export IRONIC_IMAGE=quay.io/metalkube/metalkube-ironic
++ IRONIC_IMAGE=quay.io/metalkube/metalkube-ironic
++ export IRONIC_INSPECTOR_IMAGE=quay.io/metalkube/metalkube-ironic-inspector
++ IRONIC_INSPECTOR_IMAGE=quay.io/metalkube/metalkube-ironic-inspector
++ export IRONIC_DATA_DIR=/opt/dev-scripts/ironic
++ IRONIC_DATA_DIR=/opt/dev-scripts/ironic
++ export KUBECONFIG=/home/tester/dev-scripts/ocp/auth/kubeconfig
++ KUBECONFIG=/home/tester/dev-scripts/ocp/auth/kubeconfig
++ export 'SSH=ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=5'
++ SSH='ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=5'
++ export LIBVIRT_DEFAULT_URI=qemu:///system
++ LIBVIRT_DEFAULT_URI=qemu:///system
++ '[' tester '!=' root -a '' == /run/user/0 ']'
++ [[ 1001 -eq 0 ]]
++ sudo -n uptime
+++ awk -F= '/^ID=/ { print $2 }' /etc/os-release
+++ tr -d '"'
++ [[ centos != \c\e\n\t\o\s ]]
+++ awk -F= '/^VERSION_ID=/ { print $2 }' /etc/os-release
+++ tr -d '"'
++ [[ 7 -ne 7 ]]
+++ df / --output=fstype
+++ grep -v Type
++ FSTYPE=xfs
++ case ${FSTYPE} in
+++ xfs_info /
+++ grep -q ftype=1
++ [[ -n '' ]]
++ '[' 2710 = 0 ']'
++ '[' '!' -d /opt/dev-scripts ']'
++ echo 'Creating Working Dir'
Creating Working Dir
++ mkdir /opt/dev-scripts
mkdir: cannot create directory โ/opt/dev-scriptsโ: Permission denied
make: *** [requirements] Error 1
When I create it manually and rerun make
, it proceeds further.
When we deploy the baremetal-operator using dev-scripts, it needs to be configured to talk to Ironic on the provisioning host until we have the baremetal-operator updated to run Ironic itself inside the cluster.
Right now it assumes it can reach Ironic on localhost, which fails when running the BMO as a pod.
{"level":"error","ts":1554831354.3558333,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"metalkube-baremetalhost-controller","request":"openshift-machine-api/openshift-worker-0","error":"phase validate access failed: failed to validate BMC access: failed to find existing host: failed to find node by name openshift-worker-0: Get http://localhost:6385/v1/nodes/openshift-worker-0: dial tcp [::1]:6385: connect: connection refused","errorVerbose":"Get http://localhost:6385/v1/nodes/openshift-worker-0: dial tcp [::1]:6385: connect: connection refused\nfailed to find node by name openshift-worker-0\ngithub.com/metalkube/baremetal-operator/pkg/provisioner/ironic.(*ironicProvisioner).findExistingHost\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/provisioner/ironic/ironic.go:145\ngithub.com/metalkube/baremetal-operator/pkg/provisioner/ironic.(*ironicProvisioner).ValidateManagementAccess\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/provisioner/ironic/ironic.go:161\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).phaseValidateAccess\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:403\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).(github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.phaseValidateAccess)-fm\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:268\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).Reconcile\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:283\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:213\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2361\nfailed to find existing host\ngithub.com/metalkube/baremetal-operator/pkg/provisioner/ironic.(*ironicProvisioner).ValidateManagementAccess\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/provisioner/ironic/ironic.go:163\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).phaseValidateAccess\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:403\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).(github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.phaseValidateAccess)-fm\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:268\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).Reconcile\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:283\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:213\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2361\nfailed to validate BMC access\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).phaseValidateAccess\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:405\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).(github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.phaseValidateAccess)-fm\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:268\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).Reconcile\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:283\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:213\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2361\nphase validate access failed\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).Reconcile\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:285\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:213\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2361","stacktrace":"github.com/metalkube/baremetal-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
while executing 'make clean' one of the task is failing 'TASK [teardown/user : Remove virt_power_key from remote authorized_keys]" fatal: [localhost]: FAILED! => {
"msg": "could not locate file in lookup: /home/bkr/.quickstart/id_rsa_virt_power.pub"
}
also I have observed that
'make clean' does not delete the baremetal and provisioning bridge
due to which when I execute 'make' it fails with msg": "error creating bridge interface baremetal: File exists"
[bkr@vm220 dev-scripts]$ brctl show
bridge name bridge id STP enabled interfaces
baremetal 8000.000000000000 no
provisioning 8000.000000000000 no
virbr0 8000.5254002e4699 yes virbr0-nic
I need to manually remove the bridges to make run successful.
[bkr@vm220 dev-scripts]$ sudo ip link delete baremetal type bridge
[bkr@vm220 dev-scripts]$ sudo ip link delete provisioning type bridge
[#Minor] Make Clean script is not removing /opt/dev-scripts directory .
Though this issue is minor one.
When I tried to re-run the make script with user "user2" my installation fail as user2 does not have permission to write files inside /opt/dev-script/ironic folder ,as this folder is owned by pervious user "bkr"
"make" failed with error message:
Writing manifest to image destination
Storing signatures
63c4f7bb2f7917d7397796f2e12dd36a8e6920a8c66716307f36afd55debb6ae
+ pushd /opt/dev-scripts/ironic/html/images
/opt/dev-scripts/ironic/html/images ~/dev-scripts
+ '[' '!' -e rhcos-410.8.20190410.0-compressed.qcow2 ']'
+ '[' '!' -e rhcos-410.8.20190410.0-compressed.qcow2.md5sum -o rhcos-410.8.20190410.0-compressed.qcow2 -nt rhcos-410.8.20190410.0-compressed.qcow2.md5sum ']'
+ ln -sf rhcos-410.8.20190410.0-compressed.qcow2 rhcos-ootpa-latest.qcow2
ln: cannot remove โrhcos-ootpa-latest.qcow2โ: Permission denied
make: *** [ironic] Error 1
Content of this directory are:
ls -l /opt/dev-scripts/
total 8
drwxrwxr-x. 4 **bkr bkr** 34 Apr 2 14:57 ironic
-rw-r--r--. 1 root root 3865 Apr 11 19:20 ironic_nodes.json
drwxr-xr-x. 2 root root 210 Apr 12 14:02 pool
-rw-r--r--. 1 root root 222 Apr 2 14:50 volume_pool.xml
Attaching the host_cleanup logs.
host_cleanup-2019-04-12-142513.log
Regarding to this PR #245
I've facing an error on DNS resolution and also on TLS validation:
Logs from openshift-authentication pod:
entication.svc.cluster.local] issuer="openshift-service-serving-signer@1554149617" (2019-04-01 20:21:17 +0000 UTC to 2021-03-31 20:21:18 +0000 UTC (now=2019-04-01 20:21:48.182068939 +0000 UTC))
I0401 20:21:48.182239 1 serving.go:195] [1] "/var/config/system/secrets/v4-0-config-system-serving-cert/tls.crt" serving certificate: "openshift-service-serving-signer@1554149617" [] issuer="<self>" (2019-04-01 20:13:36 +0000 UTC to 2020-03-31 20:13:37 +0000 UTC (now=2019-04-01 20:21:48.182226129 +0000 UTC))
I0401 20:21:48.182379 1 secure_serving.go:125] Serving securely on 0.0.0.0:6443
I0401 20:21:48.182483 1 serving.go:77] Starting DynamicLoader
I0401 21:30:50.704480 1 log.go:172] http: TLS handshake error from 10.128.0.1:59540: remote error: tls: unknown certificate
I0401 21:47:35.891348 1 log.go:172] http: TLS handshake error from 10.128.0.1:47572: remote error: tls: unknown certificate
I0401 22:11:13.559976 1 log.go:172] http: TLS handshake error from 10.128.0.1:42264: remote error: tls: unknown certificate
I0401 22:15:50.470017 1 log.go:172] http: TLS handshake error from 10.128.0.1:46784: remote error: tls: unknown certificate
I0401 22:19:35.864342 1 log.go:172] http: TLS handshake error from 10.128.0.1:50434: remote error: tls: unknown certificate
I0401 22:23:27.825915 1 log.go:172] http: TLS handshake error from 10.128.0.1:54202: remote error: tls: unknown certificate
I0401 22:26:29.398364 1 log.go:172] http: TLS handshake error from 10.128.0.1:57154: remote error: tls: unknown certificate
Trying to reach openshift-console:
[jparrill@sjr3 dev-scripts]$ until wget --no-check-certificate https://console-openshift-console.apps.apps.ostest.test.metalkube.org; do sleep 10 && echo "rechecking"; done
--2019-04-01 23:00:25-- https://console-openshift-console.apps.apps.ostest.test.metalkube.org/
Resolving console-openshift-console.apps.apps.ostest.test.metalkube.org (console-openshift-console.apps.apps.ostest.test.metalkube.org)... 192.168.111.21
Connecting to console-openshift-console.apps.apps.ostest.test.metalkube.org (console-openshift-console.apps.apps.ostest.test.metalkube.org)|192.168.111.21|:443... connected.
WARNING: cannot verify console-openshift-console.apps.apps.ostest.test.metalkube.org's certificate, issued by โ/CN=cluster-ingress-operator@1554150077โ:
Self-signed certificate encountered.
WARNING: no certificate subject alternative name matches
requested host name โconsole-openshift-console.apps.apps.ostest.test.metalkube.orgโ.
HTTP request sent, awaiting response... 503 Service Unavailable
2019-04-01 23:00:25 ERROR 503: Service Unavailable.
rechecking
--2019-04-01 23:00:35-- https://console-openshift-console.apps.apps.ostest.test.metalkube.org/
Resolving console-openshift-console.apps.apps.ostest.test.metalkube.org (console-openshift-console.apps.apps.ostest.test.metalkube.org)... 192.168.111.21
Connecting to console-openshift-console.apps.apps.ostest.test.metalkube.org (console-openshift-console.apps.apps.ostest.test.metalkube.org)|192.168.111.21|:443... connected.
WARNING: cannot verify console-openshift-console.apps.apps.ostest.test.metalkube.org's certificate, issued by โ/CN=cluster-ingress-operator@1554150077โ:
Self-signed certificate encountered.
WARNING: no certificate subject alternative name matches
requested host name โconsole-openshift-console.apps.apps.ostest.test.metalkube.orgโ.
HTTP request sent, awaiting response... 503 Service Unavailable
2019-04-01 23:00:35 ERROR 503: Service Unavailable.
[jparrill@sjr3 dev-scripts]$ oc get csr
No resources found.
Those traces comes from the Libvirt node which contains the master nodes
Ideas?
All of our deployments have a problem where certs aren't being approved automatically, and the following command is needed as a workaround:
oc get csr -o name | xargs -n 1 oc adm certificate approve
Other PRs / issues tracked through this investigation:
From #135
With this PR, Yurii is seeing:
OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'SELECT nodes.created_at
even with:
$ podman exec -it ironic sqlite3 /var/lib/ironic/ironic.db โPRAGMA journal_modeโ wal
Dmitri suggests:
So, our last resort (?) option with sqlite is to try setting
busy_timeout
.
Failed like:
ironic_node_v1.openshift-master-2: Still creating... (2m50s elapsed)
ironic_node_v1.openshift-master-0: Still creating... (2m50s elapsed)
ironic_node_v1.openshift-master-1: Still creating... (2m50s elapsed)
2019/03/15 10:29:45 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:
* ironic_node_v1.openshift-master-1: Internal Server Error
2019/03/15 10:29:45 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error(s) occurred:
* ironic_node_v1.openshift-master-1: Internal Server Error
2019/03/15 10:29:45 [TRACE] [walkApply] Exiting eval tree: ironic_node_v1.openshift-master-1
2019/03/15 10:29:48 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:
* ironic_node_v1.openshift-master-2: Internal Server Error
2019/03/15 10:29:48 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error(s) occurred:
* ironic_node_v1.openshift-master-2: Internal Server Error
2019/03/15 10:30:02 [DEBUG] plugin: waiting for all plugin processes to complete...
Error: Error applying plan:
3 error(s) occurred:
* ironic_node_v1.openshift-master-0: 1 error(s) occurred:
* ironic_node_v1.openshift-master-0: Internal Server Error
* ironic_node_v1.openshift-master-1: 1 error(s) occurred:
* ironic_node_v1.openshift-master-1: Internal Server Error
* ironic_node_v1.openshift-master-2: 1 error(s) occurred:
2019-03-15T10:30:02.972+0200 [DEBUG] plugin.terraform-provider-ironic: 2019/03/15 10:30:02 [ERR] plugin: stream copy 'stderr' error: stream closed
* ironic_node_v1.openshift-master-2: Internal Server Error
Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
2019-03-15T10:30:03.007+0200 [DEBUG] plugin.terraform-provider-ironic: 2019/03/15 10:30:03 [ERR] plugin: plugin server: accept unix /tmp/plugin005349256: use of closed network connection
2019-03-15T10:30:03.008+0200 [DEBUG] plugin: plugin process exited: path=/root/.terraform.d/plugins/terraform-provider-ironic
2019-03-15 08:29:25.649 44 ERROR wsme.api [req-747fc5e4-6050-463e-9d5d-8b7fa79a00f3 - - - - -] Server-side error: "(sqlite3.OperationalError) database is locked [SQL: u'SELECT anon_1.nodes_created_at AS anon_1_nodes_created_at, anon_1.nodes_updated_at AS anon_1_nodes_updated_at, anon_1.nodes_version AS anon_1_nodes_version, anon_1.nodes_id AS anon_1_nodes_id, anon_1.nodes_uuid AS anon_1_nodes_uuid, anon_1.nodes_instance_uuid AS anon_1_nodes_instance_uuid, anon_1.nodes_name AS anon_1_nodes_name, anon_1.nodes_chassis_id AS anon_1_nodes_chassis_id, anon_1.nodes_power_state AS anon_1_nodes_power_state, anon_1.nodes_target_power_state AS anon_1_nodes_target_power_state, anon_1.nodes_provision_state AS anon_1_nodes_provision_state, anon_1.nodes_target_provision_state AS anon_1_nodes_target_provision_state, anon_1.nodes_provision_updated_at AS anon_1_nodes_provision_updated_at, anon_1.nodes_last_error AS anon_1_nodes_last_error, anon_1.nodes_instance_info AS anon_1_nodes_instance_info, anon_1.nodes_properties AS anon_1_nodes_properties, anon_1.nodes_driver AS anon_1_nodes_driver, anon_1.nodes_driver_info AS anon_1_nodes_driver_info, anon_1.nodes_driver_internal_info AS anon_1_nodes_driver_internal_info, anon_1.nodes_clean_step AS anon_1_nodes_clean_step, anon_1.nodes_deploy_step AS anon_1_nodes_deploy_step, anon_1.nodes_resource_class AS anon_1_nodes_resource_class, anon_1.nodes_raid_config AS anon_1_nodes_raid_config, anon_1.nodes_target_raid_config AS anon_1_nodes_target_raid_config, anon_1.nodes_reservation AS anon_1_nodes_reservation, anon_1.nodes_conductor_affinity AS anon_1_nodes_conductor_affinity, anon_1.nodes_conductor_group AS anon_1_nodes_conductor_group, anon_1.nodes_maintenance AS anon_1_nodes_maintenance, anon_1.nodes_maintenance_reason AS anon_1_nodes_maintenance_reason, anon_1.nodes_fault AS anon_1_nodes_fault, anon_1.nodes_console_enabled AS anon_1_nodes_console_enabled, anon_1.nodes_inspection_finished_at AS anon_1_nodes_inspection_finished_at, anon_1.nodes_inspection_started_at AS anon_1_nodes_inspection_started_at, anon_1.nodes_extra AS anon_1_nodes_extra, anon_1.nodes_automated_clean AS anon_1_nodes_automated_clean, anon_1.nodes_protected AS anon_1_nodes_protected, anon_1.nodes_protected_reason AS anon_1_nodes_protected_reason, anon_1.nodes_owner AS anon_1_nodes_owner, anon_1.nodes_allocation_id AS anon_1_nodes_allocation_id, anon_1.nodes_description AS anon_1_nodes_description, anon_1.nodes_bios_interface AS anon_1_nodes_bios_interface, anon_1.nodes_boot_interface AS anon_1_nodes_boot_interface, anon_1.nodes_console_interface AS anon_1_nodes_console_interface, anon_1.nodes_deploy_interface AS anon_1_nodes_deploy_interface, anon_1.nodes_inspect_interface AS anon_1_nodes_inspect_interface, anon_1.nodes_management_interface AS anon_1_nodes_management_interface, anon_1.nodes_network_interface AS anon_1_nodes_network_interface, anon_1.nodes_raid_interface AS anon_1_nodes_raid_interface, anon_1.nodes_rescue_interface AS anon_1_nodes_rescue_interface, anon_1.nodes_storage_interface AS anon_1_nodes_storage_interface, anon_1.nodes_power_interface AS anon_1_nodes_power_interface, anon_1.nodes_vendor_interface AS anon_1_nodes_vendor_interface, node_traits_1.created_at AS node_traits_1_created_at, node_traits_1.updated_at AS node_traits_1_updated_at, node_traits_1.version AS node_traits_1_version, node_traits_1.node_id AS node_traits_1_node_id, node_traits_1.trait AS node_traits_1_trait, node_tags_1.created_at AS node_tags_1_created_at, node_tags_1.updated_at AS node_tags_1_updated_at, node_tags_1.version AS node_tags_1_version, node_tags_1.node_id AS node_tags_1_node_id, node_tags_1.tag AS node_tags_1_tag \nFROM (SELECT nodes.created_at AS nodes_created_at, nodes.updated_at AS nodes_updated_at, nodes.version AS nodes_version, nodes.id AS nodes_id, nodes.uuid AS nodes_uuid, nodes.instance_uuid AS nodes_instance_uuid, nodes.name AS nodes_name, nodes.chassis_id AS nodes_chassis_id, nodes.power_state AS nodes_power_state, nodes.target_power_state AS nodes_target_power_state, nodes.provision_state AS nodes_provision_state, nodes.target_provision_state AS nodes_target_provision_state, nodes.provision_updated_at AS nodes_provision_updated_at, nodes.last_error AS nodes_last_error, nodes.instance_info AS nodes_instance_info, nodes.properties AS nodes_properties, nodes.driver AS nodes_driver, nodes.driver_info AS nodes_driver_info, nodes.driver_internal_info AS nodes_driver_internal_info, nodes.clean_step AS nodes_clean_step, nodes.deploy_step AS nodes_deploy_step, nodes.resource_class AS nodes_resource_class, nodes.raid_config AS nodes_raid_config, nodes.target_raid_config AS nodes_target_raid_config, nodes.reservation AS nodes_reservation, nodes.conductor_affinity AS nodes_conductor_affinity, nodes.conductor_group AS nodes_conductor_group, nodes.maintenance AS nodes_maintenance, nodes.maintenance_reason AS nodes_maintenance_reason, nodes.fault AS nodes_fault, nodes.console_enabled AS nodes_console_enabled, nodes.inspection_finished_at AS nodes_inspection_finished_at, nodes.inspection_started_at AS nodes_inspection_started_at, nodes.extra AS nodes_extra, nodes.automated_clean AS nodes_automated_clean, nodes.protected AS nodes_protected, nodes.protected_reason AS nodes_protected_reason, nodes.owner AS nodes_owner, nodes.allocation_id AS nodes_allocation_id, nodes.description AS nodes_description, nodes.bios_interface AS nodes_bios_interface, nodes.boot_interface AS nodes_boot_interface, nodes.console_interface AS nodes_console_interface, nodes.deploy_interface AS nodes_deploy_interface, nodes.inspect_interface AS nodes_inspect_interface, nodes.management_interface AS nodes_management_interface, nodes.network_interface AS nodes_network_interface, nodes.raid_interface AS nodes_raid_interface, nodes.rescue_interface AS nodes_rescue_interface, nodes.storage_interface AS nodes_storage_interface, nodes.power_interface AS nodes_power_interface, nodes.vendor_interface AS nodes_vendor_interface \nFROM nodes ORDER BY nodes.id ASC\n LIMIT ? OFFSET ?) AS anon_1 LEFT OUTER JOIN node_traits AS node_traits_1 ON node_traits_1.node_id = anon_1.nodes_id LEFT OUTER JOIN node_tags AS node_tags_1 ON node_tags_1.node_id = anon_1.nodes_id ORDER BY anon_1.nodes_id ASC'] [parameters: (1000, 0)] (Background on this error at: http://sqlalche.me/e/e3q8)". Detail:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/wsmeext/pecan.py", line 85, in callfunction
result = f(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/ironic/api/controllers/v1/node.py", line 1872, in get_all
**extra_args)
File "/usr/lib/python2.7/site-packages/ironic/api/controllers/v1/node.py", line 1684, in _get_nodes_collection
filters=filters)
File "/usr/lib/python2.7/site-packages/ironic/objects/node.py", line 313, in list
sort_dir=sort_dir)
File "/usr/lib/python2.7/site-packages/ironic/db/sqlalchemy/api.py", line 400, in get_node_list
sort_key, sort_dir, query)
File "/usr/lib/python2.7/site-packages/ironic/db/sqlalchemy/api.py", line 229, in _paginate_query
return query.all()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, in all
return list(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3081, in __iter__
return self._execute_and_instances(context)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 980, in execute
return meth(self, multiparams, params)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
distilled_params,
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
e, statement, parameters, cursor, context
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception
util.raise_from_cause(newraise, exc_info)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
cursor, statement, parameters, context
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
cursor.execute(statement, parameters)
Deployment times out: Still waiting for the Kubernetes API: Get https://api.ostest.test.metalkube.org:6443/version?timeout=32s: dial tcp 192.168.123.5:6443: connect: connection refused
Logs of the discovery container on master nodes reveal that querying the DNS VIP cannot resolve _etcd-server-ssl._tcp.ostest.test.metalkube.org :
[root@master-0 core]# crictl logs -f $(crictl ps | awk '/discovery/ {print $1}')
I0409 18:33:02.044434 1 run.go:46] Version: 4.0.0-alpha.0-162-g6276485e-dirty
E0409 18:33:02.049842 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:33:02.052979 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:33:32.056696 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:33:32.059669 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:34:02.056626 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:34:02.059876 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:34:32.056434 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:34:32.059145 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:35:02.056681 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:35:02.059569 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:35:32.057436 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:35:32.061139 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:36:02.057464 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:36:02.060341 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:36:32.055545 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:36:32.058743 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:37:02.056657 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such host
E0409 18:37:02.059077 1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.ostest.test.metalkube.org on 192.168.123.6:53: no such ho
It seems in baremetal deployments it is required to run:
#!/usr/bin/env bash
set -x
set -e
source logging.sh
source utils.sh
source common.sh
source ocp_install_env.sh
# Update kube-system ep/host-etcd used by cluster-kube-apiserver-operator to
# generate storageConfig.urls
patch_ep_host_etcd "$CLUSTER_DOMAIN"
While the cluster is installing in the 'Waiting up to 1h0m0s for the bootstrap-complete event...' step:
level=debug msg="Still waiting for the Kubernetes API: Get https://api.kni1.cloud.lab.eng.bos.redhat.com:6443/version?timeout=32s: dial tcp 10.19.138.14:6443: connect: connection refused"
level=info msg="API v1.13.4+938b976 up"
level=info msg="Waiting up to 1h0m0s for the bootstrap-complete event..."
level=debug msg="added kube-controller-manager.1597d0ec8231f284: localhost.localdomain_a4283191-6507-11e9-93be-2af39db2a27a became leader"
level=debug msg="added kube-scheduler.1597d0ecbce9cf55: localhost.localdomain_a426c179-6507-11e9-ada6-2af39db2a27a became leader"
The scripts assume that the user 'stack' will be used to run the process but there's no check to verify and/or enforce that it's the actual user running the commands.
This could cause some issues during the process, e.g.
cat: /home/stack/ironic_nodes.json: Permission denied
due to the stack home dir being set by default at 700
A workaround exists for this issue, but others could arise.
In general, having the stack user hardcoded is not ideal.
After some time of cluster being up it starts to fail with errors like:
Apr 12 09:48:31 master-1 hyperkube[23740]: E0412 09:48:31.776526 23740 kubelet.go:2273] node "master-1" not found
Apr 12 09:48:31 master-1 hyperkube[23740]: E0412 09:48:31.843612 23740 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/ap
i/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.123.5:6443: connect: invalid argument
Apr 12 09:48:31 master-1 hyperkube[23740]: E0412 09:48:31.844422 23740 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.ostest.test.metalkube.org:
6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.123.5:6443: connect: invalid argument
Apr 12 09:48:31 master-1 hyperkube[23740]: E0412 09:48:31.876801 23740 kubelet.go:2273] node "master-1" not found
Apr 12 09:48:31 master-1 hyperkube[23740]: E0412 09:48:31.977082 23740 kubelet.go:2273] node "master-1" not found
Apr 12 09:48:32 master-1 hyperkube[23740]: E0412 09:48:32.054813 23740 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443
/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.123.5:6443: connect: invalid argument
Attempt to ping this ip from any other master nodes(except the one that hosts the ip) fails:
[root@master-1 ~]# ip a | grep 192.168.123.5
[root@master-1 ~]# ping 192.168.123.5
connect: Invalid argument
[root@master-1 ~]# curl -4kvL 192.168.123.5:6443
* Rebuilt URL to: 192.168.123.5:6443/
* Trying 192.168.123.5...
* TCP_NODELAY set
* Immediate connect fail for 192.168.123.5: Invalid argument
* Closing connection 0
curl: (7) Couldn't connect to server
[root@master-1 ~]# curl -4kvL api.ostest.test.metalkube.org:6443
* Rebuilt URL to: api.ostest.test.metalkube.org:6443/
* Trying 192.168.123.5...
* TCP_NODELAY set
* Immediate connect fail for 192.168.123.5: Invalid argument
* Closing connection 0
curl: (7) Couldn't connect to server
[root@master-1 ~]#
[root@master-2 ~]# ip a | grep 192.168.123.5
[root@master-2 ~]# ping 192.168.123.5
connect: Invalid argument
[root@master-2 ~]# curl -4kvL 192.168.123.5:6443
* Rebuilt URL to: 192.168.123.5:6443/
* Trying 192.168.123.5...
* TCP_NODELAY set
* Immediate connect fail for 192.168.123.5: Invalid argument
* Closing connection 0
curl: (7) Couldn't connect to server
[root@master-2 ~]# curl -4kvL api.ostest.test.metalkube.org:6443
* Rebuilt URL to: api.ostest.test.metalkube.org:6443/
* Trying 192.168.123.5...
* TCP_NODELAY set
* Immediate connect fail for 192.168.123.5: Invalid argument
* Closing connection 0
curl: (7) Couldn't connect to server
[root@master-2 ~]#
and from master-0 that handles the API VIP IP
[root@master-0 ~]# ip a |grep 192.168.123.5
inet 192.168.123.5/32 scope global ens4
[root@master-0 ~]# curl -4klv api.ostest.test.metalkube.org:6443
* Rebuilt URL to: api.ostest.test.metalkube.org:6443/
* Trying 192.168.123.5...
* TCP_NODELAY set
* Connected to api.ostest.test.metalkube.org (192.168.123.5) port 6443 (#0)
> GET / HTTP/1.1
> Host: api.ostest.test.metalkube.org:6443
> User-Agent: curl/7.61.1
> Accept: */*
>
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
* Failed writing body (0 != 7)
* Closing connection 0
[root@master-0 ~]# curl -4klv 192.168.123.5:6443
* Rebuilt URL to: 192.168.123.5:6443/
* Trying 192.168.123.5...
* TCP_NODELAY set
* Connected to 192.168.123.5 (192.168.123.5) port 6443 (#0)
> GET / HTTP/1.1
> Host: 192.168.123.5:6443
> User-Agent: curl/7.61.1
> Accept: */*
>
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
* Failed writing body (0 != 7)
* Closing connection 0
[root@master-0 ~]#
$ oc explain baremetalhosts.metalkube.org
KIND: BareMetalHost
VERSION: metalkube.org/v1alpha1
DESCRIPTION:
<empty>
$ oc get baremetalhosts --all-namespaces
No resources found.
$ oc version
Client Version: version.Info{Major:"4", Minor:"0+", GitVersion:"v4.0.22", GitCommit:"219bbe2f0c", GitTreeState:"", BuildDate:"2019-03-10T22:23:11Z", GoVersion:"", Compiler:"", Platform:""}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.4+eae8710", GitCommit:"eae8710", GitTreeState:"clean", BuildDate:"2019-03-30T18:01:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
$ git rev-parse HEAD
31b1359f92750d290fcc972350bc3a1c2dd6ebda
$ oc get nodes
NAME STATUS ROLES AGE VERSION
master-0 Ready master,worker 3h18m v1.12.4+50c2f2340a
master-1 Ready master,worker 3h18m v1.12.4+50c2f2340a
master-2 Ready master,worker 3h18m v1.12.4+50c2f2340a
Is the baremetal crd only for workers or nodes deployed with the baremetal operator?
On commit 0fb6eb9 using nest virt running first step, "make".
level=debug msg="module.masters.ironic_node_v1.openshift-master-0: Still creating... (31m31s elapsed)"
level=debug msg="module.masters.ironic_node_v1.openshift-master-0: Still creating... (31m41s elapsed)"
level=error
level=error msg="Error: Error applying plan:"
level=error
level=error msg="1 error occurred:"
level=error msg="\t* module.masters.ironic_node_v1.openshift-master-0: 1 error occurred:"
level=error msg="\t* ironic_node_v1.openshift-master-0: cannot go from state 'deploy failed' to state 'manageable'"
level=error
level=error
level=error
level=error
level=error
level=error msg="Terraform does not automatically rollback in the face of errors."
level=error msg="Instead, your Terraform state file has been partially updated with"
level=error msg="any resources that successfully completed. Please address the error"
level=error msg="above and apply again to incrementally change your infrastructure."
level=error
level=error
level=fatal msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failed to create cluster: failed to apply using Terraform"
make: *** [ocp_run] Error 1
We are no longer using virbr0, so
https://github.com/metalkube/dev-scripts/blob/master/02_configure_host.sh#L54
Needs to be updated.
Ironic timeout/failure/error on ./06_create_cluster.sh (by performing make
).
level=debug msg="2019/04/09 13:03:33 [ERROR] root.masters: eval: *terraform.EvalApplyPost, err: 1 error occurred:"
level=debug msg="\t* ironic_node_v1.openshift-master-1: Internal Server Error"
level=debug
level=debug msg="2019/04/09 13:03:33 [ERROR] root.masters: eval: *terraform.EvalSequence, err: 1 error occurred:"
level=debug msg="\t* ironic_node_v1.openshift-master-1: Internal Server Error"
level=debug
level=debug msg="2019/04/09 13:03:33 [ERROR] root.masters: eval: *terraform.EvalApplyPost, err: 1 error occurred:"
level=debug msg="\t* ironic_node_v1.openshift-master-2: Internal Server Error"
level=debug
level=debug msg="2019/04/09 13:03:33 [ERROR] root.masters: eval: *terraform.EvalSequence, err: 1 error occurred:"
level=debug msg="\t* ironic_node_v1.openshift-master-2: Internal Server Error"
level=debug
level=debug msg="2019/04/09 13:03:33 [ERROR] root.masters: eval: *terraform.EvalApplyPost, err: 1 error occurred:"
level=debug msg="\t* ironic_node_v1.openshift-master-0: Internal Server Error"
level=debug
level=debug msg="2019/04/09 13:03:33 [ERROR] root.masters: eval: *terraform.EvalSequence, err: 1 error occurred:"
level=debug msg="\t* ironic_node_v1.openshift-master-0: Internal Server Error"
level=debug
level=error
level=error msg="Error: Error applying plan:"
level=error
level=error msg="3 errors occurred:"
level=error msg="\t* module.masters.ironic_node_v1.openshift-master-1: 1 error occurred:"
level=error msg="\t* ironic_node_v1.openshift-master-1: Internal Server Error"
level=error
level=error
level=error msg="\t* module.masters.ironic_node_v1.openshift-master-2: 1 error occurred:"
level=error msg="\t* ironic_node_v1.openshift-master-2: Internal Server Error"
level=error
level=error
level=error msg="\t* module.masters.ironic_node_v1.openshift-master-0: 1 error occurred:"
level=error msg="\t* ironic_node_v1.openshift-master-0: Internal Server Error"
level=error
level=error
level=error
level=error
level=error
level=error msg="Terraform does not automatically rollback in the face of errors."
level=error msg="Instead, your Terraform state file has been partially updated with"
level=error msg="any resources that successfully completed. Please address the error"
level=error msg="above and apply again to incrementally change your infrastructure."
level=error
level=error
level=fatal msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failed to create cluster: failed to apply using Terraform"
32GB RAM
856GB Hard Drive Free
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.