openshift / must-gather Goto Github PK
View Code? Open in Web Editor NEWA client tool for gathering information about an operator managed component.
License: Apache License 2.0
A client tool for gathering information about an operator managed component.
License: Apache License 2.0
I'd like for the console output of the oc adm must-gather
command to include copy / pastable summary of the top pieces of data necessary. These should be the key pieces of information that we feel every single bug or case must include for highest level triage. Something like the following.
Output in must-gather-xyz.tar.gz
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information.
Cluster Version: 4.5.11
Platform: AWS
Degraded Operators: openshift-rocketship
We should restructure the commands / plugins so that config.openshift.io is its own plugin.
Getting below errors , while using in a IPI GCP private cluster env
.
.
rsync: connection unexpectedly closed (122395578 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [receiver=3.1.3]
rsync: connection unexpectedly closed (70267 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [generator=3.1.3]
.
.
As part of our tekton (openshift pipeline) we have a must-gather task that runs if there are test case failures. Currently this task runs the following command oc adm must-gather --image=quay.io/opencloudio/must-gather:4.5.1 -- gather -n ibm-common-services,<namespace where product under test is installed to>
. This command takes about 30 minutes to complete which greatly extends the completion time of our pipeline runs.
According to the article below by default the following modules are run: overview, system, failure, ocp, cloudpak. I was hoping by only running one mode(failure) it would run faster but it was only one minute faster.
https://www.ibm.com/support/knowledgecenter/SSHKN6/support/must-gather/1.x.x/overview.html
I ran oc adm must-gather and it took 27 minutes to complete
I ran oc adm must-gather -- gather -m failure and it took 26 minutes to complete
Are there any ways to shorten the time that must-gather takes to run?
When running must-gather - the actions fails, specifying the following errors:
$ oc adm must-gather
namespace/openshift-must-gather-pw995 created
clusterrolebinding.rbac.authorization.k8s.io/must-gather-n7jvb created
container logs unavailable: Get https://host-172-16-0-21:10250/containerLogs/openshift-must-gather-pw995/must-gather-2k9gh/gather?follow=true: remote error: tls: internal errorWARNING: cannot use rsync: rsync not available in container
WARNING: cannot use tar: tar not available in container
clusterrolebinding.rbac.authorization.k8s.io/must-gather-n7jvb deleted
namespace/openshift-must-gather-pw995 deleted
error: No available strategies to copy.
Running on a cluster with openshift 4.2 installed.
fail [github.com/openshift/origin/test/extended/cli/mustgather.go:73]: Unexpected error:
<*errors.errorString | 0xc003384320>: {
s: "expected files should not be empty: /tmp/test.oc-adm-must-gather.270681703/audit_logs/openshift-apiserver.audit_logs_listing",
}
expected files should not be empty: /tmp/test.oc-adm-must-gather.270681703/audit_logs/openshift-apiserver.audit_logs_listing
occurred
Currently I have made a custom must gather image at https://github.com/appsody/appsody-operator/tree/master/must-gather but the image is not supported on multiple architectures. It runs on linux but not linux-ppc64 or linux-s390. Please consider adding support for these architectures or insight on how to change the dockerfile at https://github.com/appsody/appsody-operator/blob/master/must-gather/Dockerfile.
Hello,
since today we can't pull openshift4/ose-must-gather
on registry.redhat.io
$ podman pull registry.redhat.io/openshift4/ose-must-gather
Trying to pull registry.redhat.io/openshift4/ose-must-gather:latest...
WARN[0001] Failed, retrying in 1s ... (1/3).
Error: determining manifest MIME type for docker://registry.redhat.io/openshift4/ose-must-gather:latest: reading manifest sha256:2cf7da4892d34247d7135930021739023a85c635c946649252151e0279686abb in registry.redhat.io/openshift4/ose-must-gather: unsupported: Not Found, or unsupported.
V2 schema 1 manifest digest are no longer supported for image pulls.
Use the equivalent schema 2 manifest digest instead.
For more information see https://access.redhat.com/articles/6138332
Do you know what changed ?
Thank you
Collect similar information than the information shown by oc describe nodes
, like CPU/memory limits and requests per pod, allocated resources in the nodes, real resource usage by pods (maybe also by containers) in must-gather.
Currently, information like the following is shown in an oc describe nodes
output:
[...]
Non-terminated Pods: (100 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
axyz daemonset-example-d4lvf 0 (0%) 0 (0%) 0 (0%) 0 (0%) 29h
axyz deployment-example-862f9x79n-z27bg 0 (0%) 0 (0%) 0 (0%) 0 (0%) 12h
axyz job-example-mxg74 0 (0%) 0 (0%) 0 (0%) 0 (0%) 12h
axyz replication-example-vhz2v 0 (0%) 0 (0%) 0 (0%) 0 (0%) 12h
axyz replication-example-yxz4c 0 (0%) 0 (0%) 0 (0%) 0 (0%) 12h
axyz replication-example-njcv2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 12h
zyxw backend-worker-1-b2bl4 150m (4%) 1 (28%) 50Mi (0%) 300Mi (4%) 7h2m
zyxw system-sphinx-1-xnvz1 80m (2%) 1 (28%) 250Mi (3%) 512Mi (7%) 12h
[...]
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 3364m (96%) 9780m (279%)
memory 7118172800 (99%) 9694Mi (141%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
But in a must-gather, not all namespaces/pods are collected.
That information could help to identify overcommitted nodes, pods without requests/limits, etc.
EDIT: Wrong repo for the PR... can be deleted. (my apologies :)
ip [-6] route doesn't provide any information on route cache entries (RCE). When ICMP "need to fragment" is seen in the wild, a RCE is created. Absent knowing an RCE exists, networking debugging wastes time looking in the wrong area.
While insights-operator archive is significantly smaller than must-gather, it manages to collect data relevant for the cluster that are not collected by must-gather.
Having insights-operator as part of the must-gather would expand the data available for troubleshooting, either to be used directly, or by the recommendation engine for the support.
Error: One or more errors ocurred gathering cluster data:
[one or more errors ocurred while gathering pod-specific data for namespace: openshift-apiserver-operator
one or more errors ocurred while gathering container data for pod openshift-apiserver-operator-77756dfc66-v57tb:
Get https://ip-10-0-4-202:10250/containerLogs/openshift-apiserver-operator/openshift-apiserver-operator-77756dfc66-v57tb/operator?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering pod-specific data for namespace: openshift-apiserver
[one or more errors ocurred while gathering container data for pod apiserver-bmfmt:
Get https://ip-10-0-4-202:10250/containerLogs/openshift-apiserver/apiserver-bmfmt/openshift-apiserver?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering container data for pod apiserver-q2wmq:
Get https://ip-10-0-17-67:10250/containerLogs/openshift-apiserver/apiserver-q2wmq/openshift-apiserver?timestamps=true: dial tcp: lookup ip-10-0-17-67 on 10.0.0.2:53: no such host], one or more errors ocurred while gathering pod-specific data for namespace: openshift-cluster-openshift-controller-manager-operator
one or more errors ocurred while gathering container data for pod openshift-cluster-openshift-controller-manager-operator-5dhl2xg:
Get https://ip-10-0-4-202:10250/containerLogs/openshift-cluster-openshift-controller-manager-operator/openshift-cluster-openshift-controller-manager-operator-5dhl2xg/operator?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering pod-specific data for namespace: openshift-cluster-samples-operator
one or more errors ocurred while gathering container data for pod cluster-samples-operator-7cf8d8cf66-rmdd5:
Get https://ip-10-0-4-202:10250/containerLogs/openshift-cluster-samples-operator/cluster-samples-operator-7cf8d8cf66-rmdd5/cluster-samples-operator?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering pod-specific data for namespace: openshift-dns
[one or more errors ocurred while gathering container data for pod dns-default-lklv8:
[Get https://ip-10-0-149-151:10250/containerLogs/openshift-dns/dns-default-lklv8/dns?timestamps=true: dial tcp: lookup ip-10-0-149-151 on 10.0.0.2:53: no such host, Get https://ip-10-0-149-151:10250/containerLogs/openshift-dns/dns-default-lklv8/dns-node-resolver?timestamps=true: dial tcp: lookup ip-10-0-149-151 on 10.0.0.2:53: no such host], one or more errors ocurred while gathering container data for pod dns-default-lplf9:
[Get https://ip-10-0-143-162:10250/containerLogs/openshift-dns/dns-default-lplf9/dns?timestamps=true: dial tcp: lookup ip-10-0-143-162 on 10.0.0.2:53: no such host, Get https://ip-10-0-143-162:10250/containerLogs/openshift-dns/dns-default-lplf9/dns-node-resolver?timestamps=true: dial tcp: lookup ip-10-0-143-162 on 10.0.0.2:53: no such host], one or more errors ocurred while gathering container data for pod dns-default-ltzhd:
[Get https://ip-10-0-4-202:10250/containerLogs/openshift-dns/dns-default-ltzhd/dns?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host, Get https://ip-10-0-4-202:10250/containerLogs/openshift-dns/dns-default-ltzhd/dns-node-resolver?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host], one or more errors ocurred while gathering container data for pod dns-default-mnq2w:
[Get https://ip-10-0-17-67:10250/containerLogs/openshift-dns/dns-default-mnq2w/dns?timestamps=true: dial tcp: lookup ip-10-0-17-67 on 10.0.0.2:53: no such host, Get https://ip-10-0-17-67:10250/containerLogs/openshift-dns/dns-default-mnq2w/dns-node-resolver?timestamps=true: dial tcp: lookup ip-10-0-17-67 on 10.0.0.2:53: no such host], one or more errors ocurred while gathering container data for pod dns-default-qcblt:
[Get https://ip-10-0-175-136:10250/containerLogs/openshift-dns/dns-default-qcblt/dns?timestamps=true: dial tcp: lookup ip-10-0-175-136 on 10.0.0.2:53: no such host, Get https://ip-10-0-175-136:10250/containerLogs/openshift-dns/dns-default-qcblt/dns-node-resolver?timestamps=true: dial tcp: lookup ip-10-0-175-136 on 10.0.0.2:53: no such host]], one or more errors ocurred while gathering pod-specific data for namespace: openshift-ingress-operator
one or more errors ocurred while gathering container data for pod ingress-operator-794cc7b555-62g87:
Get https://ip-10-0-175-136:10250/containerLogs/openshift-ingress-operator/ingress-operator-794cc7b555-62g87/ingress-operator?timestamps=true: dial tcp: lookup ip-10-0-175-136 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering pod-specific data for namespace: openshift-ingress
one or more errors ocurred while gathering container data for pod router-default-766766d8b6-zdkj9:
Get https://ip-10-0-175-136:10250/containerLogs/openshift-ingress/router-default-766766d8b6-zdkj9/router?timestamps=true: dial tcp: lookup ip-10-0-175-136 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering pod-specific data for namespace: openshift-kube-apiserver-operator
one or more errors ocurred while gathering container data for pod openshift-kube-apiserver-operator-58b8c455c5-p9drx:
: error dialing backend: dial tcp: lookup ip-10-0-38-26 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering pod-specific data for namespace: openshift-kube-apiserver
[one or more errors ocurred while gathering container data for pod openshift-kube-apiserver-ip-10-0-17-67.ec2.internal:
Get https://ip-10-0-17-67:10250/containerLogs/openshift-kube-apiserver/openshift-kube-apiserver-ip-10-0-17-67.ec2.internal/openshift-kube-apiserver?timestamps=true: dial tcp: lookup ip-10-0-17-67 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering container data for pod openshift-kube-apiserver-ip-10-0-4-202.ec2.internal:
Get https://ip-10-0-4-202:10250/containerLogs/openshift-kube-apiserver/openshift-kube-apiserver-ip-10-0-4-202.ec2.internal/openshift-kube-apiserver?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host], one or more errors ocurred while gathering pod-specific data for namespace: openshift-kube-controller-manager-operator
one or more errors ocurred while gathering container data for pod openshift-kube-controller-manager-operator-66cbb459f8-8lhcj:
Get https://ip-10-0-4-202:10250/containerLogs/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator-66cbb459f8-8lhcj/operator?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering pod-specific data for namespace: openshift-kube-controller-manager
[one or more errors ocurred while gathering container data for pod openshift-kube-controller-manager-ip-10-0-17-67.ec2.internal:
Get https://ip-10-0-17-67:10250/containerLogs/openshift-kube-controller-manager/openshift-kube-controller-manager-ip-10-0-17-67.ec2.internal/kube-controller-manager?timestamps=true: dial tcp: lookup ip-10-0-17-67 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering container data for pod openshift-kube-controller-manager-ip-10-0-4-202.ec2.internal:
Get https://ip-10-0-4-202:10250/containerLogs/openshift-kube-controller-manager/openshift-kube-controller-manager-ip-10-0-4-202.ec2.internal/kube-controller-manager?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host]]
Currently the must-gather command throws all crash* files into the directory: ./registry-redhat-io-ocs4-ocs-must-gather-rhel8-sha256-b1d22c2497fc432eb06d02f917f2ec6b0c763b16e6147b310de1bae1e8b30eef/ceph/must_gather_commands/
I'm creating this RFE to suggest we create a directory here called 'crashes', that houses all of these crash files. This will tidy up the directory and make it easier to parse when numerous crashes are present.
There is a big difference in ImageStreamTags items collected by must-gather and manual running of this command: oc get istag/python:latest -o yaml -n openshift
please take a look on attached files
must-gather_python.txt
direct_command_python.txt
In our case we need to have "release" and "version" labels to be collected
The oc describe nodes
command provides information about the requests/limits, pods in the nodes, allocated resources and events related to each node. Having that information in the must-gather report could help a lot with troubleshooting. If possible, the description of each node in separated files with the name of the node.
Hi guys,
Not sure if you aware of it, but since RHOCP4.6, some new namespaces have been added to the core, such as openshift-*-infra
, openshift-oauth-apiserver
.
And as those namespaces are now part of the RHOCP clusters, we need them as part of the must-gather to diagnose Red Hat Customers' issues.
I've been facing cases where I was looking at the must-gather, and then I had to request the inspect ns/....
for the missing namespace.
Could you please add the missing namespaces to the must-gather gather list?
Cheers,
Include pods and events information from all namespaces to the must-gather collection.
oc adm inspect --dest-dir must-gather pods --all-namespaces
oc adm inspect --dest-dir must-gather events --all-namespaces
I created a custom image with a new collection script called gather-cnv (https://github.com/djfjeff/must-gather/blob/gather-cnv-support/collection-scripts/gather-cnv).
Basically, for now, the new collection script only do : openshift-must-gather inspect ns/kubevirt
Manually running the inspect command above fetch the information but still produce some error output :
openshift-must-gather inspect ns/kubevirt
2019/04/26 14:02:40 Gathering data for ns/kubevirt...
2019/04/26 14:02:40 Collecting resources for namespace "kubevirt"...
2019/04/26 14:02:40 Gathering pod data for namespace "kubevirt"...
2019/04/26 14:02:40 Gathering data for pod "virt-api-f8d6cd97-5l68b"
2019/04/26 14:02:41 Unable to gather previous container logs: previous terminated container "virt-api" in pod "virt-api-f8d6cd97-5l68b" not found
2019/04/26 14:02:42 Skipping /version info gathering for pod "virt-api-f8d6cd97-5l68b". Endpoint not found...
2019/04/26 14:02:42 Gathering data for pod "virt-api-f8d6cd97-6h4kx"
2019/04/26 14:02:43 Unable to gather previous container logs: previous terminated container "virt-api" in pod "virt-api-f8d6cd97-6h4kx" not found
2019/04/26 14:02:43 Skipping /version info gathering for pod "virt-api-f8d6cd97-6h4kx". Endpoint not found...
2019/04/26 14:02:43 Gathering data for pod "virt-controller-865b95f6c6-gtplf"
2019/04/26 14:02:43 Unable to gather previous container logs: previous terminated container "virt-controller" in pod "virt-controller-865b95f6c6-gtplf" not found
2019/04/26 14:02:44 Gathering data for pod "virt-controller-865b95f6c6-vl6cc"
2019/04/26 14:02:44 Unable to gather previous container logs: previous terminated container "virt-controller" in pod "virt-controller-865b95f6c6-vl6cc" not found
2019/04/26 14:02:44 Gathering data for pod "virt-handler-5zd2z"
2019/04/26 14:02:44 Unable to gather previous container logs: previous terminated container "virt-handler" in pod "virt-handler-5zd2z" not found
2019/04/26 14:02:44 Gathering data for pod "virt-handler-9q9cp"
2019/04/26 14:02:44 Unable to gather previous container logs: previous terminated container "virt-handler" in pod "virt-handler-9q9cp" not found
2019/04/26 14:02:45 Gathering data for pod "virt-handler-m9q8r"
2019/04/26 14:02:45 Unable to gather previous container logs: previous terminated container "virt-handler" in pod "virt-handler-m9q8r" not found
2019/04/26 14:02:45 Gathering data for pod "virt-handler-t6kz2"
2019/04/26 14:02:45 Unable to gather previous container logs: previous terminated container "virt-handler" in pod "virt-handler-t6kz2" not found
2019/04/26 14:02:45 Gathering data for pod "virt-operator-76b568d986-6x9rc"
2019/04/26 14:02:45 Unable to gather previous container logs: previous terminated container "virt-operator" in pod "virt-operator-76b568d986-6x9rc" not found
Error: one or more errors ocurred while gathering pod-specific data for namespace: kubevirt
[one or more errors ocurred while gathering container data for pod virt-api-f8d6cd97-5l68b:
unable to gather container /healthz: unable to find any available /healthz paths hosted in pod "virt-api-f8d6cd97-5l68b", one or more errors ocurred while gathering container data for pod virt-api-f8d6cd97-6h4kx:
unable to gather container /healthz: unable to find any available /healthz paths hosted in pod "virt-api-f8d6cd97-6h4kx", one or more errors ocurred while gathering container data for pod virt-controller-865b95f6c6-gtplf:
[unable to gather container /healthz: the server could not find the requested resource, unable to gather container /version: the server could not find the requested resource], one or more errors ocurred while gathering container data for pod virt-controller-865b95f6c6-vl6cc:
[unable to gather container /healthz: the server could not find the requested resource, unable to gather container /version: the server could not find the requested resource], one or more errors ocurred while gathering container data for pod virt-handler-5zd2z:
[unable to gather container /healthz: the server could not find the requested resource, unable to gather container /version: the server could not find the requested resource], one or more errors ocurred while gathering container data for pod virt-handler-9q9cp:
[unable to gather container /healthz: the server could not find the requested resource, unable to gather container /version: the server could not find the requested resource], one or more errors ocurred while gathering container data for pod virt-handler-m9q8r:
[unable to gather container /healthz: the server could not find the requested resource, unable to gather container /version: the server could not find the requested resource], one or more errors ocurred while gathering container data for pod virt-handler-t6kz2:
[unable to gather container /healthz: the server could not find the requested resource, unable to gather container /version: the server could not find the requested resource], one or more errors ocurred while gathering container data for pod virt-operator-76b568d986-6x9rc:
[unable to gather container /healthz: the server could not find the requested resource, unable to gather container /version: the server could not find the requested resource]]
The issue I have is when running it with oc adm must-gather
, it seems the error reporting break the gather init container and make the whole process to fail :
oc adm must-gather --image=quay.io/jsaucier/must-gather-cnv:latest -- gather-cnv
namespace/openshift-must-gather-h9m84 created
clusterrolebinding.rbac.authorization.k8s.io/must-gather-q8xwf created
clusterrolebinding.rbac.authorization.k8s.io/must-gather-q8xwf deleted
namespace/openshift-must-gather-h9m84 deleted
error: pod is not running: Failed
To confirm, I try changing the namespace I gather in gather-cnv from ns/kubevirt to ns/default (which does not produce any error reporting) and everything works fine.
As part of our tekton (openshift pipeline) we have a must-gather task that runs if there are test case failures. Currently this task runs the following command oc adm must-gather --image=quay.io/opencloudio/must-gather:4.5.1 -- gather -n ibm-common-services,<namespace where product under test is installed to>
. We had a developer open up an issue against us saying certain kinds from CustomResourceDefinitions were not included in the must-gather zip. Is there a way to have these included?
The Following Resources are missing in the Must-gather:
That some unusual resources, but a miss configuration may have a big impact on the cluster performances.
Once added to the MG, we should be able to get them added in omc
and omg
for the support teams to check them.
And so I will be able to included them in my health check script to help review them quickly.
Happy to discuss it, if your think that we should not include them.
Hi,
Is there a reason why must-gather includes some secrets by default? This default behavior also collect secrets from namespaces like the following:
(tested with oc cli 4.6.x)
./namespaces/openshift-authentication/core/secrets.yaml
./namespaces/openshift-config/core/secrets.yaml
These secrets.yaml usually contain ldap username and passwords.
I would prefer a toggle where we can explicitly enable a dump of any sensitive data like secrets/configmaps to prevent data leakage.
I know that another tool called must-gather-clear could remove sensitive data afterwards. But why shouldn't we change the default behavior to a minimum and add additional information, if really needed.
I am interested to create an image with custom scripts to collect info from an OpenShift cluster. How can I get access to the doc mentioned below to get more familiar with this tool:
The directory structure, as well as specific details behind this tool can be found in this doc.
I requested access to the doc but my request hasn't been approved.
$ oc get HostSubnet -o json | head
{
"apiVersion": "v1",
"items": [
{
"apiVersion": "network.openshift.io/v1",
"host": "ip-10-0-131-141.eu-central-1.compute.internal",
"hostIP": "10.0.131.141",
"kind": "HostSubnet",
"metadata": {
"annotations": {
cluster-scoped-resources/network.openshift.io/
(where they should be):$ tree must-gather.local.1360424657569460676/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-3c30f115dc95c3fef94ea5185f386aa1af8a4b5f07ce8f41a17007d54004e1c4/cluster-scoped-resources/network.openshift.io/
must-gather.local.1360424657569460676/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-3c30f115dc95c3fef94ea5185f386aa1af8a4b5f07ce8f41a17007d54004e1c4/cluster-scoped-resources/network.openshift.io/
└── clusternetworks
└── default.yaml
$ oc version
Client Version: 4.8.0-202108312109.p0.git.0d10c3f.assembly.stream-0d10c3f
Server Version: 4.9.24
Kubernetes Version: v1.22.5+5c84e52
Get the pods' log in project gpu-operator-resources
for i in `oc get pods -n openshift-monitoring --no-headers=true | awk '{print $1}'`;
do
oc logs $i -n openshift-monitoring --all-containers=true
done
#121 removed the gathering of some openshift-sdn-specific resources, since it broke must-gather under ovn-kubernetes.
It would be good if there was some way we could gather the HostSubnet objects under openshift-sdn, but not error out under ovn-kubernetes (or calico, or ...). Likewise, in the future, we will probably want to gather some ovn-kubernetes-specific data when using ovn-kubernetes (although that may involve running commands to get data out of ovn databases (via oc exec
to the ovn pods?) rather than gathering additional API objects).
Obviously it's possible to do this with arbitrary shell-scripting, but what's the architecturally-correct way to do it? (We'd want this to be part of the default "oc adm must-gather" output.)
I recently noticed that when I executed must-gather, the pod got scheduled on master node which seems quite expected in the first place since I disabled scheduling for the remaining two workers.
Later, I provided the node name while executing the must-gather, and it was scheduled on the worker node even though the worker was marked SchedulingDisabled.
e.g $ oc adm must-gather --node-name=$worker
Then, I checked # oc describe node output and could not locate the must-gather pod. Also, the pod's scheduler appears to be default-scheduler due to which I felt that it could have prevented pod scheduling on the SchedulingDisabled node.
Tested on the following version:
Client Version: 4.7.0-202105281408.p0.git.95881af-95881af
Server Version: 4.7.14
Kubernetes Version: v1.20.0+a0b09eb
Did I miss anything?
Is this behavior expected w.r.t design of must-gather?
Currently must-gather, using oc adm inspect clusteroperators
, get the list of all the ClusterOperator
objects and then traverse all their related resources collecting information about pods, logs, etc
must-gather gets also some information about OLM, using oc adm inspect -A olm
command, including the list of Operator
objects, but it does not collect any other related resource information
must-gather could collect the same information from Operator
objects as is currently doing for ClusterOperator
objects maybe using a new oc adm inspect operators
command.
Openshift is heavily based on operators and not all of them are part of the vanilla system.
must-gather is a very valuable tool for debugging purposes and helps a lot with troubleshooting but its information about OLM operators is not as good as it could be, making harder to detect and troubleshoot problems with these components even more in some business areas like TelCo, where the number of "non-vanilla" operators is increasing.
Include the metrics from etcd, as described in https://access.redhat.com/solutions/5141931, in the "gather_etcd" script
When running must-gather via 'oc adm must-gather --image=...' , a new result folder is created.
Lots of static information (NS info, CRD's, Deployments, etc) is duplicated.
Need a way to runwith merge-results mode to have only the delta/aditional info.
Or an option to run must-gather with 'logs/events' mode - to produce only pod logs and events to be merged into a pre populated result folder.
I believe that gather_olm
script is not collecting data in 4.6 images (I've tested only in this Y).
I expect that the script gather_olm
collects OLM operator objects (sub, installPlans, catalogSources, CSVs) and save inside each place, we can see it when the command bellow is executed directly:
oc adm inspect --dest-dir=must-gather-olm -A olm
Steps to reproduce
I wrote 3 scenarios running the most recent image for Y 4.6:
A) script[0]+[1]: default must-gather - should run all collection-scripts/gather_* (including gather_olm)
B) script[0]+[2]: must-gather calling the script directly: -- /usr/bin/gather_olm
C) script[0]+[3]: collecting the same command define on gather_olm
So we can see that A
and B
are not collecting OLM objects, and when running the command directly on C
the objects can be collected.
[0] Get OCP version
$declare -r OCP_VERSION=$(oc get clusterversion version --template='{{.status.desired.version}}' |sed -n -r 's/([[:digit:]]\.[[:digit:]]).*/\1/p')
[1] default must-gather
$ echo ${OCP_VERSION}
4.6
$ oc adm must-gather \
--dest-dir=must-gather-general \
--image=quay.io/openshift/origin-must-gather:"${OCP_VERSION}"
[must-gather ] OUT Using must-gather plugin-in image: quay.io/openshift/origin-must-gather:${OCP_VERSION}
[...]
$ tree -a must-gather-general/*cluster-scoped-resources/operators.coreos.com/operators
must-gather-general/*cluster-scoped-resources/operators.coreos.com/operators [error opening dir]
0 directories, 0 files
$ tree -a must-gather-general/*cluster-scoped-resources/operators.coreos.com/operators
must-gather-general/*cluster-scoped-resources/operators.coreos.com/operators [error opening dir]
0 directories, 0 files
$ tree -a must-gather-general/*namespaces/openshift-marketplace/operators.coreos.com/catalogsources
must-gather-general/*namespaces/openshift-marketplace/operators.coreos.com/catalogsources [error opening dir]
0 directories, 0 files
[2] must-gather calling the script directly: -- /usr/bin/gather_olm
oc adm must-gather \
--dest-dir=must-gather-general-olm \
--image=quay.io/openshift/origin-must-gather:"${OCP_VERSION}" -- /usr/bin/gather_olm
$ tree -a must-gather-general-olm/*cluster-scoped-resources/operators.coreos.com/operators
must-gather-general-olm/*cluster-scoped-resources/operators.coreos.com/operators [error opening dir]
[3] running the same command on gather_olm
$ oc adm inspect --dest-dir=must-gather-olm -A olm
Wrote inspect data to must-gather-olm.
$ tree -a must-gather-olm/*cluster-scoped-resources/operators.coreos.com/operators
must-gather-olm/cluster-scoped-resources/operators.coreos.com/operators
├── 3scale-community-operator.3scale.yaml
├── 3scale-operator.3scale.yaml
├── cluster-logging.openshift-logging.yaml
├── compliance-operator.openshift-compliance.yaml
├── elasticsearch-operator.openshift-operators-redhat.yaml
└── web-terminal.openshift-operators.yaml
$ tree -a must-gather-olm/*namespaces/openshift-marketplace/operators.coreos.com/catalogsources
must-gather-olm/namespaces/openshift-marketplace/operators.coreos.com/catalogsources
├── certified-operators.yaml
├── community-operators.yaml
├── redhat-marketplace.yaml
└── redhat-operators.yaml
Questions
I have already had the same issue described on #230 in my local development, and the olm script
gather_olm
is using the option--dest-dir
and raising errors due to old cli.
When I try to call must-gather with an invalid image (i.e. any image that causes the must-gather pod to have the status Init:InvalidImageName
) the command doesn't give any error and does not terminate.
Wrong repo :)
Follow the policies, guidelines, and standards to release an OCP image of the must-gather tool.
must-gather
ImageStream to openshift-samples-operator.while experimenting with creating my own must-gather image i tried to make a build from the Dockerfile
located in the root of the project. although it built properly, when running it i see an error in that the --dest-dir
option is not available. upon further inspection it appears that the oc
binary included in the registry.svc.ci.openshift.org/openshift/origin-v4.0:cli
image does no support the inspect
command, thus the gather scripts will fail.
a snippet of error output:
[must-gather ] OUT namespace/openshift-must-gather-gc2vz created
[must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-4hkdz created
[must-gather ] OUT pod for plug-in image quay.io/fnord/must-gather created
[must-gather-2qqfs] POD 2021-05-04T00:32:02.896350743Z Error: unknown flag: --dest-dir
how to reproduce
podman build -t quay.io/fnord/must-gather
oc must-gather --image quay.io/fnord/must-gather
The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.
release-4.16
release-4.17
For more information, see the branching documentation.
Looking at the must-gather.log from my latest ci-test run, it seems like we throw INFO logs which make the log harder to read. See:
[must-gather-xxbbf] POD 2023-05-17T11:59:17.152066356Z Gathering data for ns/openshift-cluster-version...
[must-gather-xxbbf] POD 2023-05-17T11:59:19.555676602Z W0517 11:59:19.555610 8 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:19.558880211Z W0517 11:59:19.558851 8 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:19.663594946Z Gathering data for ns/default...
[must-gather-xxbbf] POD 2023-05-17T11:59:19.796052158Z W0517 11:59:19.796007 8 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:19.802501693Z W0517 11:59:19.802477 8 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:19.933839951Z Gathering data for ns/openshift...
[must-gather-xxbbf] POD 2023-05-17T11:59:20.179844728Z W0517 11:59:20.179806 8 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:20.182575941Z W0517 11:59:20.182553 8 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:20.279559456Z Gathering data for ns/kube-system...
[must-gather-xxbbf] POD 2023-05-17T11:59:20.408955535Z W0517 11:59:20.408913 8 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:20.411238487Z W0517 11:59:20.411218 8 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:20.678549529Z Gathering data for ns/openshift-etcd...
[must-gather-xxbbf] POD 2023-05-17T11:59:21.679718696Z W0517 11:59:21.679675 8 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:21.683752225Z W0517 11:59:21.683726 8 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:22.060183522Z Wrote inspect data to must-gather.
[must-gather-xxbbf] POD 2023-05-17T11:59:23.790812585Z Gathering data for ns/openshift-config...
[must-gather-xxbbf] POD 2023-05-17T11:59:23.928293099Z W0517 11:59:23.928253 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:23.932897522Z W0517 11:59:23.932869 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:24.044526544Z Gathering data for ns/openshift-config-managed...
[must-gather-xxbbf] POD 2023-05-17T11:59:24.313978398Z W0517 11:59:24.313931 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:24.319885490Z W0517 11:59:24.319850 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:24.411760399Z Gathering data for ns/openshift-authentication...
[must-gather-xxbbf] POD 2023-05-17T11:59:24.699325865Z W0517 11:59:24.699277 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:24.702712681Z W0517 11:59:24.702687 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:24.886439798Z Gathering data for ns/openshift-authentication-operator...
[must-gather-xxbbf] POD 2023-05-17T11:59:25.100911831Z W0517 11:59:25.100877 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:25.106497028Z W0517 11:59:25.106460 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
.
.
.
[must-gather-xxbbf] POD 2023-05-17T12:00:06.109510622Z Gathering data for ns/openshift-machine-config-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:06.616004749Z W0517 12:00:06.615962 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:06.619411887Z W0517 12:00:06.619376 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:07.024611216Z Gathering data for ns/openshift-kni-infra...
[must-gather-xxbbf] POD 2023-05-17T12:00:12.765883020Z W0517 12:00:12.765836 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:12.769646923Z W0517 12:00:12.769622 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:12.847601824Z Gathering data for ns/openshift-openstack-infra...
[must-gather-xxbbf] POD 2023-05-17T12:00:12.959084289Z W0517 12:00:12.959043 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:12.961865990Z W0517 12:00:12.961835 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.044327483Z Gathering data for ns/openshift-ovirt-infra...
[must-gather-xxbbf] POD 2023-05-17T12:00:13.165554607Z W0517 12:00:13.165507 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.168633601Z W0517 12:00:13.168601 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.255017243Z Gathering data for ns/openshift-vsphere-infra...
[must-gather-xxbbf] POD 2023-05-17T12:00:13.384294288Z W0517 12:00:13.384242 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.387208878Z W0517 12:00:13.387171 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.479537846Z Gathering data for ns/openshift-nutanix-infra...
[must-gather-xxbbf] POD 2023-05-17T12:00:13.619418134Z W0517 12:00:13.619377 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.635433684Z W0517 12:00:13.635400 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.736145710Z Gathering data for ns/openshift-marketplace...
[must-gather-xxbbf] POD 2023-05-17T12:00:14.031202793Z W0517 12:00:14.031171 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:14.034909709Z W0517 12:00:14.034879 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:14.237141996Z Gathering data for ns/openshift-user-workload-monitoring...
[must-gather-xxbbf] POD 2023-05-17T12:00:14.363284391Z W0517 12:00:14.363225 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:14.367420112Z W0517 12:00:14.367390 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:14.980714662Z Gathering data for ns/openshift-sdn...
[must-gather-xxbbf] POD 2023-05-17T12:00:15.717496223Z W0517 12:00:15.717454 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:15.720279047Z W0517 12:00:15.720254 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:16.021123774Z Gathering data for ns/openshift-host-network...
[must-gather-xxbbf] POD 2023-05-17T12:00:16.150152499Z W0517 12:00:16.150110 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:16.153603423Z W0517 12:00:16.153575 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:16.255697194Z Gathering data for ns/openshift-network-diagnostics...
[must-gather-xxbbf] POD 2023-05-17T12:00:16.513744169Z W0517 12:00:16.513703 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:16.516812516Z W0517 12:00:16.516630 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:16.742676795Z Gathering data for ns/openshift-network-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:17.719904200Z W0517 12:00:17.719867 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:17.722734390Z W0517 12:00:17.722701 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:17.856745666Z Gathering data for ns/openshift-cloud-network-config-controller...
[must-gather-xxbbf] POD 2023-05-17T12:00:17.971103700Z W0517 12:00:17.971046 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:17.977205771Z W0517 12:00:17.977095 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:18.120809852Z Gathering data for ns/openshift-apiserver-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:18.333571325Z W0517 12:00:18.333524 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:18.343028392Z W0517 12:00:18.342996 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:18.550916367Z Gathering data for ns/openshift-apiserver...
[must-gather-xxbbf] POD 2023-05-17T12:00:18.962479751Z W0517 12:00:18.962435 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:18.966718088Z W0517 12:00:18.966688 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:19.337594227Z Gathering data for ns/openshift-controller-manager-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:19.636247453Z W0517 12:00:19.636200 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:19.639258446Z W0517 12:00:19.639224 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:19.772732552Z Gathering data for ns/openshift-controller-manager...
[must-gather-xxbbf] POD 2023-05-17T12:00:20.012199454Z W0517 12:00:20.012149 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:20.016432685Z W0517 12:00:20.016389 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:20.168098552Z Gathering data for ns/openshift-route-controller-manager...
[must-gather-xxbbf] POD 2023-05-17T12:00:20.359217322Z W0517 12:00:20.359170 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:20.362151005Z W0517 12:00:20.362123 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:20.515942559Z Gathering data for ns/openshift-cluster-samples-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:20.740834006Z W0517 12:00:20.740216 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:20.753593652Z W0517 12:00:20.753547 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:21.223975051Z Gathering data for ns/openshift-operator-lifecycle-manager...
[must-gather-xxbbf] POD 2023-05-17T12:00:21.619559658Z W0517 12:00:21.619190 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:21.625162052Z W0517 12:00:21.625135 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:21.829182721Z Gathering data for ns/openshift-service-ca-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:21.975115463Z W0517 12:00:21.975067 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:21.981283096Z W0517 12:00:21.981246 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:22.086810692Z Gathering data for ns/openshift-service-ca...
[must-gather-xxbbf] POD 2023-05-17T12:00:22.241755336Z W0517 12:00:22.241703 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:22.246609074Z W0517 12:00:22.246575 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:22.352120347Z Gathering data for ns/openshift-cluster-csi-drivers...
[must-gather-xxbbf] POD 2023-05-17T12:00:22.454613572Z W0517 12:00:22.454565 176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:22.457398208Z W0517 12:00:22.457362 176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
I file this issue to suggest muting the the server doesn't have..
logs to provide a cleaner log output for the must-gather.
The leases resource is mainly used for Node heartbeats and Leader election,
which both have a key role in some mechanisms of different operators, for exmple in the sriov-operator.
it makes sense to make ocp collect this as part of the must-gather.
TL;DR: create a folder for each node, symlink in pods running on that node.
Background: When debugging networking issues, we need to look at the logs for the network daemon running on a specific node. Of course, we can extract this from the pod's yaml, but this is somewhat tedious and potentially error-prone.
It would be useful if there was a way for the must-gather to make navigating on a per-node basis easier. It might look something like
/per_node/NODENAME/pod/myns_podname.yaml -> /namespaces/myns/pods/podname/podname.yaml
as well as
/namespaces/myns/pods/per_node/NODENAME/podname.yaml -> /namespaces/myns/pods/podname/podname.yaml
When we specify the target namespaces to collect the logs, we are always pulling back all the logs. We have tried both -n and --namespaces.
oc adm must-gather --image=quay.io/opencloudio/must-gather:4.5.1 -- gather -n cem
oc describe po must-gather-fmzgt
Name: must-gather-fmzgt
Namespace: openshift-must-gather-4l77r
Priority: 0
Node: master1.scao3-ocp43.os.fyre.ibm.com/10.16.39.94
Start Time: Fri, 16 Oct 2020 11:55:40 -0500
Labels: app=must-gather
Annotations: k8s.v1.cni.cncf.io/networks-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.254.20.68"
],
"dns": {},
"default-route": [
"10.254.20.1"
]
}]
Status: Pending
IP: 10.254.20.68
IPs:
IP: 10.254.20.68
Init Containers:
gather:
Container ID: cri-o://fb9cd71ffd3071eac612b31d41f16edb34e0258289a50e1c262a2a1ed224c4d1
Image: quay.io/opencloudio/must-gather:4.5.1
Image ID: quay.io/opencloudio/must-gather@sha256:457b43e57fc6df8456167bfde409e45351d6cc9420fa2d1eea9eb2e4567923ff
Port: <none>
Host Port: <none>
Command:
gather
-n
cem
State: Running
Started: Fri, 16 Oct 2020 11:56:02 -0500
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/must-gather from must-gather-output (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xslj4 (ro)
Containers:
copy:
Container ID:
Image: quay.io/opencloudio/must-gather:4.5.1
Image ID:
Port: <none>
Host Port: <none>
Command:
/bin/bash
-c
trap : TERM INT; sleep infinity & wait
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/must-gather from must-gather-output (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xslj4 (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
must-gather-output:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
default-token-xslj4:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-xslj4
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned openshift-must-gather-4l77r/must-gather-fmzgt to master1.scao3-ocp43.os.fyre.ibm.com
Normal Pulling 8m4s kubelet, master1.scao3-ocp43.os.fyre.ibm.com Pulling image "quay.io/opencloudio/must-gather:4.5.1"
Normal Pulled 7m50s kubelet, master1.scao3-ocp43.os.fyre.ibm.com Successfully pulled image "quay.io/opencloudio/must-gather:4.5.1"
Normal Created 7m50s kubelet, master1.scao3-ocp43.os.fyre.ibm.com Created container gather
Normal Started 7m50s kubelet, master1.scao3-ocp43.os.fyre.ibm.com Started container gather
Output:
oc adm must-gather --image=quay.io/opencloudio/must-gather:4.5.1 -- gather -n cem
[must-gather ] OUT Using must-gather plugin-in image: quay.io/opencloudio/must-gather:4.5.1
[must-gather ] OUT namespace/openshift-must-gather-4l77r created
[must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-c2z77 created
[must-gather ] OUT pod for plug-in image quay.io/opencloudio/must-gather:4.5.1 created
[must-gather-fmzgt] POD configuration:
[must-gather-fmzgt] POD modules overview,system,failure,ocp,cloudpak
[must-gather-fmzgt] POD namespaces cem
[must-gather-fmzgt] POD labels
[must-gather-fmzgt] POD extensions
[must-gather-fmzgt] POD env
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD INFO: gather_overview collecting ...
[must-gather-fmzgt] POD # oc get nodes --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get pods --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get deploy --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get sts --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get ds --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get job --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get cronjob --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get configmap --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get secret --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get resourcequota --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get hpa --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get svc --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get endpoints --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get ingress --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get route --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get networkpolicy --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get pv --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get pvc --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get storageclass --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get namespace --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get event --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get crd --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get csv --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get certificates.certmanager.k8s.io --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get challenges.certmanager.k8s.io --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get clusterissuers.certmanager.k8s.io --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get issuers.certmanager.k8s.io --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get orders.certmanager.k8s.io --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD INFO: gather_system collecting ...
[must-gather-fmzgt] POD daemonset.apps/gather-system-ds created
[must-gather-fmzgt] POD INFO: daemonset openshift-must-gather-4l77r/gather-system-ds not ready
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD INFO: daemonset openshift-must-gather-4l77r/gather-system-ds not ready
[must-gather-fmzgt] POD INFO: daemonset openshift-must-gather-4l77r/gather-system-ds not ready
[must-gather-fmzgt] POD INFO: daemonset openshift-must-gather-4l77r/gather-system-ds not ready
[must-gather-fmzgt] POD INFO: daemonset openshift-must-gather-4l77r/gather-system-ds not ready
[must-gather-fmzgt] POD INFO: daemonset openshift-must-gather-4l77r/gather-system-ds not ready
[must-gather-fmzgt] POD INFO: daemonset openshift-must-gather-4l77r/gather-system-ds not ready
[must-gather-fmzgt] POD INFO: daemonset openshift-must-gather-4l77r/gather-system-ds is ready
[must-gather-fmzgt] POD INFO: gather_system on node master2.scao3-ocp43.os.fyre.ibm.com pod gather-system-ds-dn5n8
[must-gather-fmzgt] POD INFO: gather_system on node master1.scao3-ocp43.os.fyre.ibm.com pod gather-system-ds-nchfs
[must-gather-fmzgt] POD INFO: gather_system on node master0.scao3-ocp43.os.fyre.ibm.com pod gather-system-ds-vjnmw
[must-gather-fmzgt] POD INFO: Waiting for gather_system log collection to complete ...
[must-gather-fmzgt] POD INFO: gather_system log collection complete.
[must-gather-fmzgt] POD daemonset.apps "gather-system-ds" deleted
[must-gather-fmzgt] POD WARNING: Collecting one or more service logs on ALL master nodes in your cluster. This could take a large amount of time.
[must-gather-fmzgt] POD INFO: Collecting host service logs for kubelet
[must-gather-fmzgt] POD INFO: Collecting host service logs for crio
[must-gather-fmzgt] POD INFO: Waiting for worker host service log collection to complete ...
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD INFO: Worker host service log collection complete.
[must-gather-fmzgt] POD WARNING: Collecting network logs on ALL nodes in your cluster. This could take a long time.
[must-gather-fmzgt] POD I1016 16:57:52.513486 1572 request.go:621] Throttling request took 1.127477938s, request: GET:https://172.30.0.1:443/apis/gke.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Error from server: error dialing backend: dial tcp 10.16.39.171:10250: connect: connection refused
[must-gather-fmzgt] POD Error from server: error dialing backend: dial tcp 10.16.39.171:10250: connect: connection refused
[must-gather-fmzgt] POD Error from server: error dialing backend: dial tcp 10.16.39.171:10250: connect: connection refused
[must-gather-fmzgt] POD Error from server: error dialing backend: dial tcp 10.16.39.171:10250: connect: connection refused
[must-gather-fmzgt] POD INFO: Waiting for node network log collection to complete ...
[must-gather-fmzgt] POD INFO: Node network log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_pods collecting ...
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-cassandra-0
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-datalayer-0
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-datalayer-cron-1602724500-tk6c5
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-datalayer-cron-1602810900-596n6
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-datalayer-cron-1602810900-6nr5m
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-datalayer-cron-1602810900-d7cpl
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-datalayer-cron-1602810900-q9xjd
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-datalayer-cron-1602810900-rjzng
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-datalayer-cron-1602810900-zvcv4
[must-gather-fmzgt] POD INFO: gather_failure_pods on kube-system/multicluster-hub-console-mcmuiapi-bfbd55ddc-q7bhn
[must-gather-fmzgt] POD INFO: gather_failure_pods on kube-system/multicluster-hub-core-apiserver-7f6645f546-gwcbp
[must-gather-fmzgt] POD INFO: gather_failure_pods on kube-system/multicluster-hub-grafeas-64f6c9c7c-xkq2f
[must-gather-fmzgt] POD INFO: gather_failure_pods on kube-system/multicluster-hub-grcuiapi-75c4fcd6bb-gsqm7
[must-gather-fmzgt] POD INFO: gather_failure_pods on kube-system/multicluster-hub-legato-5bbc7f776b-f4tnt
[must-gather-fmzgt] POD INFO: gather_failure_pods on kube-system/multicluster-hub-topologyapi-5db998858f-f62wt
[must-gather-fmzgt] POD INFO: gather_failure_pods on kube-system/search-search-api-7b4c978599-hbjxg
[must-gather-fmzgt] POD INFO: gather_failure_pods on openshift-console/downloads-5fdf68d856-rzcml
[must-gather-fmzgt] POD INFO: gather_failure_pods on openshift-kube-apiserver/installer-67-master2.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_failure_pods on openshift-kube-apiserver/revision-pruner-66-master1.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_failure_pods on rook-ceph/rook-ceph-osd-6-58bd78b9b7-d2qsp
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_pods collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_pods log collection complete.
[must-gather-fmzgt] POD INFO: gather_restart_pods on cem/peter-ibm-cem-datalayer-0
[must-gather-fmzgt] POD INFO: gather_restart_pods on cem/peter-ibm-cem-rba-as-566bcffb6-tnzqp
[must-gather-fmzgt] POD INFO: gather_restart_pods on cem/peter-ibm-cem-rba-rbs-689475646d-png8d
[must-gather-fmzgt] POD INFO: gather_restart_pods on cert-manager/cert-manager-ibm-cert-manager-5cd9fb9777-cks5p
[must-gather-fmzgt] POD INFO: gather_restart_pods on cert-manager/cert-manager-webhook-cainjector-f46ff7b74-p442r
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/auth-idp-bckx9
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/auth-idp-z2bwn
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/auth-idp-zkg7d
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/auth-pdp-ff9rk
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/auth-pdp-grgp4
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/auth-pdp-ln9rd
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/catalog-ui-4ndvq
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/catalog-ui-gw6tk
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/catalog-ui-jl2lz
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/common-web-ui-wpwtq
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/common-web-ui-zk2l6
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/common-web-ui-zw46s
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/icp-mongodb-0
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/icp-mongodb-1
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/internal-management-ingress-9gwsh
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/internal-management-ingress-scgxp
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/metering-dm-c7b75c686-n2b7h
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/metering-reader-7522r
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/metering-reader-7bwdl
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/metering-reader-8x48b
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/metering-reader-bhpnk
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/metering-reader-l5wsj
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/metering-reader-ppzhz
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/metering-reader-qsj6z
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/monitoring-grafana-67ddcdf99d-k7df4
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/monitoring-prometheus-nodeexporter-2cd57
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/multicluster-hub-cluster-api-provider-aks-b84b6657-pqhgx
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/multicluster-hub-cluster-api-provider-eks-c8b6c956f-bmrlz
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/multicluster-hub-cluster-api-provider-gke-547c5c7b59-kpprf
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/multicluster-hub-cluster-api-provider-iks-7c4999db75-fm62g
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/multicluster-hub-cluster-api-provider-ocp-6d5fbd8459-ckg46
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/multicluster-hub-cluster-api-provider-sdk-b6979fffb-fm89p
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/multicluster-hub-core-apiserver-7f6645f546-gwcbp
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/multicluster-hub-core-channel-59fdf84bf4-5sjm4
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/nginx-ingress-controller-czjkn
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/nginx-ingress-controller-h95x9
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/nginx-ingress-controller-smppk
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/platform-api-794c95849c-jhjpn
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/platform-api-794c95849c-wz89k
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/platform-header-49gzc
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/platform-header-s8tp2
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/platform-header-xhp4k
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/prometheus-monitoring-prometheus-0
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/search-search-aggregator-64b75d9466-lj9ms
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/search-search-collector-6c68bd89bd-rhmt2
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/secret-watcher-779df94cc8-z7g8c
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-apiserver-operator/openshift-apiserver-operator-7cc77d965f-x8pff
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-apiserver/apiserver-8rpk8
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-apiserver/apiserver-j5gsv
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-apiserver/apiserver-lm8sl
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-authentication-operator/authentication-operator-57d4974d5d-6tk6f
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-cloud-credential-operator/cloud-credential-operator-dd496454-57rdg
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-cluster-version/cluster-version-operator-584fddff45-tgdkd
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-console-operator/console-operator-57f5bcc578-6q5vj
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-controller-manager-operator/openshift-controller-manager-operator-68dcf95c47-6bz2w
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-controller-manager/controller-manager-2zczq
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-controller-manager/controller-manager-6r22f
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-85jts
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-c9czj
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-f87fn
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-klp8r
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-m9p5v
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-mnfxh
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-p62vs
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-qbs6q
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-vvw7j
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-wxv5g
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-etcd/etcd-member-master0.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-etcd/etcd-member-master1.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-etcd/etcd-member-master2.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-55mpk
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-8gljk
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-fwp52
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-h4p5w
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-hrsqc
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-j9gfj
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-ktrnn
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-mnt9j
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-wpzqh
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-xmr4j
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-apiserver-operator/kube-apiserver-operator-566b9798-7rjbl
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-apiserver/kube-apiserver-master0.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-apiserver/kube-apiserver-master1.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-apiserver/kube-apiserver-master2.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-controller-manager-operator/kube-controller-manager-operator-7c8b7465b-gd8nf
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-controller-manager/kube-controller-manager-master0.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-controller-manager/kube-controller-manager-master2.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-scheduler-operator/openshift-kube-scheduler-operator-557777c86b-2c2rk
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-scheduler/openshift-kube-scheduler-master0.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-scheduler/openshift-kube-scheduler-master2.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-api/cluster-autoscaler-operator-ccbfbd76b-brfvj
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-api/machine-api-operator-566464fc6-ktj26
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-controller-79c987cbbf-2kj5c
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-6f5kc
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-cq68r
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-jvt64
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-m8ccg
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-q8rn6
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-tqqkz
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-vq7f4
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-x2n6c
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-xbqqd
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-xpkk6
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-operator-bd8dd7bfb-d48cr
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-server-6466d
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-server-9rsbz
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-server-z9jl7
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-2qldx
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-7ltcf
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-jdjlp
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-l2vdm
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-ldv57
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-p8rq2
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-rz2lv
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-wx6fv
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-xr2wg
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-zq6nv
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/prometheus-k8s-1
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/thanos-querier-6c57cc5d4-pvw5h
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-2j8n4
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-4cqbg
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-7fndz
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-admission-controller-4brrc
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-admission-controller-9qlgq
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-admission-controller-sqbjw
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-fbggj
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-fpth8
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-j8lsf
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-jnvdn
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-pw5bt
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-q8vlm
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-zr88j
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-45bjp
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-9vskw
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-dbkck
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-hxlsk
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-mwgvq
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-mxcjw
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-qdgkb
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-tkvkm
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-tr4x8
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-w9qjl
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-7fc54
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-controller-q9rjf
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-controller-qpvqf
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-controller-sfvcj
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-gmxls
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-hlqtr
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-hqvnt
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-jtkd4
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-kvs6j
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-m8f4s
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-sz5pc
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-wdnbf
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-zpm46
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-ca-operator/service-ca-operator-595657f77-jhx8b
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-ca/apiservice-cabundle-injector-d84c98485-ll48j
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-ca/configmap-cabundle-injector-6cc5ccdd7f-j5jpq
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-ca/service-serving-cert-signer-d59b877-jfzhp
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-catalog-apiserver-operator/openshift-service-catalog-apiserver-operator-6cddfd76cc-jp659
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-catalog-apiserver/apiserver-8b7bb
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-catalog-apiserver/apiserver-sffhz
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-catalog-apiserver/apiserver-tvl26
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-catalog-controller-manager-operator/openshift-service-catalog-controller-manager-operator-5886grk78
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-catalog-controller-manager/controller-manager-589b2
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-catalog-controller-manager/controller-manager-rh6rb
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-catalog-controller-manager/controller-manager-vtkrn
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-cephfsplugin-c29kl
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-cephfsplugin-hct5z
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-cephfsplugin-hhqhx
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-cephfsplugin-hvgft
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-cephfsplugin-knt9q
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-cephfsplugin-lmhhv
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-cephfsplugin-mgrkw
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-cephfsplugin-provisioner-75c965db4f-6x59q
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-rbdplugin-8vgks
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-rbdplugin-btpzj
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-rbdplugin-fp89d
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-rbdplugin-m57x2
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-rbdplugin-provisioner-56cbc4d585-7kjjh
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-rbdplugin-tmg4w
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-rbdplugin-tndvm
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-rbdplugin-xjd2t
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/rook-ceph-operator-6f68b5bf75-8c72r
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/rook-discover-549sm
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/rook-discover-b4dtn
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/rook-discover-fp527
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/rook-discover-hntj7
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/rook-discover-q5d75
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/rook-discover-w42vq
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/rook-discover-ztr9k
[must-gather-fmzgt] POD INFO: Waiting for gather_restart_pods collection to complete ...
[must-gather-fmzgt] POD INFO: gather_restart_pods log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_deployments collecting ...
[must-gather-fmzgt] POD INFO: gather_failure_deployments on kube-system/multicluster-hub-core-apiserver
[must-gather-fmzgt] POD INFO: gather_failure_deployments on rook-ceph/rook-ceph-osd-6
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_deployments collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_deployments log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_daemonsets collecting ...
[must-gather-fmzgt] POD INFO: gather_failure_daemonsets on openshift-service-catalog-apiserver/apiserver
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_daemonsets collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_daemonsets log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_statefulset collecting ...
[must-gather-fmzgt] POD INFO: gather_failure_statefulsets on cem/peter-cassandra
[must-gather-fmzgt] POD INFO: gather_failure_statefulsets on cem/peter-ibm-cem-datalayer
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_statefulsets collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_statefulsets log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_jobs collecting ...
[must-gather-fmzgt] POD INFO: gather_failure_jobs on cem/peter-ibm-cem-datalayer-cron-1602810900
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_jobs collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_jobs log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_routes collecting ...
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_routes collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_routes log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_services collecting ...
[must-gather-fmzgt] POD INFO: gather_failure_services on cem/peter-cassandra
[must-gather-fmzgt] POD INFO: gather_failure_services on cem/peter-ibm-cem-datalayer
[must-gather-fmzgt] POD INFO: gather_failure_services on cem/peter-ibm-redis-announce-3
[must-gather-fmzgt] POD INFO: gather_failure_services on cem/peter-ibm-redis-announce-4
[must-gather-fmzgt] POD INFO: gather_failure_services on cem/peter-ibm-redis-announce-5
[must-gather-fmzgt] POD INFO: gather_failure_services on kube-system/deployable-controller-metrics
[must-gather-fmzgt] POD INFO: gather_failure_services on kube-system/icp-oidcclient-watcher
[must-gather-fmzgt] POD INFO: gather_failure_services on kube-system/multicluster-hub-etcd
[must-gather-fmzgt] POD INFO: gather_failure_services on kube-system/subscription-controller-metrics
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_services collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_services log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_pv collecting ...
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_pv collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_pv log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_pvc collecting ...
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_pvc collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_pvc log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_namespaces collecting ...
[must-gather-fmzgt] POD INFO: gather_failure_namespaces on bodjruj-dyn-0/Terminating
[must-gather-fmzgt] POD INFO: gather_failure_namespaces on eydufza-dyn-0/Terminating
[must-gather-fmzgt] POD INFO: gather_failure_namespaces on ezqcamn-dyn-0/Terminating
[must-gather-fmzgt] POD INFO: gather_failure_namespaces on nsgbzoh-dyn-0/Terminating
[must-gather-fmzgt] POD INFO: gather_failure_namespaces on rtzlpkb-dyn-0/Terminating
[must-gather-fmzgt] POD INFO: gather_failure_namespaces on undmcps-dyn-0/Terminating
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_namespaces collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_namespaces log collection complete.
[must-gather-fmzgt] POD INFO: Waiting for gather_ocp collection to complete ...
[must-gather-fmzgt] POD Gathering data for ns/openshift-etcd...
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Gathering data for ns/openshift...
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Gathering data for ns/openshift-cluster-version...
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Gathering data for ns/default...
[must-gather-fmzgt] POD Gathering data for ns/kube-system...
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD I1016 16:58:55.386470 18923 request.go:621] Throttling request took 1.144576304s, request: GET:https://172.30.0.1:443/apis/config.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD I1016 16:58:55.476994 18930 request.go:621] Throttling request took 1.160946705s, request: GET:https://172.30.0.1:443/apis/policies.ibm.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD I1016 16:58:56.402535 18917 request.go:621] Throttling request took 1.062235725s, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD I1016 16:58:56.652810 18916 request.go:621] Throttling request took 1.145479867s, request: GET:https://172.30.0.1:443/apis/iks.mcm.ibm.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD I1016 16:58:57.573833 18910 request.go:621] Throttling request took 1.181404974s, request: GET:https://172.30.0.1:443/apis/aws.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD I1016 16:58:58.204710 18911 request.go:621] Throttling request took 1.113340658s, request: GET:https://172.30.0.1:443/apis/coordination.k8s.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD I1016 16:59:03.407018 18919 request.go:621] Throttling request took 1.139152323s, request: GET:https://172.30.0.1:443/apis/route.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD I1016 16:59:08.270827 18911 request.go:621] Throttling request took 1.548027309s, request: GET:https://172.30.0.1:443/apis/objectbucket.io/v1alpha1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-config...
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Gathering data for ns/openshift-config-managed...
[must-gather-fmzgt] POD I1016 16:59:20.884750 18911 request.go:621] Throttling request took 1.148933929s, request: GET:https://172.30.0.1:443/apis/oidc.security.ibm.com/v1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD Gathering data for ns/openshift-authentication...
[must-gather-fmzgt] POD I1016 16:59:31.832831 18911 request.go:621] Throttling request took 1.142908918s, request: GET:https://172.30.0.1:443/apis/operators.coreos.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-authentication-operator...
[must-gather-fmzgt] POD I1016 16:59:42.377686 18911 request.go:621] Throttling request took 1.156446345s, request: GET:https://172.30.0.1:443/apis/multicloud.ibm.com/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-ingress...
[must-gather-fmzgt] POD I1016 16:59:53.225686 18911 request.go:621] Throttling request took 1.138071941s, request: GET:https://172.30.0.1:443/apis/oauth.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-cloud-credential-operator...
[must-gather-fmzgt] POD I1016 17:00:06.184106 18911 request.go:621] Throttling request took 1.179412318s, request: GET:https://172.30.0.1:443/apis/webhook.certmanager.k8s.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:00:16.248806 18911 request.go:621] Throttling request took 1.127400782s, request: GET:https://172.30.0.1:443/apis/securityenforcement.admission.cloud.ibm.com/v1beta1?timeout=32s
[must-gather-fmzgt] POD I1016 17:00:26.315509 18911 request.go:621] Throttling request took 1.546428134s, request: GET:https://172.30.0.1:443/apis/autoscaling.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:00:36.390391 18911 request.go:621] Throttling request took 1.935471806s, request: GET:https://172.30.0.1:443/apis/iks.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD I1016 17:00:47.720438 18911 request.go:621] Throttling request took 1.152827244s, request: GET:https://172.30.0.1:443/apis/monitoring.coreos.com/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-machine-api...
[must-gather-fmzgt] POD I1016 17:00:58.352701 18911 request.go:621] Throttling request took 1.145100849s, request: GET:https://172.30.0.1:443/apis/tuned.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:01:08.373605 18911 request.go:621] Throttling request took 1.128955507s, request: GET:https://172.30.0.1:443/apis/certmanager.k8s.io/v1alpha1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-console-operator...
[must-gather-fmzgt] POD I1016 17:01:18.401243 18911 request.go:621] Throttling request took 1.149473075s, request: GET:https://172.30.0.1:443/apis/image.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-console...
[must-gather-fmzgt] POD I1016 17:01:28.003862 18918 request.go:621] Throttling request took 1.110309274s, request: GET:https://172.30.0.1:443/apis/ingress.operator.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:01:36.384207 18911 request.go:621] Throttling request took 1.174982991s, request: GET:https://172.30.0.1:443/apis/foundation.ibm.com/v1?timeout=32s
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD error: errors ocurred while gathering data:
[must-gather-fmzgt] POD one or more errors ocurred while gathering pod-specific data for namespace: kube-system
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [one or more errors ocurred while gathering container data for pod metering-reader-qsj6z:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/kube-system/metering-reader-qsj6z/metering-reader?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/metering-reader-qsj6z/metering-reader?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/metering-reader-qsj6z/metering-reader-secret-check?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/metering-reader-qsj6z/metering-reader-secret-check?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/metering-reader-qsj6z/metering-reader-init?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/metering-reader-qsj6z/metering-reader-init?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod monitoring-prometheus-nodeexporter-z9twl:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/kube-system/monitoring-prometheus-nodeexporter-z9twl/nodeexporter?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/monitoring-prometheus-nodeexporter-z9twl/nodeexporter?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/monitoring-prometheus-nodeexporter-z9twl/router?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/monitoring-prometheus-nodeexporter-z9twl/router?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod multicluster-hub-console-mcmuiapi-bfbd55ddc-q7bhn:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-console-mcmuiapi-bfbd55ddc-q7bhn/hcm-ui-api?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-console-mcmuiapi-bfbd55ddc-q7bhn/hcm-ui-api?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod multicluster-hub-etcd-0:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-etcd-0/etcd?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-etcd-0/etcd?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod multicluster-hub-grafeas-64f6c9c7c-xkq2f:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-grafeas-64f6c9c7c-xkq2f/grafeas?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-grafeas-64f6c9c7c-xkq2f/grafeas?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod multicluster-hub-grcuiapi-75c4fcd6bb-gsqm7:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-grcuiapi-75c4fcd6bb-gsqm7/grc-ui-api?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-grcuiapi-75c4fcd6bb-gsqm7/grc-ui-api?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod multicluster-hub-legato-5bbc7f776b-f4tnt:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-legato-5bbc7f776b-f4tnt/legato?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-legato-5bbc7f776b-f4tnt/legato?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod multicluster-hub-topologyapi-5db998858f-f62wt:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-topologyapi-5db998858f-f62wt/topology-ui-api?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-topologyapi-5db998858f-f62wt/topology-ui-api?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod search-search-api-7b4c978599-hbjxg:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/kube-system/search-search-api-7b4c978599-hbjxg/search-api?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/search-search-api-7b4c978599-hbjxg/search-api?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused]]
[must-gather-fmzgt] POD Gathering data for ns/openshift-dns-operator...
[must-gather-fmzgt] POD I1016 17:01:46.977748 18911 request.go:621] Throttling request took 1.119737143s, request: GET:https://172.30.0.1:443/apis/operators.coreos.com/v2?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD Gathering data for ns/openshift-dns...
[must-gather-fmzgt] POD I1016 17:02:04.151455 18911 request.go:621] Throttling request took 1.172491829s, request: GET:https://172.30.0.1:443/apis/autoscaling/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:02:14.281006 18911 request.go:621] Throttling request took 1.344193227s, request: GET:https://172.30.0.1:443/apis/storage.k8s.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-image-registry...
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:02:46.102991 18911 request.go:621] Throttling request took 1.169038689s, request: GET:https://172.30.0.1:443/apis/snapshot.storage.k8s.io/v1alpha1?timeout=32s
[must-gather-fmzgt] POD I1016 17:02:56.183159 18911 request.go:621] Throttling request took 1.137581381s, request: GET:https://172.30.0.1:443/apis/iks.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-ingress-operator...
[must-gather-fmzgt] POD I1016 17:03:06.332647 18911 request.go:621] Throttling request took 2.944012486s, request: GET:https://172.30.0.1:443/apis/eks.mcm.ibm.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD I1016 17:03:16.926684 18911 request.go:621] Throttling request took 1.14130196s, request: GET:https://172.30.0.1:443/apis/certmanager.k8s.io/v1alpha1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-insights...
[must-gather-fmzgt] POD I1016 17:03:27.001777 18911 request.go:621] Throttling request took 4.550486787s, request: GET:https://172.30.0.1:443/apis/policies.ibm.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD Gathering data for ns/openshift-kube-apiserver-operator...
[must-gather-fmzgt] POD I1016 17:03:37.132674 18911 request.go:621] Throttling request took 1.560671075s, request: GET:https://172.30.0.1:443/apis/metal3.io/v1alpha1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-kube-apiserver...
[must-gather-fmzgt] POD I1016 17:04:12.480810 18911 request.go:621] Throttling request took 1.136436021s, request: GET:https://172.30.0.1:443/apis/tuned.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:04:22.745906 18911 request.go:621] Throttling request took 1.133177531s, request: GET:https://172.30.0.1:443/apis/iks.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-kube-controller-manager...
[must-gather-fmzgt] POD I1016 17:04:53.124659 18911 request.go:621] Throttling request took 1.163356671s, request: GET:https://172.30.0.1:443/apis/gke.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-kube-controller-manager-operator...
[must-gather-fmzgt] POD I1016 17:05:03.625858 18911 request.go:621] Throttling request took 1.18155294s, request: GET:https://172.30.0.1:443/apis/samples.operator.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:05:13.675841 18911 request.go:621] Throttling request took 1.127826513s, request: GET:https://172.30.0.1:443/apis/foundation.ibm.com/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-kube-scheduler...
[must-gather-fmzgt] POD I1016 17:05:27.440991 18911 request.go:621] Throttling request took 1.153486096s, request: GET:https://172.30.0.1:443/apis/iks.mcm.ibm.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-kube-scheduler-operator...
[must-gather-fmzgt] POD I1016 17:05:37.831243 18911 request.go:621] Throttling request took 1.188463099s, request: GET:https://172.30.0.1:443/apis/operators.coreos.com/v2?timeout=32s
[must-gather-fmzgt] POD I1016 17:05:47.956329 18911 request.go:621] Throttling request took 1.344872495s, request: GET:https://172.30.0.1:443/apis/apiextensions.k8s.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD I1016 17:05:58.076203 18911 request.go:621] Throttling request took 1.731199689s, request: GET:https://172.30.0.1:443/apis/apiextensions.k8s.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-machine-config-operator...
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:06:10.969314 18911 request.go:621] Throttling request took 1.178139106s, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1alpha1?timeout=32s
[must-gather-fmzgt] POD I1016 17:06:21.016823 18911 request.go:621] Throttling request took 1.133973555s, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-marketplace...
[must-gather-fmzgt] POD I1016 17:06:31.341586 18911 request.go:621] Throttling request took 1.158852121s, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:06:41.478383 18911 request.go:621] Throttling request took 1.145756505s, request: GET:https://172.30.0.1:443/apis/batch/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-monitoring...
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:07:28.136788 18911 request.go:621] Throttling request took 1.18467835s, request: GET:https://172.30.0.1:443/apis/webhook.certmanager.k8s.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD I1016 17:07:39.049129 18911 request.go:621] Throttling request took 1.140491444s, request: GET:https://172.30.0.1:443/apis/policy.mcm.ibm.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:07:49.167384 18911 request.go:621] Throttling request took 1.742919383s, request: GET:https://172.30.0.1:443/apis/iks.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-multus...
[must-gather-fmzgt] POD I1016 17:08:09.954449 18911 request.go:621] Throttling request took 1.159932923s, request: GET:https://172.30.0.1:443/apis/helm.bitnami.com/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:08:19.962412 18911 request.go:621] Throttling request took 1.13352285s, request: GET:https://172.30.0.1:443/apis/ceph.rook.io/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:08:30.006264 18911 request.go:621] Throttling request took 1.344666194s, request: GET:https://172.30.0.1:443/apis/gke.mcm.ibm.com/v1beta1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD Gathering data for ns/openshift-sdn...
[must-gather-fmzgt] POD I1016 17:09:03.525301 18911 request.go:621] Throttling request took 1.114752014s, request: GET:https://172.30.0.1:443/apis/oidc.security.ibm.com/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:09:13.594780 18911 request.go:621] Throttling request took 1.138378347s, request: GET:https://172.30.0.1:443/apis/rook.io/v1alpha2?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:09:23.686777 18911 request.go:621] Throttling request took 1.544575107s, request: GET:https://172.30.0.1:443/apis/networking.k8s.io/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:09:33.875988 18911 request.go:621] Throttling request took 1.928791511s, request: GET:https://172.30.0.1:443/apis/user.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-network-operator...
[must-gather-fmzgt] POD I1016 17:09:51.800149 18911 request.go:621] Throttling request took 1.14678053s, request: GET:https://172.30.0.1:443/apis/aks.mcm.ibm.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-cluster-node-tuning-operator...
[must-gather-fmzgt] POD I1016 17:10:06.321358 18911 request.go:621] Throttling request took 1.18428208s, request: GET:https://172.30.0.1:443/apis/k8s.cni.cncf.io/v1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:10:16.562718 18911 request.go:621] Throttling request took 1.136374818s, request: GET:https://172.30.0.1:443/apis/gke.mcm.ibm.com/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-apiserver-operator...
[must-gather-fmzgt] POD I1016 17:10:26.841488 18911 request.go:621] Throttling request took 1.125865257s, request: GET:https://172.30.0.1:443/apis/imageregistry.operator.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-apiserver...
[must-gather-fmzgt] POD I1016 17:10:47.235972 18911 request.go:621] Throttling request took 1.170955106s, request: GET:https://172.30.0.1:443/apis/monitoringcontroller.cloud.ibm.com/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:10:57.244373 18911 request.go:621] Throttling request took 1.143384789s, request: GET:https://172.30.0.1:443/apis/quota.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:11:07.316725 18911 request.go:621] Throttling request took 1.545489137s, request: GET:https://172.30.0.1:443/apis/project.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:11:17.373250 18911 request.go:621] Throttling request took 1.927877252s, request: GET:https://172.30.0.1:443/apis/metrics.k8s.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-controller-manager-operator...
[must-gather-fmzgt] POD I1016 17:11:28.658040 18911 request.go:621] Throttling request took 1.163390578s, request: GET:https://172.30.0.1:443/apis/gke.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-controller-manager...
[must-gather-fmzgt] POD I1016 17:11:39.102765 18911 request.go:621] Throttling request took 1.17956634s, request: GET:https://172.30.0.1:443/apis/gke.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD I1016 17:11:49.242823 18911 request.go:621] Throttling request took 1.349993556s, request: GET:https://172.30.0.1:443/apis/rook.io/v1alpha2?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-cluster-samples-operator...
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:11:59.396366 18911 request.go:621] Throttling request took 4.985744602s, request: GET:https://172.30.0.1:443/apis/console.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift...
[must-gather-fmzgt] POD I1016 17:12:09.587400 18911 request.go:621] Throttling request took 4.763744808s, request: GET:https://172.30.0.1:443/apis/apps.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-operator-lifecycle-manager...
[must-gather-fmzgt] POD I1016 17:12:21.206747 18911 request.go:621] Throttling request took 1.189298501s, request: GET:https://172.30.0.1:443/apis/packages.operators.coreos.com/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:12:32.416335 18911 request.go:621] Throttling request took 1.142443048s, request: GET:https://172.30.0.1:443/apis/packages.operators.coreos.com/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-service-ca-operator...
[must-gather-fmzgt] POD I1016 17:12:42.531878 18911 request.go:621] Throttling request took 5.751902462s, request: GET:https://172.30.0.1:443/apis/user.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD Gathering data for ns/openshift-service-ca...
[must-gather-fmzgt] POD I1016 17:12:52.676006 18911 request.go:621] Throttling request took 5.583048544s, request: GET:https://172.30.0.1:443/apis/foundation.ibm.com/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-service-catalog-apiserver-operator...
[must-gather-fmzgt] POD I1016 17:13:02.739825 18911 request.go:621] Throttling request took 1.538497851s, request: GET:https://172.30.0.1:443/apis/build.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-service-catalog-apiserver...
[must-gather-fmzgt] POD I1016 17:13:15.233457 18911 request.go:621] Throttling request took 1.133366955s, request: GET:https://172.30.0.1:443/apis/authorization.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:13:25.297153 18911 request.go:621] Throttling request took 1.138701047s, request: GET:https://172.30.0.1:443/apis/iks.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-service-catalog-controller-manager-operator...
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:13:35.476723 18911 request.go:621] Throttling request took 5.151045348s, request: GET:https://172.30.0.1:443/apis/network.operator.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-service-catalog-controller-manager...
[must-gather-fmzgt] POD I1016 17:13:45.665777 18911 request.go:621] Throttling request took 2.777112679s, request: GET:https://172.30.0.1:443/apis/foundation.ibm.com/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-cluster-storage-operator...
[must-gather-fmzgt] POD I1016 17:13:55.747804 18911 request.go:621] Throttling request took 2.166805605s, request: GET:https://172.30.0.1:443/apis/ingress.operator.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD error: errors ocurred while gathering data:
[must-gather-fmzgt] POD [unable to retrieve the complete list of server APIs: clusterregistry.k8s.io/v1alpha1: the server is currently unable to handle the request, mcm.ibm.com/v1alpha1: the server is currently unable to handle the request, skipping gathering routes.route.openshift.io/oauth-openshift due to error: a resource cannot be retrieved by name across all namespaces, skipping gathering services/oauth-openshift due to error: a resource cannot be retrieved by name across all namespaces, skipping gathering machineautoscalers.machine.openshift.io due to error: the server doesn't have a resource type "machineautoscalers", skipping gathering clusterautoscalers.machine.openshift.io due to error: the server doesn't have a resource type "clusterautoscalers", skipping gathering namespaces/openshift-console due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-console
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD one or more errors ocurred while gathering container data for pod downloads-5fdf68d856-rzcml:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/openshift-console/downloads-5fdf68d856-rzcml/download-server?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-console/downloads-5fdf68d856-rzcml/download-server?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], skipping gathering namespaces/openshift-dns due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-dns
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD one or more errors ocurred while gathering container data for pod dns-default-85jts:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/openshift-dns/dns-default-85jts/dns?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-dns/dns-default-85jts/dns?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-dns/dns-default-85jts/dns-node-resolver?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-dns/dns-default-85jts/dns-node-resolver?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], skipping gathering namespaces/openshift-image-registry due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-image-registry
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD one or more errors ocurred while gathering container data for pod node-ca-mnt9j:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/openshift-image-registry/node-ca-mnt9j/node-ca?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-image-registry/node-ca-mnt9j/node-ca?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], skipping gathering namespaces/openshift-machine-config-operator due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-machine-config-operator
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD one or more errors ocurred while gathering container data for pod machine-config-daemon-xpkk6:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/openshift-machine-config-operator/machine-config-daemon-xpkk6/machine-config-daemon?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-machine-config-operator/machine-config-daemon-xpkk6/machine-config-daemon?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-machine-config-operator/machine-config-daemon-xpkk6/oauth-proxy?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-machine-config-operator/machine-config-daemon-xpkk6/oauth-proxy?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], skipping gathering namespaces/openshift-marketplace due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-marketplace
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD one or more errors ocurred while gathering container data for pod redhat-operators-64ddc67cdb-j5g9n:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/openshift-marketplace/redhat-operators-64ddc67cdb-j5g9n/redhat-operators?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-marketplace/redhat-operators-64ddc67cdb-j5g9n/redhat-operators?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], skipping gathering namespaces/openshift-monitoring due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-monitoring
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD one or more errors ocurred while gathering container data for pod node-exporter-xr2wg:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/openshift-monitoring/node-exporter-xr2wg/node-exporter?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-monitoring/node-exporter-xr2wg/node-exporter?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-monitoring/node-exporter-xr2wg/kube-rbac-proxy?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-monitoring/node-exporter-xr2wg/kube-rbac-proxy?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-monitoring/node-exporter-xr2wg/init-textfile?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-monitoring/node-exporter-xr2wg/init-textfile?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], skipping gathering all/openshift-monitoring due to error: the server doesn't have a resource type "all", skipping gathering namespaces/openshift-multus due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-multus
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD one or more errors ocurred while gathering container data for pod multus-zr88j:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/openshift-multus/multus-zr88j/kube-multus?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-multus/multus-zr88j/kube-multus?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-multus/multus-zr88j/cni-plugins?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-multus/multus-zr88j/cni-plugins?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-multus/multus-zr88j/multus-binary-copy?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-multus/multus-zr88j/multus-binary-copy?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], skipping gathering namespaces/openshift-sdn due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-sdn
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [one or more errors ocurred while gathering container data for pod ovs-tkvkm:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/openshift-sdn/ovs-tkvkm/openvswitch?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-sdn/ovs-tkvkm/openvswitch?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod sdn-m8f4s:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/openshift-sdn/sdn-m8f4s/sdn?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-sdn/sdn-m8f4s/sdn?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-sdn/sdn-m8f4s/install-cni-plugins?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-sdn/sdn-m8f4s/install-cni-plugins?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused]], skipping gathering namespaces/openshift-cluster-node-tuning-operator due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-cluster-node-tuning-operator
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD one or more errors ocurred while gathering container data for pod tuned-vbc84:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/openshift-cluster-node-tuning-operator/tuned-vbc84/tuned?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-cluster-node-tuning-operator/tuned-vbc84/tuned?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused]]
[must-gather-fmzgt] POD INFO: gather_ocp log collection complete.
[must-gather-fmzgt] POD WARNING: Collecting one or more service logs on ALL master nodes in your cluster. This could take a large amount of time.
[must-gather-fmzgt] POD INFO: Collecting host service logs for kubelet
[must-gather-fmzgt] POD INFO: Collecting host service logs for crio
[must-gather-fmzgt] POD INFO: Waiting for worker host service log collection to complete ...
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD INFO: Worker host service log collection complete.
[must-gather-fmzgt] POD INFO: gather_cloudpak collecting ...
[must-gather-fmzgt] POD INFO: gather_namespace_scoped_resources collecting ...
[must-gather-fmzgt] POD INFO: gather_namespace_scoped_resources collecting namespace cem.
[must-gather-fmzgt] POD INFO: gather_namespace_scoped_resources collecting resource cem.ingress.
[must-gather-fmzgt] POD INFO: Waiting for gather_namespace_scoped_resources log collection to complete ...
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Gathering data for ns/cem...
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:15:22.883189 19347 request.go:621] Throttling request took 1.150248008s, request: GET:https://172.30.0.1:443/apis/k8s.cni.cncf.io/v1?timeout=32s
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD error: errors ocurred while gathering data:
[must-gather-fmzgt] POD one or more errors ocurred while gathering pod-specific data for namespace: cem
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [one or more errors ocurred while gathering container data for pod peter-cassandra-0:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/cem/peter-cassandra-0/peter-cassandra?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-cassandra-0/peter-cassandra?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5/akora-app-cem?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5/akora-app-cem?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5/waitforredis?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5/waitforredis?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod peter-ibm-cem-datalayer-0:
[must-gather-fmzgt] POD
[must-gather-fmzgt] POD [Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-datalayer-0/datalayer?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-datalayer-0/datalayer?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-datalayer-0/waitforcassandra?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-datalayer-0/waitforcassandra?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-datalayer-0/setupdb?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-datalayer-0/setupdb?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-datalayer-0/waitforkafka?previous=true×tamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-datalayer-0/waitforkafka?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused]]
[must-gather-fmzgt] POD INFO: gather_namespace_scoped_resources log collection complete.
[must-gather-fmzgt] POD INFO: gather_cluster_scoped_resources collecting ...
[must-gather-fmzgt] POD INFO: gather_cluster_scoped_resources collecting nodes.
[must-gather-fmzgt] POD INFO: gather_cluster_scoped_resources collecting storageclasses.
[must-gather-fmzgt] POD INFO: gather_cluster_scoped_resources collecting persistentvolumes.
[must-gather-fmzgt] POD INFO: gather_cluster_scoped_resources collecting volumeattachments.
[must-gather-fmzgt] POD INFO: gather_cluster_scoped_resources collecting clusterservicestatuses.clusterhealth.ibm.com.
[must-gather-fmzgt] POD INFO: Waiting for gather_cluster_scoped_resources log collection to complete ...
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD I1016 17:15:33.821207 19390 request.go:621] Throttling request took 1.134028301s, request: GET:https://172.30.0.1:443/apis/aks.mcm.ibm.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD error: the server doesn't have a resource type "clusterservicestatuses"
[must-gather-fmzgt] POD INFO: gather_cluster_scoped_resources log collection complete.
[must-gather-fmzgt] POD INFO: gather_cloudpak log collection complete.
[must-gather-fmzgt] POD INFO: gather_cleanup collecting ...
[must-gather-fmzgt] POD INFO: checking and cleanning secrets.yaml (by default we do not collect secrets.yaml from your cluster) ...
[must-gather-fmzgt] POD ./namespaces/openshift-etcd/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-cluster-version/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/default/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/kube-system/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-config/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-config-managed/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-authentication/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-authentication-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-ingress/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-cloud-credential-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-machine-api/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-console-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-console/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-dns-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-dns/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-image-registry/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-ingress-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-insights/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-kube-apiserver-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-kube-apiserver/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-kube-controller-manager/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-kube-scheduler/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-kube-scheduler-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-machine-config-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-marketplace/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-monitoring/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-apiserver-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-apiserver/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-cluster-samples-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-controller-manager-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-controller-manager/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-multus/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-sdn/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-service-catalog-apiserver-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-service-catalog-controller-manager-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-service-catalog-controller-manager/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-network-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-service-ca-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-service-ca/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-service-catalog-apiserver/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-cluster-storage-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/cem/core/secrets.yaml
[must-gather-fmzgt] POD INFO: checking and cleanning routes.yaml ...
[must-gather-fmzgt] POD No resources found in openshift-etcd namespace.
[must-gather-fmzgt] POD No resources found in openshift namespace.
[must-gather-fmzgt] POD No resources found in openshift-cluster-version namespace.
[must-gather-fmzgt] POD No resources found in default namespace.
[must-gather-fmzgt] POD No resources found in openshift-config namespace.
[must-gather-fmzgt] POD No resources found in openshift-config-managed namespace.
[must-gather-fmzgt] POD No resources found in openshift-authentication-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-ingress namespace.
[must-gather-fmzgt] POD No resources found in openshift-cloud-credential-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-machine-api namespace.
[must-gather-fmzgt] POD No resources found in openshift-console-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-dns-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-dns namespace.
[must-gather-fmzgt] POD No resources found in openshift-ingress-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-insights namespace.
[must-gather-fmzgt] POD No resources found in openshift-kube-apiserver-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-kube-apiserver namespace.
[must-gather-fmzgt] POD No resources found in openshift-kube-controller-manager namespace.
[must-gather-fmzgt] POD No resources found in openshift-kube-controller-manager-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-kube-scheduler namespace.
[must-gather-fmzgt] POD No resources found in openshift-kube-scheduler-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-machine-config-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-marketplace namespace.
[must-gather-fmzgt] POD No resources found in openshift-apiserver-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-apiserver namespace.
[must-gather-fmzgt] POD No resources found in openshift-cluster-samples-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-controller-manager-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-controller-manager namespace.
[must-gather-fmzgt] POD No resources found in openshift-multus namespace.
[must-gather-fmzgt] POD No resources found in openshift-operator-lifecycle-manager namespace.
[must-gather-fmzgt] POD No resources found in openshift-sdn namespace.
[must-gather-fmzgt] POD No resources found in openshift-service-catalog-apiserver-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-service-catalog-controller-manager-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-service-catalog-controller-manager namespace.
[must-gather-fmzgt] POD No resources found in openshift-network-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-cluster-node-tuning-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-service-ca-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-service-ca namespace.
[must-gather-fmzgt] POD No resources found in openshift-service-catalog-apiserver namespace.
[must-gather-fmzgt] POD No resources found in openshift-cluster-storage-operator namespace.
[must-gather-fmzgt] POD No resources found in cem namespace.
[must-gather-fmzgt] POD INFO: tar collections to cloudpak-must-gather-20201016171535.tar.gz ...
[must-gather-fmzgt] POD ./cluster-scoped-resources/
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/nfs1.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-02094b7e-2147-4a3d-ae27-7a0d889b5e11.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-12737f05-b8b7-43e7-ac77-efccb4d0f6b5.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-25aedbe8-6bf3-4d3c-a833-be9eaf74b657.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-27c65d1e-e727-4004-a111-8ddecdc51354.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-3f31114a-488f-456f-aa5a-95f2ad280816.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-4b723cbe-d65b-45bb-aa6b-4e1cab7741e2.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-52cc0d71-dc14-4601-bfd3-67858c283cf1.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-62fbd259-23a2-4c2d-b9b9-1dbc700bbd42.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-683927fc-e90d-44d7-acfc-38969347620e.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-6b4375fe-eed0-4bc2-b9a3-78ae69974505.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-7e9a97e3-623c-4d21-8a24-8d9a39f5bbe2.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-80666146-c50f-4b39-9487-7d061ae8d08d.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-84fdb65b-e2ba-49b2-b87e-daac3bc9eaca.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-d8697f54-4ba7-4837-8b2c-eab906cd6d26.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-d8ac96f1-1410-4ba6-a6fe-b16acebc7b37.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/registry-storage.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/master0.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/master1.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/master2.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/worker0.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/worker1.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/worker2.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/worker3.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/worker4.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/worker5.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/worker6.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigpools/
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigpools/master.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigpools/worker.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/00-master.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/00-worker.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/01-master-container-runtime.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/01-master-kubelet.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/01-worker-container-runtime.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/01-worker-kubelet.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/99-master-6d17d94e-ed2b-48a2-8c56-40ebb111cbfb-registries.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/99-master-ssh.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/99-worker-c5cb75e7-e62e-47a8-a20a-759bb6f378c4-registries.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/99-worker-ssh.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/rendered-master-3b8c0988716af71315cbdc15f54c4f25.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/rendered-master-af998b63598fe47dcd8ad07ad544aa53.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/rendered-master-cdace211abd164b3170cd908fb3cdaa3.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/rendered-worker-00ad89a6b224986f8b48c918aeb2b2fa.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/rendered-worker-4d08c8783325e6d08177c8764e2e1777.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/rendered-worker-793cbb543301736c9e5768982d0598fd.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/controllerconfigs/
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/controllerconfigs/machine-config-controller.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/storageclasses/
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/storageclasses/rook-ceph-block-internal.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/storageclasses/rook-ceph-cephfs-internal.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/storageclasses/rook-ceph-delete-bucket-internal.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-222503cde8b309a2680a39df212408c220000d98d79312487af66be0040ea77a.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-2faabee05930403b3ccfe0eb11b85383d3ce23d64837046001248623d5d7faae.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-51920a5f6a0958f1a5c929173b8dff218b0a0163ed2c71b4218768d1eaf079a4.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-8afa11face1adb593b2509ad043c8e4d771c7eb96f3c6497f7ef1b3eebfafae0.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-b578f63ff33deae8e7e2c9be80c5ad86376df51d176c5b34546a9b692133c59f.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-d8163010f312347ce460bbbe8bd871b07cbfa98078e0454d71117b28278dc55d.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-df1f36a2ccc0d3f19f19dd4475e4afce1f0e033604e18639e36c7b27931addbf.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-e4b1be47a274221a16bcc5e5a249f2f68202675ff4e2520e76990762a33c40cf.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-f5a5a77e4d1b903153a4b5dc9e6e6ca9e51b758f2101eb50a66f013789de3b28.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-f6230732b352fef8bad1e4976b8deddc3f643e42098107210f714fd3b3d257fd.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusterversions/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/images.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/networks.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/projects.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/builds.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusterversions.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/ingresses.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/oauths.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/dnses.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/proxies.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/operatorhubs.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/infrastructures.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/authentications.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/consoles.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/schedulers.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/apiservers.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/featuregates.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/authentication.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/cloud-credential.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/cluster-autoscaler.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/machine-api.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/machine-config.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/marketplace.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/service-catalog-apiserver.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/service-catalog-controller-manager.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/authentications/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/authentications/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/infrastructures/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/oauths/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/consoles/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/proxies/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/images/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/images/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/imagecontentsourcepolicies/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/imagecontentsourcepolicies/mirror-config.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/networks/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/authentications/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/authentications/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/consoles/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/dnses/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/dnses/default.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/kubeapiservers/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/kubeschedulers/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/openshiftapiservers/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/servicecas/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/servicecatalogcontrollermanagers/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/servicecatalogcontrollermanagers/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/oauth.openshift.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/oauth.openshift.io/oauthclients/
[must-gather-fmzgt] POD ./cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/imageregistry.operator.openshift.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/imageregistry.operator.openshift.io/configs/
[must-gather-fmzgt] POD ./cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/machine-api-operator.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/machine-api-controllers.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-sdn.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-sdn-controller.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning:tuned.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-sdn.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-sdn-controller.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning:tuned.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiextensions.k8s.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusternetworks.network.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/hostsubnets.network.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/netnamespaces.network.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressnetworkpolicies.network.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/admissionregistration.k8s.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/
[must-gather-fmzgt] POD ./cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/network.openshift.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/network.openshift.io/clusternetworks/
[must-gather-fmzgt] POD ./cluster-scoped-resources/network.openshift.io/clusternetworks/default.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.servicecatalog.k8s.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/samples.operator.openshift.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/samples.operator.openshift.io/configs/
[must-gather-fmzgt] POD ./cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml
[must-gather-fmzgt] POD ./failure/
[must-gather-fmzgt] POD ./failure/pods/
[must-gather-fmzgt] POD ./failure/pods/pods
[must-gather-fmzgt] POD ./failure/pods/cem.peter-cassandra-0.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-cassandra-0.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-cassandra-0.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-cassandra-0.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-0.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-0.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-0.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-0.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602724500-tk6c5.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602724500-tk6c5.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602724500-tk6c5.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602724500-tk6c5.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-596n6.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-596n6.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-596n6.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-596n6.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-6nr5m.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-6nr5m.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-6nr5m.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-6nr5m.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-d7cpl.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-d7cpl.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-d7cpl.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-d7cpl.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-q9xjd.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-q9xjd.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-q9xjd.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-q9xjd.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-rjzng.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-rjzng.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-rjzng.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-rjzng.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-zvcv4.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-zvcv4.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-zvcv4.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-zvcv4.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-console-mcmuiapi-bfbd55ddc-q7bhn.current.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-console-mcmuiapi-bfbd55ddc-q7bhn.get.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-console-mcmuiapi-bfbd55ddc-q7bhn.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-console-mcmuiapi-bfbd55ddc-q7bhn.previous.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-core-apiserver-7f6645f546-gwcbp.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-core-apiserver-7f6645f546-gwcbp.current.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-core-apiserver-7f6645f546-gwcbp.get.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-core-apiserver-7f6645f546-gwcbp.previous.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-grafeas-64f6c9c7c-xkq2f.current.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-grafeas-64f6c9c7c-xkq2f.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-grafeas-64f6c9c7c-xkq2f.get.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-grafeas-64f6c9c7c-xkq2f.previous.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-grcuiapi-75c4fcd6bb-gsqm7.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-grcuiapi-75c4fcd6bb-gsqm7.current.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-grcuiapi-75c4fcd6bb-gsqm7.previous.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-grcuiapi-75c4fcd6bb-gsqm7.get.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-legato-5bbc7f776b-f4tnt.get.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-legato-5bbc7f776b-f4tnt.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-legato-5bbc7f776b-f4tnt.previous.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-legato-5bbc7f776b-f4tnt.current.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-topologyapi-5db998858f-f62wt.get.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-topologyapi-5db998858f-f62wt.previous.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-topologyapi-5db998858f-f62wt.current.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-topologyapi-5db998858f-f62wt.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.search-search-api-7b4c978599-hbjxg.get.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.search-search-api-7b4c978599-hbjxg.current.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.search-search-api-7b4c978599-hbjxg.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.search-search-api-7b4c978599-hbjxg.previous.log
[must-gather-fmzgt] POD ./failure/pods/openshift-console.downloads-5fdf68d856-rzcml.get.yaml
[must-gather-fmzgt] POD ./failure/pods/openshift-console.downloads-5fdf68d856-rzcml.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/openshift-console.downloads-5fdf68d856-rzcml.current.log
[must-gather-fmzgt] POD ./failure/pods/openshift-console.downloads-5fdf68d856-rzcml.previous.log
[must-gather-fmzgt] POD ./failure/pods/openshift-kube-apiserver.[installer-67](https://issues.redhat.com/browse/installer-67)-master2.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/openshift-kube-apiserver.[installer-67](https://issues.redhat.com/browse/installer-67)-master2.scao3-ocp43.os.fyre.ibm.com.current.log
[must-gather-fmzgt] POD ./failure/pods/openshift-kube-apiserver.[installer-67](https://issues.redhat.com/browse/installer-67)-master2.scao3-ocp43.os.fyre.ibm.com.get.yaml
[must-gather-fmzgt] POD ./failure/pods/openshift-kube-apiserver.[installer-67](https://issues.redhat.com/browse/installer-67)-master2.scao3-ocp43.os.fyre.ibm.com.previous.log
[must-gather-fmzgt] POD ./failure/pods/openshift-kube-apiserver.revision-pruner-66-master1.scao3-ocp43.os.fyre.ibm.com.current.log
[must-gather-fmzgt] POD ./failure/pods/openshift-kube-apiserver.revision-pruner-66-master1.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/openshift-kube-apiserver.revision-pruner-66-master1.scao3-ocp43.os.fyre.ibm.com.get.yaml
[must-gather-fmzgt] POD ./failure/pods/openshift-kube-apiserver.revision-pruner-66-master1.scao3-ocp43.os.fyre.ibm.com.previous.log
[must-gather-fmzgt] POD ./failure/pods/rook-ceph.rook-ceph-osd-6-58bd78b9b7-d2qsp.previous.log
[must-gather-fmzgt] POD ./failure/pods/rook-ceph.rook-ceph-osd-6-58bd78b9b7-d2qsp.current.log
[must-gather-fmzgt] POD ./failure/pods/rook-ceph.rook-ceph-osd-6-58bd78b9b7-d2qsp.get.yaml
[must-gather-fmzgt] POD ./failure/pods/rook-ceph.rook-ceph-osd-6-58bd78b9b7-d2qsp.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/
[must-gather-fmzgt] POD ./failure/restart_pods/restart_pods
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-datalayer-0.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-datalayer-0.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-datalayer-0.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-datalayer-0.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-rba-as-566bcffb6-tnzqp.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-rba-as-566bcffb6-tnzqp.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-rba-as-566bcffb6-tnzqp.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-rba-as-566bcffb6-tnzqp.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-rba-rbs-689475646d-png8d.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-rba-rbs-689475646d-png8d.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-rba-rbs-689475646d-png8d.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-rba-rbs-689475646d-png8d.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cert-manager.cert-manager-ibm-cert-manager-5cd9fb9777-cks5p.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/cert-manager.cert-manager-ibm-cert-manager-5cd9fb9777-cks5p.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cert-manager.cert-manager-ibm-cert-manager-5cd9fb9777-cks5p.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/cert-manager.cert-manager-ibm-cert-manager-5cd9fb9777-cks5p.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cert-manager.cert-manager-webhook-cainjector-f46ff7b74-p442r.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cert-manager.cert-manager-webhook-cainjector-f46ff7b74-p442r.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/cert-manager.cert-manager-webhook-cainjector-f46ff7b74-p442r.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cert-manager.cert-manager-webhook-cainjector-f46ff7b74-p442r.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-bckx9.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-bckx9.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-bckx9.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-bckx9.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-z2bwn.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-z2bwn.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-z2bwn.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-z2bwn.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-zkg7d.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-zkg7d.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-zkg7d.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-zkg7d.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-ff9rk.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-ff9rk.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-ff9rk.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-ff9rk.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-grgp4.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-grgp4.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-grgp4.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-grgp4.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-ln9rd.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-ln9rd.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-ln9rd.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-ln9rd.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-4ndvq.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-4ndvq.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-4ndvq.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-4ndvq.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-gw6tk.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-gw6tk.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-gw6tk.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-gw6tk.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-jl2lz.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-jl2lz.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-jl2lz.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-jl2lz.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-wpwtq.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-wpwtq.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-wpwtq.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-wpwtq.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-zk2l6.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-zk2l6.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-zk2l6.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-zk2l6.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-zw46s.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-zw46s.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-zw46s.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-zw46s.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.icp-mongodb-0.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.icp-mongodb-0.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.icp-mongodb-0.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.icp-mongodb-0.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.icp-mongodb-1.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.icp-mongodb-1.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.icp-mongodb-1.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.icp-mongodb-1.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.internal-management-ingress-9gwsh.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.internal-management-ingress-9gwsh.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.internal-management-ingress-9gwsh.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.internal-management-ingress-9gwsh.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.internal-management-ingress-scgxp.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.internal-management-ingress-scgxp.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.internal-management-ingress-scgxp.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.internal-management-ingress-scgxp.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-dm-c7b75c686-n2b7h.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-dm-c7b75c686-n2b7h.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-dm-c7b75c686-n2b7h.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-dm-c7b75c686-n2b7h.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-7522r.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-7522r.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-7522r.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-7522r.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-7bwdl.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-7bwdl.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-7bwdl.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-7bwdl.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-8x48b.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-8x48b.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-8x48b.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-8x48b.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-bhpnk.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-bhpnk.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-bhpnk.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-bhpnk.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-l5wsj.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-l5wsj.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-l5wsj.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-l5wsj.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-ppzhz.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-ppzhz.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-ppzhz.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-ppzhz.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-qsj6z.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-qsj6z.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-qsj6z.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-qsj6z.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.monitoring-grafana-67ddcdf99d-k7df4.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.monitoring-grafana-67ddcdf99d-k7df4.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.monitoring-grafana-67ddcdf99d-k7df4.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.monitoring-grafana-67ddcdf99d-k7df4.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.monitoring-prometheus-nodeexporter-2cd57.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.monitoring-prometheus-nodeexporter-2cd57.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.monitoring-prometheus-nodeexporter-2cd57.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.monitoring-prometheus-nodeexporter-2cd57.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-aks-b84b6657-pqhgx.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-aks-b84b6657-pqhgx.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-aks-b84b6657-pqhgx.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-aks-b84b6657-pqhgx.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-eks-c8b6c956f-bmrlz.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-eks-c8b6c956f-bmrlz.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-eks-c8b6c956f-bmrlz.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-eks-c8b6c956f-bmrlz.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-gke-547c5c7b59-kpprf.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-gke-547c5c7b59-kpprf.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-gke-547c5c7b59-kpprf.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-gke-547c5c7b59-kpprf.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-iks-7c4999db75-fm62g.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-iks-7c4999db75-fm62g.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-iks-7c4999db75-fm62g.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-iks-7c4999db75-fm62g.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-ocp-6d5fbd8459-ckg46.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-ocp-6d5fbd8459-ckg46.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-ocp-6d5fbd8459-ckg46.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-ocp-6d5fbd8459-ckg46.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-sdk-b6979fffb-fm89p.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-sdk-b6979fffb-fm89p.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-sdk-b6979fffb-fm89p.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-sdk-b6979fffb-fm89p.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-core-apiserver-7f6645f546-gwcbp.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-core-apiserver-7f6645f546-gwcbp.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-core-apiserver-7f6645f546-gwcbp.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-core-apiserver-7f6645f546-gwcbp.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-core-channel-59fdf84bf4-5sjm4.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-core-channel-59fdf84bf4-5sjm4.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-core-channel-59fdf84bf4-5sjm4.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-core-channel-59fdf84bf4-5sjm4.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-czjkn.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-czjkn.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-czjkn.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-czjkn.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-h95x9.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-h95x9.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-h95x9.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-h95x9.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-smppk.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-smppk.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-smppk.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-smppk.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-api-794c95849c-jhjpn.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-api-794c95849c-jhjpn.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-api-794c95849c-jhjpn.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-api-794c95849c-jhjpn.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-api-794c95849c-wz89k.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-api-794c95849c-wz89k.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-api-794c95849c-wz89k.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-api-794c95849c-wz89k.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-49gzc.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-49gzc.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-49gzc.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-49gzc.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-s8tp2.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-s8tp2.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-s8tp2.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-s8tp2.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-xhp4k.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-xhp4k.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-xhp4k.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-xhp4k.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.prometheus-monitoring-prometheus-0.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.prometheus-monitoring-prometheus-0.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.prometheus-monitoring-prometheus-0.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.prometheus-monitoring-prometheus-0.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.search-search-aggregator-64b75d9466-lj9ms.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.search-search-aggregator-64b75d9466-lj9ms.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.search-search-aggregator-64b75d9466-lj9ms.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.search-search-aggregator-64b75d9466-lj9ms.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.search-search-collector-6c68bd89bd-rhmt2.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.search-search-collector-6c68bd89bd-rhmt2.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.search-search-collector-6c68bd89bd-rhmt2.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.search-search-collector-6c68bd89bd-rhmt2.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.secret-watcher-779df94cc8-z7g8c.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.secret-watcher-779df94cc8-z7g8c.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.secret-watcher-779df94cc8-z7g8c.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.secret-watcher-779df94cc8-z7g8c.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver-operator.openshift-apiserver-operator-7cc77d965f-x8pff.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver-operator.openshift-apiserver-operator-7cc77d965f-x8pff.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver-operator.openshift-apiserver-operator-7cc77d965f-x8pff.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver-operator.openshift-apiserver-operator-7cc77d965f-x8pff.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-8rpk8.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-8rpk8.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-8rpk8.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-8rpk8.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-j5gsv.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-j5gsv.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-j5gsv.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-j5gsv.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-lm8sl.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-lm8sl.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-lm8sl.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-lm8sl.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-authentication-operator.authentication-operator-57d4974d5d-6tk6f.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-authentication-operator.authentication-operator-57d4974d5d-6tk6f.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-authentication-operator.authentication-operator-57d4974d5d-6tk6f.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-authentication-operator.authentication-operator-57d4974d5d-6tk6f.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-cloud-credential-operator.cloud-credential-operator-dd496454-57rdg.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-cloud-credential-operator.cloud-credential-operator-dd496454-57rdg.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-cluster-version.cluster-version-operator-584fddff45-tgdkd.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-cloud-credential-operator.cloud-credential-operator-dd496454-57rdg.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-cloud-credential-operator.cloud-credential-operator-dd496454-57rdg.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-cluster-version.cluster-version-operator-584fddff45-tgdkd.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-cluster-version.cluster-version-operator-584fddff45-tgdkd.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-cluster-version.cluster-version-operator-584fddff45-tgdkd.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-console-operator.console-operator-57f5bcc578-6q5vj.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-console-operator.console-operator-57f5bcc578-6q5vj.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-console-operator.console-operator-57f5bcc578-6q5vj.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-console-operator.console-operator-57f5bcc578-6q5vj.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager-operator.openshift-controller-manager-operator-68dcf95c47-6bz2w.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager-operator.openshift-controller-manager-operator-68dcf95c47-6bz2w.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager-operator.openshift-controller-manager-operator-68dcf95c47-6bz2w.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager-operator.openshift-controller-manager-operator-68dcf95c47-6bz2w.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager.controller-manager-2zczq.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager.controller-manager-2zczq.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager.controller-manager-2zczq.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager.controller-manager-2zczq.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager.controller-manager-6r22f.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager.controller-manager-6r22f.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager.controller-manager-6r22f.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager.controller-manager-6r22f.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-85jts.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-85jts.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-85jts.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-85jts.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-c9czj.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-c9czj.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-c9czj.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-c9czj.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-f87fn.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-f87fn.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-f87fn.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-f87fn.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-klp8r.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-klp8r.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-klp8r.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-klp8r.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-m9p5v.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-m9p5v.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-m9p5v.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-m9p5v.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-mnfxh.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-mnfxh.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-mnfxh.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-mnfxh.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-p62vs.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-p62vs.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-p62vs.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-p62vs.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-qbs6q.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-qbs6q.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-qbs6q.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-qbs6q.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-vvw7j.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-vvw7j.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-vvw7j.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-vvw7j.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-wxv5g.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-wxv5g.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-wxv5g.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-wxv5g.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master0.scao3-ocp43.os.fyre.ibm.com.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master0.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master0.scao3-ocp43.os.fyre.ibm.com.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master0.scao3-ocp43.os.fyre.ibm.com.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master1.scao3-ocp43.os.fyre.ibm.com.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master1.scao3-ocp43.os.fyre.ibm.com.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master1.scao3-ocp43.os.fyre.ibm.com.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master1.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master2.scao3-ocp43.os.fyre.ibm.com.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master2.scao3-ocp43.os.fyre.ibm.com.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master2.scao3-ocp43.os.fyre.ibm.com.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master2.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-55mpk.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-55mpk.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-55mpk.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-55mpk.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-8gljk.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-8gljk.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-8gljk.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-8gljk.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-fwp52.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-fwp52.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-fwp52.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-fwp52.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-h4p5w.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-h4p5w.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-h4p5w.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-h4p5w.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-hrsqc.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-hrsqc.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-hrsqc.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-hrsqc.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-j9gfj.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-j9gfj.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-j9gfj.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-j9gfj.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-ktrnn.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-ktrnn.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-ktrnn.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-ktrnn.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-mnt9j.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-mnt9j.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-mnt9j.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-mnt9j.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-wpzqh.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-wpzqh.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-wpzqh.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-wpzqh.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-xmr4j.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-xmr4j.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-xmr4j.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-xmr4j.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver-operator.kube-apiserver-operator-566b9798-7rjbl.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver-operator.kube-apiserver-operator-566b9798-7rjbl.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver-operator.kube-apiserver-operator-566b9798-7rjbl.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver-operator.kube-apiserver-operator-566b9798-7rjbl.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master0.scao3-ocp43.os.fyre.ibm.com.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master0.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master0.scao3-ocp43.os.fyre.ibm.com.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master0.scao3-ocp43.os.fyre.ibm.com.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master1.scao3-ocp43.os.fyre.ibm.com.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master1.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master1.scao3-ocp43.os.fyre.ibm.com.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master1.scao3-ocp43.os.fyre.ibm.com.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master2.scao3-ocp43.os.fyre.ibm.com.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master2.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master2.scao3-ocp43.os.fyre.ibm.com.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master2.scao3-ocp43.os.fyre.ibm.com.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-controller-manager-operator.kube-controller-manager-operator-7c8b7465b-gd8nf.get.yaml
...
[must-gather-fmzgt] POD ./overview/nodes.worker1.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./overview/nodes.worker2.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./overview/nodes.worker3.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./overview/nodes.worker4.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./overview/nodes.worker5.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./overview/nodes.worker6.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./overview/pods
[must-gather-fmzgt] POD ./overview/deployments
[must-gather-fmzgt] POD ./overview/statefulsets
[must-gather-fmzgt] POD ./overview/daemonsets
[must-gather-fmzgt] POD ./overview/jobs
[must-gather-fmzgt] POD ./overview/cronjobs
[must-gather-fmzgt] POD ./overview/pod_resources
[must-gather-fmzgt] POD ./overview/configmaps
[must-gather-fmzgt] POD ./overview/secrets
[must-gather-fmzgt] POD ./overview/resourcequotas
[must-gather-fmzgt] POD ./overview/hpas
[must-gather-fmzgt] POD ./overview/services
[must-gather-fmzgt] POD ./overview/endpoints
[must-gather-fmzgt] POD ./overview/ingresses
[must-gather-fmzgt] POD ./overview/routes
[must-gather-fmzgt] POD ./overview/networkpolicies
[must-gather-fmzgt] POD ./overview/persistentvolumes
[must-gather-fmzgt] POD ./overview/persistentvolumeclaims
[must-gather-fmzgt] POD ./overview/storageclasses
[must-gather-fmzgt] POD ./overview/namespaces
[must-gather-fmzgt] POD ./overview/events
[must-gather-fmzgt] POD ./overview/crds
[must-gather-fmzgt] POD ./overview/operator.ibm.com
[must-gather-fmzgt] POD ./overview/csvs
[must-gather-fmzgt] POD ./overview/certificates.certmanager
[must-gather-fmzgt] POD ./overview/challenges.certmanager
[must-gather-fmzgt] POD ./overview/clusterissuers.certmanager
[must-gather-fmzgt] POD ./overview/issuers.certmanager
[must-gather-fmzgt] POD ./overview/orders.certmanager
[must-gather-fmzgt] POD ./overview/status
[must-gather-fmzgt] POD ./overview/clusterservicestatus
[must-gather-fmzgt] POD ./summary
[must-gather-fmzgt] POD ./system/
[must-gather-fmzgt] POD ./system/host_info/
[must-gather-fmzgt] POD ./system/host_info/master2.scao3-ocp43.os.fyre.ibm.com.proc.cpuinfo
[must-gather-fmzgt] POD ./system/host_info/master2.scao3-ocp43.os.fyre.ibm.com.proc.meminfo
[must-gather-fmzgt] POD ./system/host_info/master2.scao3-ocp43.os.fyre.ibm.com.ip.address
[must-gather-fmzgt] POD ./system/host_info/master2.scao3-ocp43.os.fyre.ibm.com.df
[must-gather-fmzgt] POD ./system/host_info/master2.scao3-ocp43.os.fyre.ibm.com.dmesg
[must-gather-fmzgt] POD ./system/host_info/master1.scao3-ocp43.os.fyre.ibm.com.proc.cpuinfo
[must-gather-fmzgt] POD ./system/host_info/master1.scao3-ocp43.os.fyre.ibm.com.proc.meminfo
[must-gather-fmzgt] POD ./system/host_info/master1.scao3-ocp43.os.fyre.ibm.com.df
[must-gather-fmzgt] POD ./system/host_info/master1.scao3-ocp43.os.fyre.ibm.com.dmesg
[must-gather-fmzgt] POD ./system/host_info/master1.scao3-ocp43.os.fyre.ibm.com.ip.address
[must-gather-fmzgt] POD ./system/host_info/master0.scao3-ocp43.os.fyre.ibm.com.proc.cpuinfo
[must-gather-fmzgt] POD ./system/host_info/master0.scao3-ocp43.os.fyre.ibm.com.proc.meminfo
[must-gather-fmzgt] POD ./system/host_info/master0.scao3-ocp43.os.fyre.ibm.com.df
[must-gather-fmzgt] POD ./system/host_info/master0.scao3-ocp43.os.fyre.ibm.com.ip.address
[must-gather-fmzgt] POD ./system/host_info/master0.scao3-ocp43.os.fyre.ibm.com.dmesg
[must-gather-fmzgt] POD ./system/host_service/
[must-gather-fmzgt] POD ./system/host_service/masters/
[must-gather-fmzgt] POD ./system/host_service/masters/master0.scao3-ocp43.os.fyre.ibm.com.kubelet.log
[must-gather-fmzgt] POD ./system/host_service/masters/master2.scao3-ocp43.os.fyre.ibm.com.kubelet.log
[must-gather-fmzgt] POD ./system/host_service/masters/master1.scao3-ocp43.os.fyre.ibm.com.kubelet.log
[must-gather-fmzgt] POD ./system/host_service/masters/master0.scao3-ocp43.os.fyre.ibm.com.crio.log
[must-gather-fmzgt] POD ./system/host_service/masters/master1.scao3-ocp43.os.fyre.ibm.com.crio.log
[must-gather-fmzgt] POD ./system/host_service/masters/master2.scao3-ocp43.os.fyre.ibm.com.crio.log
[must-gather-fmzgt] POD ./system/network/
[must-gather-fmzgt] POD ./system/network/master0.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/master0.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/master0.scao3-ocp43.os.fyre.ibm.com_ovs-9vskw_vswitchd_log
[must-gather-fmzgt] POD ./system/network/master0.scao3-ocp43.os.fyre.ibm.com_ovs-9vskw_ovsdb_log
[must-gather-fmzgt] POD ./system/network/master1.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/master1.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/master1.scao3-ocp43.os.fyre.ibm.com_ovs-w9qjl_vswitchd_log
[must-gather-fmzgt] POD ./system/network/master1.scao3-ocp43.os.fyre.ibm.com_ovs-w9qjl_ovsdb_log
[must-gather-fmzgt] POD ./system/network/master2.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/master2.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/master2.scao3-ocp43.os.fyre.ibm.com_ovs-tr4x8_vswitchd_log
[must-gather-fmzgt] POD ./system/network/master2.scao3-ocp43.os.fyre.ibm.com_ovs-tr4x8_ovsdb_log
[must-gather-fmzgt] POD ./system/network/worker0.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/worker0.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/worker0.scao3-ocp43.os.fyre.ibm.com_ovs-hxlsk_vswitchd_log
[must-gather-fmzgt] POD ./system/network/worker0.scao3-ocp43.os.fyre.ibm.com_ovs-hxlsk_ovsdb_log
[must-gather-fmzgt] POD ./system/network/worker1.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/worker1.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/worker1.scao3-ocp43.os.fyre.ibm.com_ovs-mwgvq_vswitchd_log
[must-gather-fmzgt] POD ./system/network/worker1.scao3-ocp43.os.fyre.ibm.com_ovs-mwgvq_ovsdb_log
[must-gather-fmzgt] POD ./system/network/worker2.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/worker2.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/worker2.scao3-ocp43.os.fyre.ibm.com_ovs-45bjp_vswitchd_log
[must-gather-fmzgt] POD ./system/network/worker2.scao3-ocp43.os.fyre.ibm.com_ovs-45bjp_ovsdb_log
[must-gather-fmzgt] POD ./system/network/worker3.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/worker3.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/worker3.scao3-ocp43.os.fyre.ibm.com_ovs-mxcjw_vswitchd_log
[must-gather-fmzgt] POD ./system/network/worker3.scao3-ocp43.os.fyre.ibm.com_ovs-mxcjw_ovsdb_log
[must-gather-fmzgt] POD ./system/network/worker4.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/worker4.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/worker4.scao3-ocp43.os.fyre.ibm.com_ovs-tkvkm_vswitchd_log
[must-gather-fmzgt] POD ./system/network/worker4.scao3-ocp43.os.fyre.ibm.com_ovs-tkvkm_ovsdb_log
[must-gather-fmzgt] POD ./system/network/worker5.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/worker5.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/worker5.scao3-ocp43.os.fyre.ibm.com_ovs-dbkck_vswitchd_log
[must-gather-fmzgt] POD ./system/network/worker5.scao3-ocp43.os.fyre.ibm.com_ovs-dbkck_ovsdb_log
[must-gather-fmzgt] POD ./system/network/worker6.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/worker6.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/worker6.scao3-ocp43.os.fyre.ibm.com_ovs-qdgkb_vswitchd_log
[must-gather-fmzgt] POD ./system/network/worker6.scao3-ocp43.os.fyre.ibm.com_ovs-qdgkb_ovsdb_log
[must-gather-fmzgt] POD ./version
[must-gather-fmzgt] POD INFO: cloudpak-must-gather-20201016171535.tar.gz will to copied back to your local host ./must-gather.local.<rand> or what you defined in 'oc adm must-gather --dest-dir'
[must-gather-fmzgt] OUT waiting for gather to complete
[must-gather-fmzgt] OUT downloading gather output
[must-gather-fmzgt] OUT receiving file list ... done
[must-gather-fmzgt] OUT ./
[must-gather-fmzgt] OUT cloudpak-must-gather-20201016171535.tar.gz
[must-gather-fmzgt] OUT
[must-gather-fmzgt] OUT sent 44 bytes received 499036579 bytes 2404995.77 bytes/sec
[must-gather-fmzgt] OUT total size is 498914627 speedup is 1.00
[must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-c2z77 deleted
[must-gather ] OUT namespace/openshift-must-gather-4l77r deleted
In the README, the link to the enhancement is broken.
The directory structure, as well as specific details behind this tool can be found in this enhancement.
Example:
$ bin/openshift-must-gather info clusteroperators
Error: the server doesn't have a resource type "clusteroperators"
is caused because the user's kubeconfig are not set and/or the active context is not authenticated with the cluster.
We should provide a better error and/or things to try (like logging in) if something like this happens.
Most of the times, the version
file (generated by the version function) only shows "4.0-art-dev
" or "Unknown
", which is not very useful.
If it's not possible to get the version of the image used, at least the branch in this repo (and maybe the last commit applied) could be added to the output of the version
function (release-4.6
, release-4.7
, release-4.8
, release-4.9
, release-4.10
, ...).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.