openshift / must-gather Goto Github PK

A client tool for gathering information about an operator managed component.

License: Apache License 2.0

Makefile 0.96% Go 0.14% Shell 98.90%

must-gather's Introduction

must-gather

openshift-must-gather is a tool for collecting cluster data. It dumps clusteroperator data, and associated namespace data, into a specified --base-dir location. The directory structure, as well as specific details behind this tool can be found in this enhancement.

Collection Scripts

Data collection scripts are kept in ./collection-scripts. The content of that folder is placed in /usr/bin in the image. The data collection scripts should only include collection logic for components that are included as part of the OpenShift CVO payload. Outside components are encouraged to produce a similar "must-gather" image, but this is not the spot to be included.

How to run

You can run must-gather on a cluster with oc adm must-gather. Use -h flag to see available options.

Obfuscate confidential information

There is a dedicated to obfuscate and omit confidential information. Head over to openshift/must-gather-clean for more information.

must-gather's People

Contributors

Stargazers

Watchers

Forkers

juanvallejo deads2k derekwaynecarr sferich888 soltysh jameswmills mfojtik tnozicka damemi ironcladlou sanchezl benjaminapetersen csrwng stuartchuan cucushift jwendell donn1e cgwalters ashishranjan738 rcarrillocruz schseba openshift-cherrypick-robot enj sallyom ravisantoshgudimetla cuiyingd abhinavdahiya hexfusion pkliczewski jsafrane mgugino-upstream-stage danwinship ravitri paulfantom navidsh rphillips wking rgupta1234 aravindhp alevitter ashishkumsingh maysamacedo sergioperez pecameron haoqing0110 a7vicky rvanderp3 zshi-redhat nilsanderselde cybertron rdiazgav bostrt palonsoro mauroseb sbuvaneshkumar runcom xiaojiey sttts sebsoto p0lyn0mial davemulford dougsland astoycos openshift-bot sgreene570 rbbratta jupierce ecordell vpickard bcrochet juanluisvaladas global-localhost global19 global19-atlassian-net zerodayz barryzhounb mtulio nanyte25 nci-student-pentesters msherif1234 s1061123 jluhrsen isabella232 andreaskaris oshoval elmiko bpickard22 2uasimojo sthaha nicklesimba npinaeva amorenoz manny27nyc tiraboschi lmzuccarelli rhmilti tjungblu csheremeta devotionzhu fgiudici

must-gather's Issues

RFE: Collect etcd metrics in must-gather

Include the metrics from etcd, as described in https://access.redhat.com/solutions/5141931, in the "gather_etcd" script

elide secret data from must-gather

Hi,

Is there a reason why must-gather includes some secrets by default? This default behavior also collect secrets from namespaces like the following:

(tested with oc cli 4.6.x)
./namespaces/openshift-authentication/core/secrets.yaml
./namespaces/openshift-config/core/secrets.yaml

These secrets.yaml usually contain ldap username and passwords.

I would prefer a toggle where we can explicitly enable a dump of any sensitive data like secrets/configmaps to prevent data leakage.
I know that another tool called must-gather-clear could remove sensitive data afterwards. But why shouldn't we change the default behavior to a minimum and add additional information, if really needed.

Error reporting in a collection script cause "oc adm must-gather" to fail

I created a custom image with a new collection script called gather-cnv (https://github.com/djfjeff/must-gather/blob/gather-cnv-support/collection-scripts/gather-cnv).

Basically, for now, the new collection script only do : openshift-must-gather inspect ns/kubevirt

Manually running the inspect command above fetch the information but still produce some error output :

openshift-must-gather inspect ns/kubevirt

2019/04/26 14:02:40 Gathering data for ns/kubevirt...
2019/04/26 14:02:40     Collecting resources for namespace "kubevirt"...
2019/04/26 14:02:40     Gathering pod data for namespace "kubevirt"...
2019/04/26 14:02:40         Gathering data for pod "virt-api-f8d6cd97-5l68b"
2019/04/26 14:02:41         Unable to gather previous container logs: previous terminated container "virt-api" in pod "virt-api-f8d6cd97-5l68b" not found
2019/04/26 14:02:42         Skipping /version info gathering for pod "virt-api-f8d6cd97-5l68b". Endpoint not found...
2019/04/26 14:02:42         Gathering data for pod "virt-api-f8d6cd97-6h4kx"
2019/04/26 14:02:43         Unable to gather previous container logs: previous terminated container "virt-api" in pod "virt-api-f8d6cd97-6h4kx" not found
2019/04/26 14:02:43         Skipping /version info gathering for pod "virt-api-f8d6cd97-6h4kx". Endpoint not found...
2019/04/26 14:02:43         Gathering data for pod "virt-controller-865b95f6c6-gtplf"
2019/04/26 14:02:43         Unable to gather previous container logs: previous terminated container "virt-controller" in pod "virt-controller-865b95f6c6-gtplf" not found
2019/04/26 14:02:44         Gathering data for pod "virt-controller-865b95f6c6-vl6cc"
2019/04/26 14:02:44         Unable to gather previous container logs: previous terminated container "virt-controller" in pod "virt-controller-865b95f6c6-vl6cc" not found
2019/04/26 14:02:44         Gathering data for pod "virt-handler-5zd2z"
2019/04/26 14:02:44         Unable to gather previous container logs: previous terminated container "virt-handler" in pod "virt-handler-5zd2z" not found
2019/04/26 14:02:44         Gathering data for pod "virt-handler-9q9cp"
2019/04/26 14:02:44         Unable to gather previous container logs: previous terminated container "virt-handler" in pod "virt-handler-9q9cp" not found
2019/04/26 14:02:45         Gathering data for pod "virt-handler-m9q8r"
2019/04/26 14:02:45         Unable to gather previous container logs: previous terminated container "virt-handler" in pod "virt-handler-m9q8r" not found
2019/04/26 14:02:45         Gathering data for pod "virt-handler-t6kz2"
2019/04/26 14:02:45         Unable to gather previous container logs: previous terminated container "virt-handler" in pod "virt-handler-t6kz2" not found
2019/04/26 14:02:45         Gathering data for pod "virt-operator-76b568d986-6x9rc"
2019/04/26 14:02:45         Unable to gather previous container logs: previous terminated container "virt-operator" in pod "virt-operator-76b568d986-6x9rc" not found
Error: one or more errors ocurred while gathering pod-specific data for namespace: kubevirt

    [one or more errors ocurred while gathering container data for pod virt-api-f8d6cd97-5l68b:

    unable to gather container /healthz: unable to find any available /healthz paths hosted in pod "virt-api-f8d6cd97-5l68b", one or more errors ocurred while gathering container data for pod virt-api-f8d6cd97-6h4kx:

    unable to gather container /healthz: unable to find any available /healthz paths hosted in pod "virt-api-f8d6cd97-6h4kx", one or more errors ocurred while gathering container data for pod virt-controller-865b95f6c6-gtplf:

    [unable to gather container /healthz: the server could not find the requested resource, unable to gather container /version: the server could not find the requested resource], one or more errors ocurred while gathering container data for pod virt-controller-865b95f6c6-vl6cc:

    [unable to gather container /healthz: the server could not find the requested resource, unable to gather container /version: the server could not find the requested resource], one or more errors ocurred while gathering container data for pod virt-handler-5zd2z:

    [unable to gather container /healthz: the server could not find the requested resource, unable to gather container /version: the server could not find the requested resource], one or more errors ocurred while gathering container data for pod virt-handler-9q9cp:

    [unable to gather container /healthz: the server could not find the requested resource, unable to gather container /version: the server could not find the requested resource], one or more errors ocurred while gathering container data for pod virt-handler-m9q8r:

    [unable to gather container /healthz: the server could not find the requested resource, unable to gather container /version: the server could not find the requested resource], one or more errors ocurred while gathering container data for pod virt-handler-t6kz2:

    [unable to gather container /healthz: the server could not find the requested resource, unable to gather container /version: the server could not find the requested resource], one or more errors ocurred while gathering container data for pod virt-operator-76b568d986-6x9rc:

    [unable to gather container /healthz: the server could not find the requested resource, unable to gather container /version: the server could not find the requested resource]]

The issue I have is when running it with oc adm must-gather, it seems the error reporting break the gather init container and make the whole process to fail :

oc adm must-gather --image=quay.io/jsaucier/must-gather-cnv:latest -- gather-cnv

namespace/openshift-must-gather-h9m84 created
clusterrolebinding.rbac.authorization.k8s.io/must-gather-q8xwf created
clusterrolebinding.rbac.authorization.k8s.io/must-gather-q8xwf deleted
namespace/openshift-must-gather-h9m84 deleted
error: pod is not running: Failed

To confirm, I try changing the namespace I gather in gather-cnv from ns/kubevirt to ns/default (which does not produce any error reporting) and everything works fine.

EDIT: wrong repo

EDIT: Wrong repo for the PR... can be deleted. (my apologies :)

[RFE] Include the output of oc describe nodes in the must-gather

The oc describe nodes command provides information about the requests/limits, pods in the nodes, allocated resources and events related to each node. Having that information in the must-gather report could help a lot with troubleshooting. If possible, the description of each node in separated files with the name of the node.

Several errors when running must-gather

When running must-gather - the actions fails, specifying the following errors:

$ oc adm must-gather
namespace/openshift-must-gather-pw995 created
clusterrolebinding.rbac.authorization.k8s.io/must-gather-n7jvb created
container logs unavailable: Get https://host-172-16-0-21:10250/containerLogs/openshift-must-gather-pw995/must-gather-2k9gh/gather?follow=true: remote error: tls: internal errorWARNING: cannot use rsync: rsync not available in container
WARNING: cannot use tar: tar not available in container
clusterrolebinding.rbac.authorization.k8s.io/must-gather-n7jvb deleted
namespace/openshift-must-gather-pw995 deleted
error: No available strategies to copy.

Running on a cluster with openshift 4.2 installed.

RFE: per-node pod symlinks

TL;DR: create a folder for each node, symlink in pods running on that node.

Background: When debugging networking issues, we need to look at the logs for the network daemon running on a specific node. Of course, we can extract this from the pod's yaml, but this is somewhat tedious and potentially error-prone.

It would be useful if there was a way for the must-gather to make navigating on a per-node basis easier. It might look something like

/per_node/NODENAME/pod/myns_podname.yaml -> /namespaces/myns/pods/podname/podname.yaml
as well as
/namespaces/myns/pods/per_node/NODENAME/podname.yaml -> /namespaces/myns/pods/podname/podname.yaml

RFE - Create a dir in ceph/must_gather_commands to hold all ceph crash outputs

Currently the must-gather command throws all crash* files into the directory: ./registry-redhat-io-ocs4-ocs-must-gather-rhel8-sha256-b1d22c2497fc432eb06d02f917f2ec6b0c763b16e6147b310de1bae1e8b30eef/ceph/must_gather_commands/

I'm creating this RFE to suggest we create a directory here called 'crashes', that houses all of these crash files. This will tidy up the directory and make it easier to parse when numerous crashes are present.

Gather the same information from OLM managed Operators as from ClusterOperators

Current behaviour

Currently must-gather, using oc adm inspect clusteroperators, get the list of all the ClusterOperator objects and then traverse all their related resources collecting information about pods, logs, etc

must-gather gets also some information about OLM, using oc adm inspect -A olm command, including the list of Operator objects, but it does not collect any other related resource information

Desired behaviour

must-gather could collect the same information from Operator objects as is currently doing for ClusterOperator objects maybe using a new oc adm inspect operators command.

Motivation

Openshift is heavily based on operators and not all of them are part of the vanilla system.
must-gather is a very valuable tool for debugging purposes and helps a lot with troubleshooting but its information about OLM operators is not as good as it could be, making harder to detect and troubleshoot problems with these components even more in some business areas like TelCo, where the number of "non-vanilla" operators is increasing.

`HostSubnet` resources not collected from must-gather

The resources that are missing:

$ oc get HostSubnet -o json | head
{
    "apiVersion": "v1",
    "items": [
        {
            "apiVersion": "network.openshift.io/v1",
            "host": "ip-10-0-131-141.eu-central-1.compute.internal",
            "hostIP": "10.0.131.141",
            "kind": "HostSubnet",
            "metadata": {
                "annotations": {

The content of the folder cluster-scoped-resources/network.openshift.io/ (where they should be):

$ tree must-gather.local.1360424657569460676/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-3c30f115dc95c3fef94ea5185f386aa1af8a4b5f07ce8f41a17007d54004e1c4/cluster-scoped-resources/network.openshift.io/
must-gather.local.1360424657569460676/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-3c30f115dc95c3fef94ea5185f386aa1af8a4b5f07ce8f41a17007d54004e1c4/cluster-scoped-resources/network.openshift.io/
└── clusternetworks
    └── default.yaml

Additional information

$ oc version
Client Version: 4.8.0-202108312109.p0.git.0d10c3f.assembly.stream-0d10c3f
Server Version: 4.9.24
Kubernetes Version: v1.22.5+5c84e52

Include insights-operator archive in must-gather

While insights-operator archive is significantly smaller than must-gather, it manages to collect data relevant for the cluster that are not collected by must-gather.

Having insights-operator as part of the must-gather would expand the data available for troubleshooting, either to be used directly, or by the recommendation engine for the support.

RFE: Include pods and events information from all namespaces

Include pods and events information from all namespaces to the must-gather collection.

oc adm inspect --dest-dir must-gather pods --all-namespaces
oc adm inspect --dest-dir must-gather events --all-namespaces

[RFE] Collect additional information about all pods in the cluster

Collect similar information than the information shown by oc describe nodes, like CPU/memory limits and requests per pod, allocated resources in the nodes, real resource usage by pods (maybe also by containers) in must-gather.

Currently, information like the following is shown in an oc describe nodes output:

[...]
Non-terminated Pods:                         (100 in total)
  Namespace                                  Name                                                          CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                                  ----                                                          ------------  ----------  ---------------  -------------  ---
  axyz                                        daemonset-example-d4lvf                                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         29h
  axyz                                        deployment-example-862f9x79n-z27bg                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         12h
  axyz                                        job-example-mxg74                                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         12h
  axyz                                        replication-example-vhz2v                                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         12h
  axyz                                        replication-example-yxz4c                                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         12h
  axyz                                        replication-example-njcv2                                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         12h
  zyxw                                      backend-worker-1-b2bl4                                        150m (4%)     1 (28%)     50Mi (0%)        300Mi (4%)     7h2m
  zyxw                                      system-sphinx-1-xnvz1                                         80m (2%)      1 (28%)     250Mi (3%)       512Mi (7%)     12h
[...]

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests          Limits
  --------           --------          ------
  cpu                3364m (96%)       9780m (279%)
  memory             7118172800 (99%)  9694Mi (141%)
  ephemeral-storage  0 (0%)            0 (0%)
  hugepages-1Gi      0 (0%)            0 (0%)
  hugepages-2Mi      0 (0%)            0 (0%)

But in a must-gather, not all namespaces/pods are collected.

That information could help to identify overcommitted nodes, pods without requests/limits, etc.

Can't pull ose-must-gather container

Hello,

since today we can't pull openshift4/ose-must-gather on registry.redhat.io

$ podman pull registry.redhat.io/openshift4/ose-must-gather
Trying to pull registry.redhat.io/openshift4/ose-must-gather:latest...
WARN[0001] Failed, retrying in 1s ... (1/3). 
Error: determining manifest MIME type for docker://registry.redhat.io/openshift4/ose-must-gather:latest: reading manifest sha256:2cf7da4892d34247d7135930021739023a85c635c946649252151e0279686abb in registry.redhat.io/openshift4/ose-must-gather: unsupported: Not Found, or unsupported.
V2 schema 1 manifest digest are no longer supported for image pulls.
Use the equivalent schema 2 manifest digest instead.
For more information see https://access.redhat.com/articles/6138332

Do you know what changed ?

Thank you

Future Release Branches Frozen For Merging | branch:release-4.16 branch:release-4.17

The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

release-4.16
release-4.17

For more information, see the branching documentation.

[RFE] Adding the missing clusterresourcequota and Flowcontrol

The Following Resources are missing in the Must-gather:

clusterresourcequotas.quota.openshift.io
prioritylevelconfigurations.flowcontrol.apiserver.k8s.io
flowschemas.flowcontrol.apiserver.k8s.io

That some unusual resources, but a miss configuration may have a big impact on the cluster performances.
Once added to the MG, we should be able to get them added in omc and omg for the support teams to check them.
And so I will be able to included them in my health check script to help review them quickly.

Happy to discuss it, if your think that we should not include them.

Include IPv4 and IPv6 Route Cache Entries in networking must gather

ip [-6] route doesn't provide any information on route cache entries (RCE). When ICMP "need to fragment" is seen in the wild, a RCE is created. Absent knowing an RCE exists, networking debugging wastes time looking in the wrong area.

images produces from Dockerfile have an old version of oc

while experimenting with creating my own must-gather image i tried to make a build from the Dockerfile located in the root of the project. although it built properly, when running it i see an error in that the --dest-dir option is not available. upon further inspection it appears that the oc binary included in the registry.svc.ci.openshift.org/openshift/origin-v4.0:cli image does no support the inspect command, thus the gather scripts will fail.

a snippet of error output:

[must-gather      ] OUT namespace/openshift-must-gather-gc2vz created                                                                                                                          
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-4hkdz created                                                                                                 
[must-gather      ] OUT pod for plug-in image quay.io/fnord/must-gather created                                                                                                         
[must-gather-2qqfs] POD 2021-05-04T00:32:02.896350743Z Error: unknown flag: --dest-dir

how to reproduce

create a must gather image using the default dockerfile, eg podman build -t quay.io/fnord/must-gather
upload image
run a must gather against a cluster using oc must-gather --image quay.io/fnord/must-gather

Publish OCP Image

Follow the policies, guidelines, and standards to release an OCP image of the must-gather tool.

Ensure image DistGit repository exists
- Request an new build repository.
- Wait for new issue to appear in the Release Engineering (RCM) JIRA
- RCM JIRA task completed.
Create task in ART JIRA to have the image build managed by the automated release team.
Add image to release payload
Add must-gather ImageStream to openshift-samples-operator.

must-gather script is always pulling back logs from every namespace

When we specify the target namespaces to collect the logs, we are always pulling back all the logs. We have tried both -n and --namespaces.

oc adm must-gather --image=quay.io/opencloudio/must-gather:4.5.1 -- gather -n cem

oc describe po must-gather-fmzgt
Name:         must-gather-fmzgt
Namespace:    openshift-must-gather-4l77r
Priority:     0
Node:         master1.scao3-ocp43.os.fyre.ibm.com/10.16.39.94
Start Time:   Fri, 16 Oct 2020 11:55:40 -0500
Labels:       app=must-gather
Annotations:  k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "openshift-sdn",
                    "interface": "eth0",
                    "ips": [
                        "10.254.20.68"
                    ],
                    "dns": {},
                    "default-route": [
                        "10.254.20.1"
                    ]
                }]
Status:       Pending
IP:           10.254.20.68
IPs:
  IP:  10.254.20.68
Init Containers:
  gather:
    Container ID:  cri-o://fb9cd71ffd3071eac612b31d41f16edb34e0258289a50e1c262a2a1ed224c4d1
    Image:         quay.io/opencloudio/must-gather:4.5.1
    Image ID:      quay.io/opencloudio/must-gather@sha256:457b43e57fc6df8456167bfde409e45351d6cc9420fa2d1eea9eb2e4567923ff
    Port:          <none>
    Host Port:     <none>
    Command:
      gather
      -n
      cem
    State:          Running
      Started:      Fri, 16 Oct 2020 11:56:02 -0500
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /must-gather from must-gather-output (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xslj4 (ro)
Containers:
  copy:
    Container ID:  
    Image:         quay.io/opencloudio/must-gather:4.5.1
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
      -c
      trap : TERM INT; sleep infinity & wait
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /must-gather from must-gather-output (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xslj4 (ro)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  must-gather-output:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  default-token-xslj4:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-xslj4
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     
Events:
  Type    Reason     Age        From                                          Message
  ----    ------     ----       ----                                          -------
  Normal  Scheduled  <unknown>  default-scheduler                             Successfully assigned openshift-must-gather-4l77r/must-gather-fmzgt to master1.scao3-ocp43.os.fyre.ibm.com
  Normal  Pulling    8m4s       kubelet, master1.scao3-ocp43.os.fyre.ibm.com  Pulling image "quay.io/opencloudio/must-gather:4.5.1"
  Normal  Pulled     7m50s      kubelet, master1.scao3-ocp43.os.fyre.ibm.com  Successfully pulled image "quay.io/opencloudio/must-gather:4.5.1"
  Normal  Created    7m50s      kubelet, master1.scao3-ocp43.os.fyre.ibm.com  Created container gather
  Normal  Started    7m50s      kubelet, master1.scao3-ocp43.os.fyre.ibm.com  Started container gather

Output:

oc adm must-gather --image=quay.io/opencloudio/must-gather:4.5.1 -- gather -n cem
[must-gather      ] OUT Using must-gather plugin-in image: quay.io/opencloudio/must-gather:4.5.1
[must-gather      ] OUT namespace/openshift-must-gather-4l77r created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-c2z77 created
[must-gather      ] OUT pod for plug-in image quay.io/opencloudio/must-gather:4.5.1 created
[must-gather-fmzgt] POD configuration:
[must-gather-fmzgt] POD modules overview,system,failure,ocp,cloudpak
[must-gather-fmzgt] POD namespaces cem
[must-gather-fmzgt] POD labels 
[must-gather-fmzgt] POD extensions 
[must-gather-fmzgt] POD env 
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD INFO: gather_overview collecting ...
[must-gather-fmzgt] POD # oc get nodes --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get pods --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get deploy --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get sts --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get ds --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get job --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get cronjob --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get configmap --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get secret --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get resourcequota --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get hpa --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get svc --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get endpoints --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get ingress --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get route --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get networkpolicy --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get pv --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get pvc --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get storageclass --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get namespace --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get event --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get crd --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get csv --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get certificates.certmanager.k8s.io --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get challenges.certmanager.k8s.io --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get clusterissuers.certmanager.k8s.io --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get issuers.certmanager.k8s.io --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD # oc get orders.certmanager.k8s.io --all-namespaces -owide --show-labels
[must-gather-fmzgt] POD INFO: gather_system collecting ...
[must-gather-fmzgt] POD daemonset.apps/gather-system-ds created
[must-gather-fmzgt] POD INFO: daemonset openshift-must-gather-4l77r/gather-system-ds not ready
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD INFO: daemonset openshift-must-gather-4l77r/gather-system-ds not ready
[must-gather-fmzgt] POD INFO: daemonset openshift-must-gather-4l77r/gather-system-ds not ready
[must-gather-fmzgt] POD INFO: daemonset openshift-must-gather-4l77r/gather-system-ds not ready
[must-gather-fmzgt] POD INFO: daemonset openshift-must-gather-4l77r/gather-system-ds not ready
[must-gather-fmzgt] POD INFO: daemonset openshift-must-gather-4l77r/gather-system-ds not ready
[must-gather-fmzgt] POD INFO: daemonset openshift-must-gather-4l77r/gather-system-ds not ready
[must-gather-fmzgt] POD INFO: daemonset openshift-must-gather-4l77r/gather-system-ds is ready
[must-gather-fmzgt] POD INFO: gather_system on node master2.scao3-ocp43.os.fyre.ibm.com pod gather-system-ds-dn5n8
[must-gather-fmzgt] POD INFO: gather_system on node master1.scao3-ocp43.os.fyre.ibm.com pod gather-system-ds-nchfs
[must-gather-fmzgt] POD INFO: gather_system on node master0.scao3-ocp43.os.fyre.ibm.com pod gather-system-ds-vjnmw
[must-gather-fmzgt] POD INFO: Waiting for gather_system log collection to complete ...
[must-gather-fmzgt] POD INFO: gather_system log collection complete.
[must-gather-fmzgt] POD daemonset.apps "gather-system-ds" deleted
[must-gather-fmzgt] POD WARNING: Collecting one or more service logs on ALL master nodes in your cluster. This could take a large amount of time.
[must-gather-fmzgt] POD INFO: Collecting host service logs for kubelet
[must-gather-fmzgt] POD INFO: Collecting host service logs for crio
[must-gather-fmzgt] POD INFO: Waiting for worker host service log collection to complete ...
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD INFO: Worker host service log collection complete.
[must-gather-fmzgt] POD WARNING: Collecting network logs on ALL nodes in your cluster. This could take a long time.
[must-gather-fmzgt] POD I1016 16:57:52.513486    1572 request.go:621] Throttling request took 1.127477938s, request: GET:https://172.30.0.1:443/apis/gke.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Error from server: error dialing backend: dial tcp 10.16.39.171:10250: connect: connection refused
[must-gather-fmzgt] POD Error from server: error dialing backend: dial tcp 10.16.39.171:10250: connect: connection refused
[must-gather-fmzgt] POD Error from server: error dialing backend: dial tcp 10.16.39.171:10250: connect: connection refused
[must-gather-fmzgt] POD Error from server: error dialing backend: dial tcp 10.16.39.171:10250: connect: connection refused
[must-gather-fmzgt] POD INFO: Waiting for node network log collection to complete ...
[must-gather-fmzgt] POD INFO: Node network log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_pods collecting ...
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-cassandra-0
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-datalayer-0
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-datalayer-cron-1602724500-tk6c5
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-datalayer-cron-1602810900-596n6
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-datalayer-cron-1602810900-6nr5m
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-datalayer-cron-1602810900-d7cpl
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-datalayer-cron-1602810900-q9xjd
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-datalayer-cron-1602810900-rjzng
[must-gather-fmzgt] POD INFO: gather_failure_pods on cem/peter-ibm-cem-datalayer-cron-1602810900-zvcv4
[must-gather-fmzgt] POD INFO: gather_failure_pods on kube-system/multicluster-hub-console-mcmuiapi-bfbd55ddc-q7bhn
[must-gather-fmzgt] POD INFO: gather_failure_pods on kube-system/multicluster-hub-core-apiserver-7f6645f546-gwcbp
[must-gather-fmzgt] POD INFO: gather_failure_pods on kube-system/multicluster-hub-grafeas-64f6c9c7c-xkq2f
[must-gather-fmzgt] POD INFO: gather_failure_pods on kube-system/multicluster-hub-grcuiapi-75c4fcd6bb-gsqm7
[must-gather-fmzgt] POD INFO: gather_failure_pods on kube-system/multicluster-hub-legato-5bbc7f776b-f4tnt
[must-gather-fmzgt] POD INFO: gather_failure_pods on kube-system/multicluster-hub-topologyapi-5db998858f-f62wt
[must-gather-fmzgt] POD INFO: gather_failure_pods on kube-system/search-search-api-7b4c978599-hbjxg
[must-gather-fmzgt] POD INFO: gather_failure_pods on openshift-console/downloads-5fdf68d856-rzcml
[must-gather-fmzgt] POD INFO: gather_failure_pods on openshift-kube-apiserver/installer-67-master2.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_failure_pods on openshift-kube-apiserver/revision-pruner-66-master1.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_failure_pods on rook-ceph/rook-ceph-osd-6-58bd78b9b7-d2qsp
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_pods collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_pods log collection complete.
[must-gather-fmzgt] POD INFO: gather_restart_pods on cem/peter-ibm-cem-datalayer-0
[must-gather-fmzgt] POD INFO: gather_restart_pods on cem/peter-ibm-cem-rba-as-566bcffb6-tnzqp
[must-gather-fmzgt] POD INFO: gather_restart_pods on cem/peter-ibm-cem-rba-rbs-689475646d-png8d
[must-gather-fmzgt] POD INFO: gather_restart_pods on cert-manager/cert-manager-ibm-cert-manager-5cd9fb9777-cks5p
[must-gather-fmzgt] POD INFO: gather_restart_pods on cert-manager/cert-manager-webhook-cainjector-f46ff7b74-p442r
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/auth-idp-bckx9
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/auth-idp-z2bwn
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/auth-idp-zkg7d
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/auth-pdp-ff9rk
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/auth-pdp-grgp4
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/auth-pdp-ln9rd
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/catalog-ui-4ndvq
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/catalog-ui-gw6tk
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/catalog-ui-jl2lz
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/common-web-ui-wpwtq
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/common-web-ui-zk2l6
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/common-web-ui-zw46s
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/icp-mongodb-0
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/icp-mongodb-1
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/internal-management-ingress-9gwsh
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/internal-management-ingress-scgxp
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/metering-dm-c7b75c686-n2b7h
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/metering-reader-7522r
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/metering-reader-7bwdl
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/metering-reader-8x48b
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/metering-reader-bhpnk
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/metering-reader-l5wsj
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/metering-reader-ppzhz
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/metering-reader-qsj6z
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/monitoring-grafana-67ddcdf99d-k7df4
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/monitoring-prometheus-nodeexporter-2cd57
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/multicluster-hub-cluster-api-provider-aks-b84b6657-pqhgx
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/multicluster-hub-cluster-api-provider-eks-c8b6c956f-bmrlz
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/multicluster-hub-cluster-api-provider-gke-547c5c7b59-kpprf
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/multicluster-hub-cluster-api-provider-iks-7c4999db75-fm62g
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/multicluster-hub-cluster-api-provider-ocp-6d5fbd8459-ckg46
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/multicluster-hub-cluster-api-provider-sdk-b6979fffb-fm89p
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/multicluster-hub-core-apiserver-7f6645f546-gwcbp
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/multicluster-hub-core-channel-59fdf84bf4-5sjm4
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/nginx-ingress-controller-czjkn
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/nginx-ingress-controller-h95x9
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/nginx-ingress-controller-smppk
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/platform-api-794c95849c-jhjpn
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/platform-api-794c95849c-wz89k
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/platform-header-49gzc
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/platform-header-s8tp2
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/platform-header-xhp4k
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/prometheus-monitoring-prometheus-0
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/search-search-aggregator-64b75d9466-lj9ms
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/search-search-collector-6c68bd89bd-rhmt2
[must-gather-fmzgt] POD INFO: gather_restart_pods on kube-system/secret-watcher-779df94cc8-z7g8c
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-apiserver-operator/openshift-apiserver-operator-7cc77d965f-x8pff
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-apiserver/apiserver-8rpk8
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-apiserver/apiserver-j5gsv
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-apiserver/apiserver-lm8sl
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-authentication-operator/authentication-operator-57d4974d5d-6tk6f
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-cloud-credential-operator/cloud-credential-operator-dd496454-57rdg
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-cluster-version/cluster-version-operator-584fddff45-tgdkd
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-console-operator/console-operator-57f5bcc578-6q5vj
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-controller-manager-operator/openshift-controller-manager-operator-68dcf95c47-6bz2w
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-controller-manager/controller-manager-2zczq
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-controller-manager/controller-manager-6r22f
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-85jts
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-c9czj
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-f87fn
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-klp8r
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-m9p5v
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-mnfxh
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-p62vs
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-qbs6q
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-vvw7j
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-dns/dns-default-wxv5g
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-etcd/etcd-member-master0.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-etcd/etcd-member-master1.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-etcd/etcd-member-master2.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-55mpk
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-8gljk
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-fwp52
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-h4p5w
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-hrsqc
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-j9gfj
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-ktrnn
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-mnt9j
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-wpzqh
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-image-registry/node-ca-xmr4j
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-apiserver-operator/kube-apiserver-operator-566b9798-7rjbl
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-apiserver/kube-apiserver-master0.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-apiserver/kube-apiserver-master1.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-apiserver/kube-apiserver-master2.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-controller-manager-operator/kube-controller-manager-operator-7c8b7465b-gd8nf
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-controller-manager/kube-controller-manager-master0.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-controller-manager/kube-controller-manager-master2.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-scheduler-operator/openshift-kube-scheduler-operator-557777c86b-2c2rk
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-scheduler/openshift-kube-scheduler-master0.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-kube-scheduler/openshift-kube-scheduler-master2.scao3-ocp43.os.fyre.ibm.com
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-api/cluster-autoscaler-operator-ccbfbd76b-brfvj
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-api/machine-api-operator-566464fc6-ktj26
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-controller-79c987cbbf-2kj5c
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-6f5kc
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-cq68r
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-jvt64
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-m8ccg
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-q8rn6
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-tqqkz
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-vq7f4
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-x2n6c
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-xbqqd
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-daemon-xpkk6
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-operator-bd8dd7bfb-d48cr
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-server-6466d
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-server-9rsbz
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-machine-config-operator/machine-config-server-z9jl7
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-2qldx
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-7ltcf
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-jdjlp
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-l2vdm
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-ldv57
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-p8rq2
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-rz2lv
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-wx6fv
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-xr2wg
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/node-exporter-zq6nv
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/prometheus-k8s-1
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-monitoring/thanos-querier-6c57cc5d4-pvw5h
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-2j8n4
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-4cqbg
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-7fndz
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-admission-controller-4brrc
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-admission-controller-9qlgq
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-admission-controller-sqbjw
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-fbggj
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-fpth8
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-j8lsf
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-jnvdn
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-pw5bt
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-q8vlm
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-multus/multus-zr88j
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-45bjp
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-9vskw
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-dbkck
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-hxlsk
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-mwgvq
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-mxcjw
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-qdgkb
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-tkvkm
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-tr4x8
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/ovs-w9qjl
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-7fc54
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-controller-q9rjf
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-controller-qpvqf
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-controller-sfvcj
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-gmxls
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-hlqtr
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-hqvnt
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-jtkd4
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-kvs6j
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-m8f4s
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-sz5pc
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-wdnbf
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-sdn/sdn-zpm46
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-ca-operator/service-ca-operator-595657f77-jhx8b
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-ca/apiservice-cabundle-injector-d84c98485-ll48j
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-ca/configmap-cabundle-injector-6cc5ccdd7f-j5jpq
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-ca/service-serving-cert-signer-d59b877-jfzhp
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-catalog-apiserver-operator/openshift-service-catalog-apiserver-operator-6cddfd76cc-jp659
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-catalog-apiserver/apiserver-8b7bb
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-catalog-apiserver/apiserver-sffhz
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-catalog-apiserver/apiserver-tvl26
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-catalog-controller-manager-operator/openshift-service-catalog-controller-manager-operator-5886grk78
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-catalog-controller-manager/controller-manager-589b2
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-catalog-controller-manager/controller-manager-rh6rb
[must-gather-fmzgt] POD INFO: gather_restart_pods on openshift-service-catalog-controller-manager/controller-manager-vtkrn
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-cephfsplugin-c29kl
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-cephfsplugin-hct5z
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-cephfsplugin-hhqhx
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-cephfsplugin-hvgft
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-cephfsplugin-knt9q
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-cephfsplugin-lmhhv
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-cephfsplugin-mgrkw
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-cephfsplugin-provisioner-75c965db4f-6x59q
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-rbdplugin-8vgks
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-rbdplugin-btpzj
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-rbdplugin-fp89d
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-rbdplugin-m57x2
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-rbdplugin-provisioner-56cbc4d585-7kjjh
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-rbdplugin-tmg4w
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-rbdplugin-tndvm
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/csi-rbdplugin-xjd2t
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/rook-ceph-operator-6f68b5bf75-8c72r
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/rook-discover-549sm
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/rook-discover-b4dtn
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/rook-discover-fp527
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/rook-discover-hntj7
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/rook-discover-q5d75
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/rook-discover-w42vq
[must-gather-fmzgt] POD INFO: gather_restart_pods on rook-ceph/rook-discover-ztr9k
[must-gather-fmzgt] POD INFO: Waiting for gather_restart_pods collection to complete ...
[must-gather-fmzgt] POD INFO: gather_restart_pods log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_deployments collecting ...
[must-gather-fmzgt] POD INFO: gather_failure_deployments on kube-system/multicluster-hub-core-apiserver
[must-gather-fmzgt] POD INFO: gather_failure_deployments on rook-ceph/rook-ceph-osd-6
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_deployments collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_deployments log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_daemonsets collecting ...
[must-gather-fmzgt] POD INFO: gather_failure_daemonsets on openshift-service-catalog-apiserver/apiserver
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_daemonsets collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_daemonsets log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_statefulset  collecting ...
[must-gather-fmzgt] POD INFO: gather_failure_statefulsets on cem/peter-cassandra
[must-gather-fmzgt] POD INFO: gather_failure_statefulsets on cem/peter-ibm-cem-datalayer
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_statefulsets collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_statefulsets log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_jobs collecting ...
[must-gather-fmzgt] POD INFO: gather_failure_jobs on cem/peter-ibm-cem-datalayer-cron-1602810900
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_jobs collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_jobs log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_routes collecting ...
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_routes collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_routes log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_services collecting ...
[must-gather-fmzgt] POD INFO: gather_failure_services on cem/peter-cassandra
[must-gather-fmzgt] POD INFO: gather_failure_services on cem/peter-ibm-cem-datalayer
[must-gather-fmzgt] POD INFO: gather_failure_services on cem/peter-ibm-redis-announce-3
[must-gather-fmzgt] POD INFO: gather_failure_services on cem/peter-ibm-redis-announce-4
[must-gather-fmzgt] POD INFO: gather_failure_services on cem/peter-ibm-redis-announce-5
[must-gather-fmzgt] POD INFO: gather_failure_services on kube-system/deployable-controller-metrics
[must-gather-fmzgt] POD INFO: gather_failure_services on kube-system/icp-oidcclient-watcher
[must-gather-fmzgt] POD INFO: gather_failure_services on kube-system/multicluster-hub-etcd
[must-gather-fmzgt] POD INFO: gather_failure_services on kube-system/subscription-controller-metrics
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_services collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_services log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_pv collecting ...
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_pv collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_pv log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_pvc collecting ...
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_pvc collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_pvc log collection complete.
[must-gather-fmzgt] POD INFO: gather_failure_namespaces collecting ...
[must-gather-fmzgt] POD INFO: gather_failure_namespaces on bodjruj-dyn-0/Terminating
[must-gather-fmzgt] POD INFO: gather_failure_namespaces on eydufza-dyn-0/Terminating
[must-gather-fmzgt] POD INFO: gather_failure_namespaces on ezqcamn-dyn-0/Terminating
[must-gather-fmzgt] POD INFO: gather_failure_namespaces on nsgbzoh-dyn-0/Terminating
[must-gather-fmzgt] POD INFO: gather_failure_namespaces on rtzlpkb-dyn-0/Terminating
[must-gather-fmzgt] POD INFO: gather_failure_namespaces on undmcps-dyn-0/Terminating
[must-gather-fmzgt] POD INFO: Waiting for gather_failure_namespaces collection to complete ...
[must-gather-fmzgt] POD INFO: gather_failure_namespaces log collection complete.
[must-gather-fmzgt] POD INFO: Waiting for gather_ocp collection to complete ...
[must-gather-fmzgt] POD Gathering data for ns/openshift-etcd...
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Gathering data for ns/openshift...
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Gathering data for ns/openshift-cluster-version...
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Gathering data for ns/default...
[must-gather-fmzgt] POD Gathering data for ns/kube-system...
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD I1016 16:58:55.386470   18923 request.go:621] Throttling request took 1.144576304s, request: GET:https://172.30.0.1:443/apis/config.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD I1016 16:58:55.476994   18930 request.go:621] Throttling request took 1.160946705s, request: GET:https://172.30.0.1:443/apis/policies.ibm.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD I1016 16:58:56.402535   18917 request.go:621] Throttling request took 1.062235725s, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD I1016 16:58:56.652810   18916 request.go:621] Throttling request took 1.145479867s, request: GET:https://172.30.0.1:443/apis/iks.mcm.ibm.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD I1016 16:58:57.573833   18910 request.go:621] Throttling request took 1.181404974s, request: GET:https://172.30.0.1:443/apis/aws.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD I1016 16:58:58.204710   18911 request.go:621] Throttling request took 1.113340658s, request: GET:https://172.30.0.1:443/apis/coordination.k8s.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD I1016 16:59:03.407018   18919 request.go:621] Throttling request took 1.139152323s, request: GET:https://172.30.0.1:443/apis/route.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD I1016 16:59:08.270827   18911 request.go:621] Throttling request took 1.548027309s, request: GET:https://172.30.0.1:443/apis/objectbucket.io/v1alpha1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-config...
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Gathering data for ns/openshift-config-managed...
[must-gather-fmzgt] POD I1016 16:59:20.884750   18911 request.go:621] Throttling request took 1.148933929s, request: GET:https://172.30.0.1:443/apis/oidc.security.ibm.com/v1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD Gathering data for ns/openshift-authentication...
[must-gather-fmzgt] POD I1016 16:59:31.832831   18911 request.go:621] Throttling request took 1.142908918s, request: GET:https://172.30.0.1:443/apis/operators.coreos.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-authentication-operator...
[must-gather-fmzgt] POD I1016 16:59:42.377686   18911 request.go:621] Throttling request took 1.156446345s, request: GET:https://172.30.0.1:443/apis/multicloud.ibm.com/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-ingress...
[must-gather-fmzgt] POD I1016 16:59:53.225686   18911 request.go:621] Throttling request took 1.138071941s, request: GET:https://172.30.0.1:443/apis/oauth.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-cloud-credential-operator...
[must-gather-fmzgt] POD I1016 17:00:06.184106   18911 request.go:621] Throttling request took 1.179412318s, request: GET:https://172.30.0.1:443/apis/webhook.certmanager.k8s.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:00:16.248806   18911 request.go:621] Throttling request took 1.127400782s, request: GET:https://172.30.0.1:443/apis/securityenforcement.admission.cloud.ibm.com/v1beta1?timeout=32s
[must-gather-fmzgt] POD I1016 17:00:26.315509   18911 request.go:621] Throttling request took 1.546428134s, request: GET:https://172.30.0.1:443/apis/autoscaling.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:00:36.390391   18911 request.go:621] Throttling request took 1.935471806s, request: GET:https://172.30.0.1:443/apis/iks.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD I1016 17:00:47.720438   18911 request.go:621] Throttling request took 1.152827244s, request: GET:https://172.30.0.1:443/apis/monitoring.coreos.com/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-machine-api...
[must-gather-fmzgt] POD I1016 17:00:58.352701   18911 request.go:621] Throttling request took 1.145100849s, request: GET:https://172.30.0.1:443/apis/tuned.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:01:08.373605   18911 request.go:621] Throttling request took 1.128955507s, request: GET:https://172.30.0.1:443/apis/certmanager.k8s.io/v1alpha1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-console-operator...
[must-gather-fmzgt] POD I1016 17:01:18.401243   18911 request.go:621] Throttling request took 1.149473075s, request: GET:https://172.30.0.1:443/apis/image.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-console...
[must-gather-fmzgt] POD I1016 17:01:28.003862   18918 request.go:621] Throttling request took 1.110309274s, request: GET:https://172.30.0.1:443/apis/ingress.operator.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:01:36.384207   18911 request.go:621] Throttling request took 1.174982991s, request: GET:https://172.30.0.1:443/apis/foundation.ibm.com/v1?timeout=32s
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD error: errors ocurred while gathering data:
[must-gather-fmzgt] POD     one or more errors ocurred while gathering pod-specific data for namespace: kube-system
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [one or more errors ocurred while gathering container data for pod metering-reader-qsj6z:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/kube-system/metering-reader-qsj6z/metering-reader?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/metering-reader-qsj6z/metering-reader?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/metering-reader-qsj6z/metering-reader-secret-check?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/metering-reader-qsj6z/metering-reader-secret-check?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/metering-reader-qsj6z/metering-reader-init?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/metering-reader-qsj6z/metering-reader-init?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod monitoring-prometheus-nodeexporter-z9twl:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/kube-system/monitoring-prometheus-nodeexporter-z9twl/nodeexporter?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/monitoring-prometheus-nodeexporter-z9twl/nodeexporter?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/monitoring-prometheus-nodeexporter-z9twl/router?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/monitoring-prometheus-nodeexporter-z9twl/router?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod multicluster-hub-console-mcmuiapi-bfbd55ddc-q7bhn:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-console-mcmuiapi-bfbd55ddc-q7bhn/hcm-ui-api?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-console-mcmuiapi-bfbd55ddc-q7bhn/hcm-ui-api?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod multicluster-hub-etcd-0:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-etcd-0/etcd?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-etcd-0/etcd?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod multicluster-hub-grafeas-64f6c9c7c-xkq2f:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-grafeas-64f6c9c7c-xkq2f/grafeas?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-grafeas-64f6c9c7c-xkq2f/grafeas?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod multicluster-hub-grcuiapi-75c4fcd6bb-gsqm7:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-grcuiapi-75c4fcd6bb-gsqm7/grc-ui-api?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-grcuiapi-75c4fcd6bb-gsqm7/grc-ui-api?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod multicluster-hub-legato-5bbc7f776b-f4tnt:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-legato-5bbc7f776b-f4tnt/legato?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-legato-5bbc7f776b-f4tnt/legato?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod multicluster-hub-topologyapi-5db998858f-f62wt:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-topologyapi-5db998858f-f62wt/topology-ui-api?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/multicluster-hub-topologyapi-5db998858f-f62wt/topology-ui-api?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod search-search-api-7b4c978599-hbjxg:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/kube-system/search-search-api-7b4c978599-hbjxg/search-api?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/kube-system/search-search-api-7b4c978599-hbjxg/search-api?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused]]
[must-gather-fmzgt] POD Gathering data for ns/openshift-dns-operator...
[must-gather-fmzgt] POD I1016 17:01:46.977748   18911 request.go:621] Throttling request took 1.119737143s, request: GET:https://172.30.0.1:443/apis/operators.coreos.com/v2?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD Gathering data for ns/openshift-dns...
[must-gather-fmzgt] POD I1016 17:02:04.151455   18911 request.go:621] Throttling request took 1.172491829s, request: GET:https://172.30.0.1:443/apis/autoscaling/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:02:14.281006   18911 request.go:621] Throttling request took 1.344193227s, request: GET:https://172.30.0.1:443/apis/storage.k8s.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-image-registry...
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:02:46.102991   18911 request.go:621] Throttling request took 1.169038689s, request: GET:https://172.30.0.1:443/apis/snapshot.storage.k8s.io/v1alpha1?timeout=32s
[must-gather-fmzgt] POD I1016 17:02:56.183159   18911 request.go:621] Throttling request took 1.137581381s, request: GET:https://172.30.0.1:443/apis/iks.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-ingress-operator...
[must-gather-fmzgt] POD I1016 17:03:06.332647   18911 request.go:621] Throttling request took 2.944012486s, request: GET:https://172.30.0.1:443/apis/eks.mcm.ibm.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD I1016 17:03:16.926684   18911 request.go:621] Throttling request took 1.14130196s, request: GET:https://172.30.0.1:443/apis/certmanager.k8s.io/v1alpha1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-insights...
[must-gather-fmzgt] POD I1016 17:03:27.001777   18911 request.go:621] Throttling request took 4.550486787s, request: GET:https://172.30.0.1:443/apis/policies.ibm.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD Gathering data for ns/openshift-kube-apiserver-operator...
[must-gather-fmzgt] POD I1016 17:03:37.132674   18911 request.go:621] Throttling request took 1.560671075s, request: GET:https://172.30.0.1:443/apis/metal3.io/v1alpha1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-kube-apiserver...
[must-gather-fmzgt] POD I1016 17:04:12.480810   18911 request.go:621] Throttling request took 1.136436021s, request: GET:https://172.30.0.1:443/apis/tuned.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:04:22.745906   18911 request.go:621] Throttling request took 1.133177531s, request: GET:https://172.30.0.1:443/apis/iks.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-kube-controller-manager...
[must-gather-fmzgt] POD I1016 17:04:53.124659   18911 request.go:621] Throttling request took 1.163356671s, request: GET:https://172.30.0.1:443/apis/gke.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-kube-controller-manager-operator...
[must-gather-fmzgt] POD I1016 17:05:03.625858   18911 request.go:621] Throttling request took 1.18155294s, request: GET:https://172.30.0.1:443/apis/samples.operator.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:05:13.675841   18911 request.go:621] Throttling request took 1.127826513s, request: GET:https://172.30.0.1:443/apis/foundation.ibm.com/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-kube-scheduler...
[must-gather-fmzgt] POD I1016 17:05:27.440991   18911 request.go:621] Throttling request took 1.153486096s, request: GET:https://172.30.0.1:443/apis/iks.mcm.ibm.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-kube-scheduler-operator...
[must-gather-fmzgt] POD I1016 17:05:37.831243   18911 request.go:621] Throttling request took 1.188463099s, request: GET:https://172.30.0.1:443/apis/operators.coreos.com/v2?timeout=32s
[must-gather-fmzgt] POD I1016 17:05:47.956329   18911 request.go:621] Throttling request took 1.344872495s, request: GET:https://172.30.0.1:443/apis/apiextensions.k8s.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD I1016 17:05:58.076203   18911 request.go:621] Throttling request took 1.731199689s, request: GET:https://172.30.0.1:443/apis/apiextensions.k8s.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-machine-config-operator...
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:06:10.969314   18911 request.go:621] Throttling request took 1.178139106s, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1alpha1?timeout=32s
[must-gather-fmzgt] POD I1016 17:06:21.016823   18911 request.go:621] Throttling request took 1.133973555s, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-marketplace...
[must-gather-fmzgt] POD I1016 17:06:31.341586   18911 request.go:621] Throttling request took 1.158852121s, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:06:41.478383   18911 request.go:621] Throttling request took 1.145756505s, request: GET:https://172.30.0.1:443/apis/batch/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-monitoring...
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:07:28.136788   18911 request.go:621] Throttling request took 1.18467835s, request: GET:https://172.30.0.1:443/apis/webhook.certmanager.k8s.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD I1016 17:07:39.049129   18911 request.go:621] Throttling request took 1.140491444s, request: GET:https://172.30.0.1:443/apis/policy.mcm.ibm.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:07:49.167384   18911 request.go:621] Throttling request took 1.742919383s, request: GET:https://172.30.0.1:443/apis/iks.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-multus...
[must-gather-fmzgt] POD I1016 17:08:09.954449   18911 request.go:621] Throttling request took 1.159932923s, request: GET:https://172.30.0.1:443/apis/helm.bitnami.com/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:08:19.962412   18911 request.go:621] Throttling request took 1.13352285s, request: GET:https://172.30.0.1:443/apis/ceph.rook.io/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:08:30.006264   18911 request.go:621] Throttling request took 1.344666194s, request: GET:https://172.30.0.1:443/apis/gke.mcm.ibm.com/v1beta1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD Gathering data for ns/openshift-sdn...
[must-gather-fmzgt] POD I1016 17:09:03.525301   18911 request.go:621] Throttling request took 1.114752014s, request: GET:https://172.30.0.1:443/apis/oidc.security.ibm.com/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:09:13.594780   18911 request.go:621] Throttling request took 1.138378347s, request: GET:https://172.30.0.1:443/apis/rook.io/v1alpha2?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:09:23.686777   18911 request.go:621] Throttling request took 1.544575107s, request: GET:https://172.30.0.1:443/apis/networking.k8s.io/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:09:33.875988   18911 request.go:621] Throttling request took 1.928791511s, request: GET:https://172.30.0.1:443/apis/user.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-network-operator...
[must-gather-fmzgt] POD I1016 17:09:51.800149   18911 request.go:621] Throttling request took 1.14678053s, request: GET:https://172.30.0.1:443/apis/aks.mcm.ibm.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-cluster-node-tuning-operator...
[must-gather-fmzgt] POD I1016 17:10:06.321358   18911 request.go:621] Throttling request took 1.18428208s, request: GET:https://172.30.0.1:443/apis/k8s.cni.cncf.io/v1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:10:16.562718   18911 request.go:621] Throttling request took 1.136374818s, request: GET:https://172.30.0.1:443/apis/gke.mcm.ibm.com/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-apiserver-operator...
[must-gather-fmzgt] POD I1016 17:10:26.841488   18911 request.go:621] Throttling request took 1.125865257s, request: GET:https://172.30.0.1:443/apis/imageregistry.operator.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-apiserver...
[must-gather-fmzgt] POD I1016 17:10:47.235972   18911 request.go:621] Throttling request took 1.170955106s, request: GET:https://172.30.0.1:443/apis/monitoringcontroller.cloud.ibm.com/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:10:57.244373   18911 request.go:621] Throttling request took 1.143384789s, request: GET:https://172.30.0.1:443/apis/quota.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:11:07.316725   18911 request.go:621] Throttling request took 1.545489137s, request: GET:https://172.30.0.1:443/apis/project.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:11:17.373250   18911 request.go:621] Throttling request took 1.927877252s, request: GET:https://172.30.0.1:443/apis/metrics.k8s.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-controller-manager-operator...
[must-gather-fmzgt] POD I1016 17:11:28.658040   18911 request.go:621] Throttling request took 1.163390578s, request: GET:https://172.30.0.1:443/apis/gke.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-controller-manager...
[must-gather-fmzgt] POD I1016 17:11:39.102765   18911 request.go:621] Throttling request took 1.17956634s, request: GET:https://172.30.0.1:443/apis/gke.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD I1016 17:11:49.242823   18911 request.go:621] Throttling request took 1.349993556s, request: GET:https://172.30.0.1:443/apis/rook.io/v1alpha2?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-cluster-samples-operator...
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:11:59.396366   18911 request.go:621] Throttling request took 4.985744602s, request: GET:https://172.30.0.1:443/apis/console.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift...
[must-gather-fmzgt] POD I1016 17:12:09.587400   18911 request.go:621] Throttling request took 4.763744808s, request: GET:https://172.30.0.1:443/apis/apps.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-operator-lifecycle-manager...
[must-gather-fmzgt] POD I1016 17:12:21.206747   18911 request.go:621] Throttling request took 1.189298501s, request: GET:https://172.30.0.1:443/apis/packages.operators.coreos.com/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:12:32.416335   18911 request.go:621] Throttling request took 1.142443048s, request: GET:https://172.30.0.1:443/apis/packages.operators.coreos.com/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-service-ca-operator...
[must-gather-fmzgt] POD I1016 17:12:42.531878   18911 request.go:621] Throttling request took 5.751902462s, request: GET:https://172.30.0.1:443/apis/user.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD Gathering data for ns/openshift-service-ca...
[must-gather-fmzgt] POD I1016 17:12:52.676006   18911 request.go:621] Throttling request took 5.583048544s, request: GET:https://172.30.0.1:443/apis/foundation.ibm.com/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-service-catalog-apiserver-operator...
[must-gather-fmzgt] POD I1016 17:13:02.739825   18911 request.go:621] Throttling request took 1.538497851s, request: GET:https://172.30.0.1:443/apis/build.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-service-catalog-apiserver...
[must-gather-fmzgt] POD I1016 17:13:15.233457   18911 request.go:621] Throttling request took 1.133366955s, request: GET:https://172.30.0.1:443/apis/authorization.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD I1016 17:13:25.297153   18911 request.go:621] Throttling request took 1.138701047s, request: GET:https://172.30.0.1:443/apis/iks.clusterapi.io/v1beta1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-service-catalog-controller-manager-operator...
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:13:35.476723   18911 request.go:621] Throttling request took 5.151045348s, request: GET:https://172.30.0.1:443/apis/network.operator.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-service-catalog-controller-manager...
[must-gather-fmzgt] POD I1016 17:13:45.665777   18911 request.go:621] Throttling request took 2.777112679s, request: GET:https://172.30.0.1:443/apis/foundation.ibm.com/v1?timeout=32s
[must-gather-fmzgt] POD Gathering data for ns/openshift-cluster-storage-operator...
[must-gather-fmzgt] POD I1016 17:13:55.747804   18911 request.go:621] Throttling request took 2.166805605s, request: GET:https://172.30.0.1:443/apis/ingress.operator.openshift.io/v1?timeout=32s
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD error: errors ocurred while gathering data:
[must-gather-fmzgt] POD     [unable to retrieve the complete list of server APIs: clusterregistry.k8s.io/v1alpha1: the server is currently unable to handle the request, mcm.ibm.com/v1alpha1: the server is currently unable to handle the request, skipping gathering routes.route.openshift.io/oauth-openshift due to error: a resource cannot be retrieved by name across all namespaces, skipping gathering services/oauth-openshift due to error: a resource cannot be retrieved by name across all namespaces, skipping gathering machineautoscalers.machine.openshift.io due to error: the server doesn't have a resource type "machineautoscalers", skipping gathering clusterautoscalers.machine.openshift.io due to error: the server doesn't have a resource type "clusterautoscalers", skipping gathering namespaces/openshift-console due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-console
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     one or more errors ocurred while gathering container data for pod downloads-5fdf68d856-rzcml:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/openshift-console/downloads-5fdf68d856-rzcml/download-server?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-console/downloads-5fdf68d856-rzcml/download-server?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], skipping gathering namespaces/openshift-dns due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-dns
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     one or more errors ocurred while gathering container data for pod dns-default-85jts:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/openshift-dns/dns-default-85jts/dns?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-dns/dns-default-85jts/dns?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-dns/dns-default-85jts/dns-node-resolver?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-dns/dns-default-85jts/dns-node-resolver?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], skipping gathering namespaces/openshift-image-registry due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-image-registry
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     one or more errors ocurred while gathering container data for pod node-ca-mnt9j:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/openshift-image-registry/node-ca-mnt9j/node-ca?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-image-registry/node-ca-mnt9j/node-ca?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], skipping gathering namespaces/openshift-machine-config-operator due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-machine-config-operator
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     one or more errors ocurred while gathering container data for pod machine-config-daemon-xpkk6:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/openshift-machine-config-operator/machine-config-daemon-xpkk6/machine-config-daemon?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-machine-config-operator/machine-config-daemon-xpkk6/machine-config-daemon?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-machine-config-operator/machine-config-daemon-xpkk6/oauth-proxy?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-machine-config-operator/machine-config-daemon-xpkk6/oauth-proxy?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], skipping gathering namespaces/openshift-marketplace due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-marketplace
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     one or more errors ocurred while gathering container data for pod redhat-operators-64ddc67cdb-j5g9n:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/openshift-marketplace/redhat-operators-64ddc67cdb-j5g9n/redhat-operators?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-marketplace/redhat-operators-64ddc67cdb-j5g9n/redhat-operators?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], skipping gathering namespaces/openshift-monitoring due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-monitoring
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     one or more errors ocurred while gathering container data for pod node-exporter-xr2wg:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/openshift-monitoring/node-exporter-xr2wg/node-exporter?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-monitoring/node-exporter-xr2wg/node-exporter?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-monitoring/node-exporter-xr2wg/kube-rbac-proxy?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-monitoring/node-exporter-xr2wg/kube-rbac-proxy?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-monitoring/node-exporter-xr2wg/init-textfile?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-monitoring/node-exporter-xr2wg/init-textfile?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], skipping gathering all/openshift-monitoring due to error: the server doesn't have a resource type "all", skipping gathering namespaces/openshift-multus due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-multus
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     one or more errors ocurred while gathering container data for pod multus-zr88j:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/openshift-multus/multus-zr88j/kube-multus?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-multus/multus-zr88j/kube-multus?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-multus/multus-zr88j/cni-plugins?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-multus/multus-zr88j/cni-plugins?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-multus/multus-zr88j/multus-binary-copy?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-multus/multus-zr88j/multus-binary-copy?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], skipping gathering namespaces/openshift-sdn due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-sdn
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [one or more errors ocurred while gathering container data for pod ovs-tkvkm:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/openshift-sdn/ovs-tkvkm/openvswitch?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-sdn/ovs-tkvkm/openvswitch?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod sdn-m8f4s:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/openshift-sdn/sdn-m8f4s/sdn?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-sdn/sdn-m8f4s/sdn?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-sdn/sdn-m8f4s/install-cni-plugins?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-sdn/sdn-m8f4s/install-cni-plugins?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused]], skipping gathering namespaces/openshift-cluster-node-tuning-operator due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-cluster-node-tuning-operator
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     one or more errors ocurred while gathering container data for pod tuned-vbc84:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/openshift-cluster-node-tuning-operator/tuned-vbc84/tuned?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/openshift-cluster-node-tuning-operator/tuned-vbc84/tuned?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused]]
[must-gather-fmzgt] POD INFO: gather_ocp log collection complete.
[must-gather-fmzgt] POD WARNING: Collecting one or more service logs on ALL master nodes in your cluster. This could take a large amount of time.
[must-gather-fmzgt] POD INFO: Collecting host service logs for kubelet
[must-gather-fmzgt] POD INFO: Collecting host service logs for crio
[must-gather-fmzgt] POD INFO: Waiting for worker host service log collection to complete ...
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD INFO: Worker host service log collection complete.
[must-gather-fmzgt] POD INFO: gather_cloudpak collecting ...
[must-gather-fmzgt] POD INFO: gather_namespace_scoped_resources collecting ...
[must-gather-fmzgt] POD INFO: gather_namespace_scoped_resources collecting namespace cem.
[must-gather-fmzgt] POD INFO: gather_namespace_scoped_resources collecting resource cem.ingress.
[must-gather-fmzgt] POD INFO: Waiting for gather_namespace_scoped_resources log collection to complete ...
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Gathering data for ns/cem...
[must-gather-fmzgt] POD INFO: Waiting for collection to complete ...
[must-gather-fmzgt] POD I1016 17:15:22.883189   19347 request.go:621] Throttling request took 1.150248008s, request: GET:https://172.30.0.1:443/apis/k8s.cni.cncf.io/v1?timeout=32s
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD error: errors ocurred while gathering data:
[must-gather-fmzgt] POD     one or more errors ocurred while gathering pod-specific data for namespace: cem
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [one or more errors ocurred while gathering container data for pod peter-cassandra-0:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/cem/peter-cassandra-0/peter-cassandra?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-cassandra-0/peter-cassandra?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5/akora-app-cem?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5/akora-app-cem?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5/waitforredis?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5/waitforredis?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused], one or more errors ocurred while gathering container data for pod peter-ibm-cem-datalayer-0:
[must-gather-fmzgt] POD 
[must-gather-fmzgt] POD     [Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-datalayer-0/datalayer?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-datalayer-0/datalayer?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-datalayer-0/waitforcassandra?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-datalayer-0/waitforcassandra?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-datalayer-0/setupdb?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-datalayer-0/setupdb?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-datalayer-0/waitforkafka?previous=true&timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused, Get https://10.16.39.171:10250/containerLogs/cem/peter-ibm-cem-datalayer-0/waitforkafka?timestamps=true: dial tcp 10.16.39.171:10250: connect: connection refused]]
[must-gather-fmzgt] POD INFO: gather_namespace_scoped_resources log collection complete.
[must-gather-fmzgt] POD INFO: gather_cluster_scoped_resources collecting ...
[must-gather-fmzgt] POD INFO: gather_cluster_scoped_resources collecting nodes.
[must-gather-fmzgt] POD INFO: gather_cluster_scoped_resources collecting storageclasses.
[must-gather-fmzgt] POD INFO: gather_cluster_scoped_resources collecting persistentvolumes.
[must-gather-fmzgt] POD INFO: gather_cluster_scoped_resources collecting volumeattachments.
[must-gather-fmzgt] POD INFO: gather_cluster_scoped_resources collecting clusterservicestatuses.clusterhealth.ibm.com.
[must-gather-fmzgt] POD INFO: Waiting for gather_cluster_scoped_resources log collection to complete ...
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD Wrote inspect data to /tmp/must-gather.
[must-gather-fmzgt] POD I1016 17:15:33.821207   19390 request.go:621] Throttling request took 1.134028301s, request: GET:https://172.30.0.1:443/apis/aks.mcm.ibm.com/v1alpha1?timeout=32s
[must-gather-fmzgt] POD error: the server doesn't have a resource type "clusterservicestatuses"
[must-gather-fmzgt] POD INFO: gather_cluster_scoped_resources log collection complete.
[must-gather-fmzgt] POD INFO: gather_cloudpak log collection complete.
[must-gather-fmzgt] POD INFO: gather_cleanup collecting ...
[must-gather-fmzgt] POD INFO: checking and cleanning secrets.yaml (by default we do not collect secrets.yaml from your cluster) ...
[must-gather-fmzgt] POD ./namespaces/openshift-etcd/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-cluster-version/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/default/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/kube-system/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-config/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-config-managed/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-authentication/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-authentication-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-ingress/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-cloud-credential-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-machine-api/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-console-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-console/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-dns-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-dns/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-image-registry/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-ingress-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-insights/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-kube-apiserver-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-kube-apiserver/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-kube-controller-manager/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-kube-scheduler/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-kube-scheduler-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-machine-config-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-marketplace/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-monitoring/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-apiserver-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-apiserver/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-cluster-samples-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-controller-manager-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-controller-manager/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-multus/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-sdn/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-service-catalog-apiserver-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-service-catalog-controller-manager-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-service-catalog-controller-manager/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-network-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-service-ca-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-service-ca/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-service-catalog-apiserver/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/openshift-cluster-storage-operator/core/secrets.yaml
[must-gather-fmzgt] POD ./namespaces/cem/core/secrets.yaml
[must-gather-fmzgt] POD INFO: checking and cleanning routes.yaml ...
[must-gather-fmzgt] POD No resources found in openshift-etcd namespace.
[must-gather-fmzgt] POD No resources found in openshift namespace.
[must-gather-fmzgt] POD No resources found in openshift-cluster-version namespace.
[must-gather-fmzgt] POD No resources found in default namespace.
[must-gather-fmzgt] POD No resources found in openshift-config namespace.
[must-gather-fmzgt] POD No resources found in openshift-config-managed namespace.
[must-gather-fmzgt] POD No resources found in openshift-authentication-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-ingress namespace.
[must-gather-fmzgt] POD No resources found in openshift-cloud-credential-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-machine-api namespace.
[must-gather-fmzgt] POD No resources found in openshift-console-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-dns-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-dns namespace.
[must-gather-fmzgt] POD No resources found in openshift-ingress-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-insights namespace.
[must-gather-fmzgt] POD No resources found in openshift-kube-apiserver-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-kube-apiserver namespace.
[must-gather-fmzgt] POD No resources found in openshift-kube-controller-manager namespace.
[must-gather-fmzgt] POD No resources found in openshift-kube-controller-manager-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-kube-scheduler namespace.
[must-gather-fmzgt] POD No resources found in openshift-kube-scheduler-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-machine-config-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-marketplace namespace.
[must-gather-fmzgt] POD No resources found in openshift-apiserver-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-apiserver namespace.
[must-gather-fmzgt] POD No resources found in openshift-cluster-samples-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-controller-manager-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-controller-manager namespace.
[must-gather-fmzgt] POD No resources found in openshift-multus namespace.
[must-gather-fmzgt] POD No resources found in openshift-operator-lifecycle-manager namespace.
[must-gather-fmzgt] POD No resources found in openshift-sdn namespace.
[must-gather-fmzgt] POD No resources found in openshift-service-catalog-apiserver-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-service-catalog-controller-manager-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-service-catalog-controller-manager namespace.
[must-gather-fmzgt] POD No resources found in openshift-network-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-cluster-node-tuning-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-service-ca-operator namespace.
[must-gather-fmzgt] POD No resources found in openshift-service-ca namespace.
[must-gather-fmzgt] POD No resources found in openshift-service-catalog-apiserver namespace.
[must-gather-fmzgt] POD No resources found in openshift-cluster-storage-operator namespace.
[must-gather-fmzgt] POD No resources found in cem namespace.
[must-gather-fmzgt] POD INFO: tar collections to cloudpak-must-gather-20201016171535.tar.gz ...
[must-gather-fmzgt] POD ./cluster-scoped-resources/
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/nfs1.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-02094b7e-2147-4a3d-ae27-7a0d889b5e11.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-12737f05-b8b7-43e7-ac77-efccb4d0f6b5.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-25aedbe8-6bf3-4d3c-a833-be9eaf74b657.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-27c65d1e-e727-4004-a111-8ddecdc51354.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-3f31114a-488f-456f-aa5a-95f2ad280816.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-4b723cbe-d65b-45bb-aa6b-4e1cab7741e2.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-52cc0d71-dc14-4601-bfd3-67858c283cf1.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-62fbd259-23a2-4c2d-b9b9-1dbc700bbd42.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-683927fc-e90d-44d7-acfc-38969347620e.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-6b4375fe-eed0-4bc2-b9a3-78ae69974505.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-7e9a97e3-623c-4d21-8a24-8d9a39f5bbe2.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-80666146-c50f-4b39-9487-7d061ae8d08d.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-84fdb65b-e2ba-49b2-b87e-daac3bc9eaca.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-d8697f54-4ba7-4837-8b2c-eab906cd6d26.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/pvc-d8ac96f1-1410-4ba6-a6fe-b16acebc7b37.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/persistentvolumes/registry-storage.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/master0.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/master1.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/master2.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/worker0.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/worker1.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/worker2.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/worker3.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/worker4.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/worker5.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/core/nodes/worker6.scao3-ocp43.os.fyre.ibm.com.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigpools/
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigpools/master.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigpools/worker.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/00-master.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/00-worker.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/01-master-container-runtime.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/01-master-kubelet.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/01-worker-container-runtime.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/01-worker-kubelet.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/99-master-6d17d94e-ed2b-48a2-8c56-40ebb111cbfb-registries.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/99-master-ssh.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/99-worker-c5cb75e7-e62e-47a8-a20a-759bb6f378c4-registries.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/99-worker-ssh.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/rendered-master-3b8c0988716af71315cbdc15f54c4f25.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/rendered-master-af998b63598fe47dcd8ad07ad544aa53.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/rendered-master-cdace211abd164b3170cd908fb3cdaa3.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/rendered-worker-00ad89a6b224986f8b48c918aeb2b2fa.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/rendered-worker-4d08c8783325e6d08177c8764e2e1777.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigs/rendered-worker-793cbb543301736c9e5768982d0598fd.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/controllerconfigs/
[must-gather-fmzgt] POD ./cluster-scoped-resources/machineconfiguration.openshift.io/controllerconfigs/machine-config-controller.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/storageclasses/
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/storageclasses/rook-ceph-block-internal.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/storageclasses/rook-ceph-cephfs-internal.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/storageclasses/rook-ceph-delete-bucket-internal.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-222503cde8b309a2680a39df212408c220000d98d79312487af66be0040ea77a.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-2faabee05930403b3ccfe0eb11b85383d3ce23d64837046001248623d5d7faae.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-51920a5f6a0958f1a5c929173b8dff218b0a0163ed2c71b4218768d1eaf079a4.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-8afa11face1adb593b2509ad043c8e4d771c7eb96f3c6497f7ef1b3eebfafae0.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-b578f63ff33deae8e7e2c9be80c5ad86376df51d176c5b34546a9b692133c59f.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-d8163010f312347ce460bbbe8bd871b07cbfa98078e0454d71117b28278dc55d.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-df1f36a2ccc0d3f19f19dd4475e4afce1f0e033604e18639e36c7b27931addbf.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-e4b1be47a274221a16bcc5e5a249f2f68202675ff4e2520e76990762a33c40cf.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-f5a5a77e4d1b903153a4b5dc9e6e6ca9e51b758f2101eb50a66f013789de3b28.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-f6230732b352fef8bad1e4976b8deddc3f643e42098107210f714fd3b3d257fd.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusterversions/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/images.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/networks.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/projects.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/builds.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusterversions.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/ingresses.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/oauths.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/dnses.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/proxies.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/operatorhubs.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/infrastructures.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/authentications.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/consoles.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/schedulers.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/apiservers.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/featuregates.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/authentication.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/cloud-credential.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/cluster-autoscaler.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/machine-api.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/machine-config.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/marketplace.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/service-catalog-apiserver.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/service-catalog-controller-manager.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/authentications/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/authentications/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/infrastructures/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/oauths/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/consoles/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/proxies/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/images/
[must-gather-fmzgt] POD ./cluster-scoped-resources/config.openshift.io/images/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/imagecontentsourcepolicies/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/imagecontentsourcepolicies/mirror-config.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/networks/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/authentications/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/authentications/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/consoles/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/dnses/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/dnses/default.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/kubeapiservers/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/kubeschedulers/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/openshiftapiservers/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/servicecas/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/servicecatalogcontrollermanagers/
[must-gather-fmzgt] POD ./cluster-scoped-resources/operator.openshift.io/servicecatalogcontrollermanagers/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/oauth.openshift.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/oauth.openshift.io/oauthclients/
[must-gather-fmzgt] POD ./cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/imageregistry.operator.openshift.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/imageregistry.operator.openshift.io/configs/
[must-gather-fmzgt] POD ./cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/machine-api-operator.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/machine-api-controllers.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-sdn.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-sdn-controller.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning:tuned.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-sdn.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-sdn-controller.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning:tuned.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiextensions.k8s.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusternetworks.network.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/hostsubnets.network.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/netnamespaces.network.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressnetworkpolicies.network.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/admissionregistration.k8s.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/
[must-gather-fmzgt] POD ./cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/network.openshift.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/network.openshift.io/clusternetworks/
[must-gather-fmzgt] POD ./cluster-scoped-resources/network.openshift.io/clusternetworks/default.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.servicecatalog.k8s.io.yaml
[must-gather-fmzgt] POD ./cluster-scoped-resources/samples.operator.openshift.io/
[must-gather-fmzgt] POD ./cluster-scoped-resources/samples.operator.openshift.io/configs/
[must-gather-fmzgt] POD ./cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml
[must-gather-fmzgt] POD ./failure/
[must-gather-fmzgt] POD ./failure/pods/
[must-gather-fmzgt] POD ./failure/pods/pods
[must-gather-fmzgt] POD ./failure/pods/cem.peter-cassandra-0.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-cassandra-0.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-cassandra-0.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-cassandra-0.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-akora-app-cem-5c76f8b699-pt9d5.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-0.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-0.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-0.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-0.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602724500-tk6c5.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602724500-tk6c5.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602724500-tk6c5.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602724500-tk6c5.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-596n6.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-596n6.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-596n6.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-596n6.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-6nr5m.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-6nr5m.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-6nr5m.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-6nr5m.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-d7cpl.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-d7cpl.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-d7cpl.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-d7cpl.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-q9xjd.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-q9xjd.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-q9xjd.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-q9xjd.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-rjzng.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-rjzng.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-rjzng.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-rjzng.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-zvcv4.get.yaml
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-zvcv4.current.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-zvcv4.previous.log
[must-gather-fmzgt] POD ./failure/pods/cem.peter-ibm-cem-datalayer-cron-1602810900-zvcv4.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-console-mcmuiapi-bfbd55ddc-q7bhn.current.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-console-mcmuiapi-bfbd55ddc-q7bhn.get.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-console-mcmuiapi-bfbd55ddc-q7bhn.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-console-mcmuiapi-bfbd55ddc-q7bhn.previous.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-core-apiserver-7f6645f546-gwcbp.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-core-apiserver-7f6645f546-gwcbp.current.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-core-apiserver-7f6645f546-gwcbp.get.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-core-apiserver-7f6645f546-gwcbp.previous.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-grafeas-64f6c9c7c-xkq2f.current.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-grafeas-64f6c9c7c-xkq2f.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-grafeas-64f6c9c7c-xkq2f.get.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-grafeas-64f6c9c7c-xkq2f.previous.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-grcuiapi-75c4fcd6bb-gsqm7.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-grcuiapi-75c4fcd6bb-gsqm7.current.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-grcuiapi-75c4fcd6bb-gsqm7.previous.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-grcuiapi-75c4fcd6bb-gsqm7.get.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-legato-5bbc7f776b-f4tnt.get.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-legato-5bbc7f776b-f4tnt.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-legato-5bbc7f776b-f4tnt.previous.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-legato-5bbc7f776b-f4tnt.current.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-topologyapi-5db998858f-f62wt.get.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-topologyapi-5db998858f-f62wt.previous.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-topologyapi-5db998858f-f62wt.current.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.multicluster-hub-topologyapi-5db998858f-f62wt.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.search-search-api-7b4c978599-hbjxg.get.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.search-search-api-7b4c978599-hbjxg.current.log
[must-gather-fmzgt] POD ./failure/pods/kube-system.search-search-api-7b4c978599-hbjxg.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/kube-system.search-search-api-7b4c978599-hbjxg.previous.log
[must-gather-fmzgt] POD ./failure/pods/openshift-console.downloads-5fdf68d856-rzcml.get.yaml
[must-gather-fmzgt] POD ./failure/pods/openshift-console.downloads-5fdf68d856-rzcml.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/openshift-console.downloads-5fdf68d856-rzcml.current.log
[must-gather-fmzgt] POD ./failure/pods/openshift-console.downloads-5fdf68d856-rzcml.previous.log
[must-gather-fmzgt] POD ./failure/pods/openshift-kube-apiserver.[installer-67](https://issues.redhat.com/browse/installer-67)-master2.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/openshift-kube-apiserver.[installer-67](https://issues.redhat.com/browse/installer-67)-master2.scao3-ocp43.os.fyre.ibm.com.current.log
[must-gather-fmzgt] POD ./failure/pods/openshift-kube-apiserver.[installer-67](https://issues.redhat.com/browse/installer-67)-master2.scao3-ocp43.os.fyre.ibm.com.get.yaml
[must-gather-fmzgt] POD ./failure/pods/openshift-kube-apiserver.[installer-67](https://issues.redhat.com/browse/installer-67)-master2.scao3-ocp43.os.fyre.ibm.com.previous.log
[must-gather-fmzgt] POD ./failure/pods/openshift-kube-apiserver.revision-pruner-66-master1.scao3-ocp43.os.fyre.ibm.com.current.log
[must-gather-fmzgt] POD ./failure/pods/openshift-kube-apiserver.revision-pruner-66-master1.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./failure/pods/openshift-kube-apiserver.revision-pruner-66-master1.scao3-ocp43.os.fyre.ibm.com.get.yaml
[must-gather-fmzgt] POD ./failure/pods/openshift-kube-apiserver.revision-pruner-66-master1.scao3-ocp43.os.fyre.ibm.com.previous.log
[must-gather-fmzgt] POD ./failure/pods/rook-ceph.rook-ceph-osd-6-58bd78b9b7-d2qsp.previous.log
[must-gather-fmzgt] POD ./failure/pods/rook-ceph.rook-ceph-osd-6-58bd78b9b7-d2qsp.current.log
[must-gather-fmzgt] POD ./failure/pods/rook-ceph.rook-ceph-osd-6-58bd78b9b7-d2qsp.get.yaml
[must-gather-fmzgt] POD ./failure/pods/rook-ceph.rook-ceph-osd-6-58bd78b9b7-d2qsp.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/
[must-gather-fmzgt] POD ./failure/restart_pods/restart_pods
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-datalayer-0.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-datalayer-0.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-datalayer-0.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-datalayer-0.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-rba-as-566bcffb6-tnzqp.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-rba-as-566bcffb6-tnzqp.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-rba-as-566bcffb6-tnzqp.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-rba-as-566bcffb6-tnzqp.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-rba-rbs-689475646d-png8d.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-rba-rbs-689475646d-png8d.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-rba-rbs-689475646d-png8d.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/cem.peter-ibm-cem-rba-rbs-689475646d-png8d.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cert-manager.cert-manager-ibm-cert-manager-5cd9fb9777-cks5p.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/cert-manager.cert-manager-ibm-cert-manager-5cd9fb9777-cks5p.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cert-manager.cert-manager-ibm-cert-manager-5cd9fb9777-cks5p.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/cert-manager.cert-manager-ibm-cert-manager-5cd9fb9777-cks5p.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cert-manager.cert-manager-webhook-cainjector-f46ff7b74-p442r.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cert-manager.cert-manager-webhook-cainjector-f46ff7b74-p442r.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/cert-manager.cert-manager-webhook-cainjector-f46ff7b74-p442r.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/cert-manager.cert-manager-webhook-cainjector-f46ff7b74-p442r.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-bckx9.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-bckx9.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-bckx9.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-bckx9.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-z2bwn.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-z2bwn.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-z2bwn.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-z2bwn.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-zkg7d.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-zkg7d.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-zkg7d.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-idp-zkg7d.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-ff9rk.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-ff9rk.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-ff9rk.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-ff9rk.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-grgp4.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-grgp4.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-grgp4.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-grgp4.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-ln9rd.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-ln9rd.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-ln9rd.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.auth-pdp-ln9rd.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-4ndvq.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-4ndvq.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-4ndvq.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-4ndvq.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-gw6tk.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-gw6tk.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-gw6tk.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-gw6tk.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-jl2lz.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-jl2lz.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-jl2lz.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.catalog-ui-jl2lz.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-wpwtq.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-wpwtq.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-wpwtq.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-wpwtq.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-zk2l6.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-zk2l6.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-zk2l6.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-zk2l6.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-zw46s.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-zw46s.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-zw46s.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.common-web-ui-zw46s.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.icp-mongodb-0.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.icp-mongodb-0.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.icp-mongodb-0.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.icp-mongodb-0.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.icp-mongodb-1.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.icp-mongodb-1.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.icp-mongodb-1.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.icp-mongodb-1.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.internal-management-ingress-9gwsh.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.internal-management-ingress-9gwsh.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.internal-management-ingress-9gwsh.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.internal-management-ingress-9gwsh.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.internal-management-ingress-scgxp.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.internal-management-ingress-scgxp.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.internal-management-ingress-scgxp.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.internal-management-ingress-scgxp.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-dm-c7b75c686-n2b7h.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-dm-c7b75c686-n2b7h.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-dm-c7b75c686-n2b7h.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-dm-c7b75c686-n2b7h.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-7522r.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-7522r.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-7522r.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-7522r.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-7bwdl.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-7bwdl.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-7bwdl.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-7bwdl.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-8x48b.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-8x48b.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-8x48b.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-8x48b.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-bhpnk.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-bhpnk.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-bhpnk.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-bhpnk.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-l5wsj.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-l5wsj.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-l5wsj.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-l5wsj.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-ppzhz.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-ppzhz.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-ppzhz.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-ppzhz.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-qsj6z.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-qsj6z.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-qsj6z.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.metering-reader-qsj6z.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.monitoring-grafana-67ddcdf99d-k7df4.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.monitoring-grafana-67ddcdf99d-k7df4.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.monitoring-grafana-67ddcdf99d-k7df4.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.monitoring-grafana-67ddcdf99d-k7df4.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.monitoring-prometheus-nodeexporter-2cd57.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.monitoring-prometheus-nodeexporter-2cd57.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.monitoring-prometheus-nodeexporter-2cd57.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.monitoring-prometheus-nodeexporter-2cd57.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-aks-b84b6657-pqhgx.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-aks-b84b6657-pqhgx.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-aks-b84b6657-pqhgx.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-aks-b84b6657-pqhgx.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-eks-c8b6c956f-bmrlz.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-eks-c8b6c956f-bmrlz.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-eks-c8b6c956f-bmrlz.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-eks-c8b6c956f-bmrlz.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-gke-547c5c7b59-kpprf.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-gke-547c5c7b59-kpprf.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-gke-547c5c7b59-kpprf.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-gke-547c5c7b59-kpprf.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-iks-7c4999db75-fm62g.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-iks-7c4999db75-fm62g.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-iks-7c4999db75-fm62g.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-iks-7c4999db75-fm62g.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-ocp-6d5fbd8459-ckg46.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-ocp-6d5fbd8459-ckg46.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-ocp-6d5fbd8459-ckg46.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-ocp-6d5fbd8459-ckg46.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-sdk-b6979fffb-fm89p.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-sdk-b6979fffb-fm89p.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-sdk-b6979fffb-fm89p.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-cluster-api-provider-sdk-b6979fffb-fm89p.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-core-apiserver-7f6645f546-gwcbp.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-core-apiserver-7f6645f546-gwcbp.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-core-apiserver-7f6645f546-gwcbp.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-core-apiserver-7f6645f546-gwcbp.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-core-channel-59fdf84bf4-5sjm4.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-core-channel-59fdf84bf4-5sjm4.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-core-channel-59fdf84bf4-5sjm4.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.multicluster-hub-core-channel-59fdf84bf4-5sjm4.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-czjkn.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-czjkn.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-czjkn.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-czjkn.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-h95x9.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-h95x9.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-h95x9.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-h95x9.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-smppk.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-smppk.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-smppk.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.nginx-ingress-controller-smppk.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-api-794c95849c-jhjpn.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-api-794c95849c-jhjpn.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-api-794c95849c-jhjpn.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-api-794c95849c-jhjpn.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-api-794c95849c-wz89k.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-api-794c95849c-wz89k.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-api-794c95849c-wz89k.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-api-794c95849c-wz89k.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-49gzc.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-49gzc.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-49gzc.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-49gzc.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-s8tp2.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-s8tp2.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-s8tp2.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-s8tp2.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-xhp4k.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-xhp4k.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-xhp4k.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.platform-header-xhp4k.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.prometheus-monitoring-prometheus-0.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.prometheus-monitoring-prometheus-0.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.prometheus-monitoring-prometheus-0.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.prometheus-monitoring-prometheus-0.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.search-search-aggregator-64b75d9466-lj9ms.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.search-search-aggregator-64b75d9466-lj9ms.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.search-search-aggregator-64b75d9466-lj9ms.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.search-search-aggregator-64b75d9466-lj9ms.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.search-search-collector-6c68bd89bd-rhmt2.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.search-search-collector-6c68bd89bd-rhmt2.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.search-search-collector-6c68bd89bd-rhmt2.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.search-search-collector-6c68bd89bd-rhmt2.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.secret-watcher-779df94cc8-z7g8c.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.secret-watcher-779df94cc8-z7g8c.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.secret-watcher-779df94cc8-z7g8c.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/kube-system.secret-watcher-779df94cc8-z7g8c.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver-operator.openshift-apiserver-operator-7cc77d965f-x8pff.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver-operator.openshift-apiserver-operator-7cc77d965f-x8pff.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver-operator.openshift-apiserver-operator-7cc77d965f-x8pff.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver-operator.openshift-apiserver-operator-7cc77d965f-x8pff.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-8rpk8.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-8rpk8.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-8rpk8.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-8rpk8.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-j5gsv.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-j5gsv.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-j5gsv.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-j5gsv.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-lm8sl.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-lm8sl.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-lm8sl.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-apiserver.apiserver-lm8sl.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-authentication-operator.authentication-operator-57d4974d5d-6tk6f.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-authentication-operator.authentication-operator-57d4974d5d-6tk6f.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-authentication-operator.authentication-operator-57d4974d5d-6tk6f.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-authentication-operator.authentication-operator-57d4974d5d-6tk6f.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-cloud-credential-operator.cloud-credential-operator-dd496454-57rdg.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-cloud-credential-operator.cloud-credential-operator-dd496454-57rdg.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-cluster-version.cluster-version-operator-584fddff45-tgdkd.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-cloud-credential-operator.cloud-credential-operator-dd496454-57rdg.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-cloud-credential-operator.cloud-credential-operator-dd496454-57rdg.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-cluster-version.cluster-version-operator-584fddff45-tgdkd.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-cluster-version.cluster-version-operator-584fddff45-tgdkd.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-cluster-version.cluster-version-operator-584fddff45-tgdkd.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-console-operator.console-operator-57f5bcc578-6q5vj.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-console-operator.console-operator-57f5bcc578-6q5vj.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-console-operator.console-operator-57f5bcc578-6q5vj.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-console-operator.console-operator-57f5bcc578-6q5vj.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager-operator.openshift-controller-manager-operator-68dcf95c47-6bz2w.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager-operator.openshift-controller-manager-operator-68dcf95c47-6bz2w.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager-operator.openshift-controller-manager-operator-68dcf95c47-6bz2w.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager-operator.openshift-controller-manager-operator-68dcf95c47-6bz2w.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager.controller-manager-2zczq.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager.controller-manager-2zczq.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager.controller-manager-2zczq.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager.controller-manager-2zczq.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager.controller-manager-6r22f.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager.controller-manager-6r22f.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager.controller-manager-6r22f.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-controller-manager.controller-manager-6r22f.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-85jts.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-85jts.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-85jts.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-85jts.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-c9czj.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-c9czj.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-c9czj.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-c9czj.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-f87fn.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-f87fn.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-f87fn.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-f87fn.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-klp8r.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-klp8r.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-klp8r.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-klp8r.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-m9p5v.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-m9p5v.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-m9p5v.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-m9p5v.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-mnfxh.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-mnfxh.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-mnfxh.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-mnfxh.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-p62vs.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-p62vs.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-p62vs.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-p62vs.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-qbs6q.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-qbs6q.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-qbs6q.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-qbs6q.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-vvw7j.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-vvw7j.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-vvw7j.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-vvw7j.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-wxv5g.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-wxv5g.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-wxv5g.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-dns.dns-default-wxv5g.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master0.scao3-ocp43.os.fyre.ibm.com.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master0.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master0.scao3-ocp43.os.fyre.ibm.com.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master0.scao3-ocp43.os.fyre.ibm.com.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master1.scao3-ocp43.os.fyre.ibm.com.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master1.scao3-ocp43.os.fyre.ibm.com.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master1.scao3-ocp43.os.fyre.ibm.com.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master1.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master2.scao3-ocp43.os.fyre.ibm.com.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master2.scao3-ocp43.os.fyre.ibm.com.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master2.scao3-ocp43.os.fyre.ibm.com.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-etcd.etcd-member-master2.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-55mpk.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-55mpk.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-55mpk.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-55mpk.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-8gljk.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-8gljk.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-8gljk.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-8gljk.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-fwp52.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-fwp52.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-fwp52.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-fwp52.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-h4p5w.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-h4p5w.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-h4p5w.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-h4p5w.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-hrsqc.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-hrsqc.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-hrsqc.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-hrsqc.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-j9gfj.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-j9gfj.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-j9gfj.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-j9gfj.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-ktrnn.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-ktrnn.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-ktrnn.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-ktrnn.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-mnt9j.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-mnt9j.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-mnt9j.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-mnt9j.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-wpzqh.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-wpzqh.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-wpzqh.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-wpzqh.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-xmr4j.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-xmr4j.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-xmr4j.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-image-registry.node-ca-xmr4j.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver-operator.kube-apiserver-operator-566b9798-7rjbl.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver-operator.kube-apiserver-operator-566b9798-7rjbl.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver-operator.kube-apiserver-operator-566b9798-7rjbl.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver-operator.kube-apiserver-operator-566b9798-7rjbl.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master0.scao3-ocp43.os.fyre.ibm.com.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master0.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master0.scao3-ocp43.os.fyre.ibm.com.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master0.scao3-ocp43.os.fyre.ibm.com.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master1.scao3-ocp43.os.fyre.ibm.com.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master1.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master1.scao3-ocp43.os.fyre.ibm.com.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master1.scao3-ocp43.os.fyre.ibm.com.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master2.scao3-ocp43.os.fyre.ibm.com.get.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master2.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master2.scao3-ocp43.os.fyre.ibm.com.current.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-apiserver.kube-apiserver-master2.scao3-ocp43.os.fyre.ibm.com.previous.log
[must-gather-fmzgt] POD ./failure/restart_pods/openshift-kube-controller-manager-operator.kube-controller-manager-operator-7c8b7465b-gd8nf.get.yaml
...
[must-gather-fmzgt] POD ./overview/nodes.worker1.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./overview/nodes.worker2.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./overview/nodes.worker3.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./overview/nodes.worker4.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./overview/nodes.worker5.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./overview/nodes.worker6.scao3-ocp43.os.fyre.ibm.com.describe.yaml
[must-gather-fmzgt] POD ./overview/pods
[must-gather-fmzgt] POD ./overview/deployments
[must-gather-fmzgt] POD ./overview/statefulsets
[must-gather-fmzgt] POD ./overview/daemonsets
[must-gather-fmzgt] POD ./overview/jobs
[must-gather-fmzgt] POD ./overview/cronjobs
[must-gather-fmzgt] POD ./overview/pod_resources
[must-gather-fmzgt] POD ./overview/configmaps
[must-gather-fmzgt] POD ./overview/secrets
[must-gather-fmzgt] POD ./overview/resourcequotas
[must-gather-fmzgt] POD ./overview/hpas
[must-gather-fmzgt] POD ./overview/services
[must-gather-fmzgt] POD ./overview/endpoints
[must-gather-fmzgt] POD ./overview/ingresses
[must-gather-fmzgt] POD ./overview/routes
[must-gather-fmzgt] POD ./overview/networkpolicies
[must-gather-fmzgt] POD ./overview/persistentvolumes
[must-gather-fmzgt] POD ./overview/persistentvolumeclaims
[must-gather-fmzgt] POD ./overview/storageclasses
[must-gather-fmzgt] POD ./overview/namespaces
[must-gather-fmzgt] POD ./overview/events
[must-gather-fmzgt] POD ./overview/crds
[must-gather-fmzgt] POD ./overview/operator.ibm.com
[must-gather-fmzgt] POD ./overview/csvs
[must-gather-fmzgt] POD ./overview/certificates.certmanager
[must-gather-fmzgt] POD ./overview/challenges.certmanager
[must-gather-fmzgt] POD ./overview/clusterissuers.certmanager
[must-gather-fmzgt] POD ./overview/issuers.certmanager
[must-gather-fmzgt] POD ./overview/orders.certmanager
[must-gather-fmzgt] POD ./overview/status
[must-gather-fmzgt] POD ./overview/clusterservicestatus
[must-gather-fmzgt] POD ./summary
[must-gather-fmzgt] POD ./system/
[must-gather-fmzgt] POD ./system/host_info/
[must-gather-fmzgt] POD ./system/host_info/master2.scao3-ocp43.os.fyre.ibm.com.proc.cpuinfo
[must-gather-fmzgt] POD ./system/host_info/master2.scao3-ocp43.os.fyre.ibm.com.proc.meminfo
[must-gather-fmzgt] POD ./system/host_info/master2.scao3-ocp43.os.fyre.ibm.com.ip.address
[must-gather-fmzgt] POD ./system/host_info/master2.scao3-ocp43.os.fyre.ibm.com.df
[must-gather-fmzgt] POD ./system/host_info/master2.scao3-ocp43.os.fyre.ibm.com.dmesg
[must-gather-fmzgt] POD ./system/host_info/master1.scao3-ocp43.os.fyre.ibm.com.proc.cpuinfo
[must-gather-fmzgt] POD ./system/host_info/master1.scao3-ocp43.os.fyre.ibm.com.proc.meminfo
[must-gather-fmzgt] POD ./system/host_info/master1.scao3-ocp43.os.fyre.ibm.com.df
[must-gather-fmzgt] POD ./system/host_info/master1.scao3-ocp43.os.fyre.ibm.com.dmesg
[must-gather-fmzgt] POD ./system/host_info/master1.scao3-ocp43.os.fyre.ibm.com.ip.address
[must-gather-fmzgt] POD ./system/host_info/master0.scao3-ocp43.os.fyre.ibm.com.proc.cpuinfo
[must-gather-fmzgt] POD ./system/host_info/master0.scao3-ocp43.os.fyre.ibm.com.proc.meminfo
[must-gather-fmzgt] POD ./system/host_info/master0.scao3-ocp43.os.fyre.ibm.com.df
[must-gather-fmzgt] POD ./system/host_info/master0.scao3-ocp43.os.fyre.ibm.com.ip.address
[must-gather-fmzgt] POD ./system/host_info/master0.scao3-ocp43.os.fyre.ibm.com.dmesg
[must-gather-fmzgt] POD ./system/host_service/
[must-gather-fmzgt] POD ./system/host_service/masters/
[must-gather-fmzgt] POD ./system/host_service/masters/master0.scao3-ocp43.os.fyre.ibm.com.kubelet.log
[must-gather-fmzgt] POD ./system/host_service/masters/master2.scao3-ocp43.os.fyre.ibm.com.kubelet.log
[must-gather-fmzgt] POD ./system/host_service/masters/master1.scao3-ocp43.os.fyre.ibm.com.kubelet.log
[must-gather-fmzgt] POD ./system/host_service/masters/master0.scao3-ocp43.os.fyre.ibm.com.crio.log
[must-gather-fmzgt] POD ./system/host_service/masters/master1.scao3-ocp43.os.fyre.ibm.com.crio.log
[must-gather-fmzgt] POD ./system/host_service/masters/master2.scao3-ocp43.os.fyre.ibm.com.crio.log
[must-gather-fmzgt] POD ./system/network/
[must-gather-fmzgt] POD ./system/network/master0.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/master0.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/master0.scao3-ocp43.os.fyre.ibm.com_ovs-9vskw_vswitchd_log
[must-gather-fmzgt] POD ./system/network/master0.scao3-ocp43.os.fyre.ibm.com_ovs-9vskw_ovsdb_log
[must-gather-fmzgt] POD ./system/network/master1.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/master1.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/master1.scao3-ocp43.os.fyre.ibm.com_ovs-w9qjl_vswitchd_log
[must-gather-fmzgt] POD ./system/network/master1.scao3-ocp43.os.fyre.ibm.com_ovs-w9qjl_ovsdb_log
[must-gather-fmzgt] POD ./system/network/master2.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/master2.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/master2.scao3-ocp43.os.fyre.ibm.com_ovs-tr4x8_vswitchd_log
[must-gather-fmzgt] POD ./system/network/master2.scao3-ocp43.os.fyre.ibm.com_ovs-tr4x8_ovsdb_log
[must-gather-fmzgt] POD ./system/network/worker0.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/worker0.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/worker0.scao3-ocp43.os.fyre.ibm.com_ovs-hxlsk_vswitchd_log
[must-gather-fmzgt] POD ./system/network/worker0.scao3-ocp43.os.fyre.ibm.com_ovs-hxlsk_ovsdb_log
[must-gather-fmzgt] POD ./system/network/worker1.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/worker1.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/worker1.scao3-ocp43.os.fyre.ibm.com_ovs-mwgvq_vswitchd_log
[must-gather-fmzgt] POD ./system/network/worker1.scao3-ocp43.os.fyre.ibm.com_ovs-mwgvq_ovsdb_log
[must-gather-fmzgt] POD ./system/network/worker2.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/worker2.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/worker2.scao3-ocp43.os.fyre.ibm.com_ovs-45bjp_vswitchd_log
[must-gather-fmzgt] POD ./system/network/worker2.scao3-ocp43.os.fyre.ibm.com_ovs-45bjp_ovsdb_log
[must-gather-fmzgt] POD ./system/network/worker3.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/worker3.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/worker3.scao3-ocp43.os.fyre.ibm.com_ovs-mxcjw_vswitchd_log
[must-gather-fmzgt] POD ./system/network/worker3.scao3-ocp43.os.fyre.ibm.com_ovs-mxcjw_ovsdb_log
[must-gather-fmzgt] POD ./system/network/worker4.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/worker4.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/worker4.scao3-ocp43.os.fyre.ibm.com_ovs-tkvkm_vswitchd_log
[must-gather-fmzgt] POD ./system/network/worker4.scao3-ocp43.os.fyre.ibm.com_ovs-tkvkm_ovsdb_log
[must-gather-fmzgt] POD ./system/network/worker5.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/worker5.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/worker5.scao3-ocp43.os.fyre.ibm.com_ovs-dbkck_vswitchd_log
[must-gather-fmzgt] POD ./system/network/worker5.scao3-ocp43.os.fyre.ibm.com_ovs-dbkck_ovsdb_log
[must-gather-fmzgt] POD ./system/network/worker6.scao3-ocp43.os.fyre.ibm.com_iptables
[must-gather-fmzgt] POD ./system/network/worker6.scao3-ocp43.os.fyre.ibm.com_ovs_dump
[must-gather-fmzgt] POD ./system/network/worker6.scao3-ocp43.os.fyre.ibm.com_ovs-qdgkb_vswitchd_log
[must-gather-fmzgt] POD ./system/network/worker6.scao3-ocp43.os.fyre.ibm.com_ovs-qdgkb_ovsdb_log
[must-gather-fmzgt] POD ./version
[must-gather-fmzgt] POD INFO: cloudpak-must-gather-20201016171535.tar.gz will to copied back to your local host ./must-gather.local.<rand> or what you defined in 'oc adm must-gather --dest-dir'
[must-gather-fmzgt] OUT waiting for gather to complete
[must-gather-fmzgt] OUT downloading gather output
[must-gather-fmzgt] OUT receiving file list ... done
[must-gather-fmzgt] OUT ./
[must-gather-fmzgt] OUT cloudpak-must-gather-20201016171535.tar.gz
[must-gather-fmzgt] OUT 
[must-gather-fmzgt] OUT sent 44 bytes  received 499036579 bytes  2404995.77 bytes/sec
[must-gather-fmzgt] OUT total size is 498914627  speedup is 1.00
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-c2z77 deleted
[must-gather      ] OUT namespace/openshift-must-gather-4l77r deleted

[RFE] Add the version of the branch to the version file

Most of the times, the version file (generated by the version function) only shows "4.0-art-dev" or "Unknown", which is not very useful.
If it's not possible to get the version of the image used, at least the branch in this repo (and maybe the last commit applied) could be added to the output of the version function (release-4.6, release-4.7, release-4.8, release-4.9, release-4.10, ...).

latest images of must-gather is not collecting OLM data

I believe that gather_olm script is not collecting data in 4.6 images (I've tested only in this Y).

I expect that the script gather_olm collects OLM operator objects (sub, installPlans, catalogSources, CSVs) and save inside each place, we can see it when the command bellow is executed directly:

oc adm inspect --dest-dir=must-gather-olm -A olm

Steps to reproduce

I wrote 3 scenarios running the most recent image for Y 4.6:
A) script[0]+[1]: default must-gather - should run all collection-scripts/gather_* (including gather_olm)
B) script[0]+[2]: must-gather calling the script directly: -- /usr/bin/gather_olm
C) script[0]+[3]: collecting the same command define on gather_olm

So we can see that A and B are not collecting OLM objects, and when running the command directly on C the objects can be collected.

[0] Get OCP version

$declare -r OCP_VERSION=$(oc get clusterversion version --template='{{.status.desired.version}}' |sed -n -r 's/([[:digit:]]\.[[:digit:]]).*/\1/p')

[1] default must-gather

$ echo ${OCP_VERSION}
4.6

$ oc adm must-gather \
    --dest-dir=must-gather-general \
    --image=quay.io/openshift/origin-must-gather:"${OCP_VERSION}"

[must-gather      ] OUT Using must-gather plugin-in image: quay.io/openshift/origin-must-gather:${OCP_VERSION}
[...]

$ tree -a must-gather-general/*cluster-scoped-resources/operators.coreos.com/operators
must-gather-general/*cluster-scoped-resources/operators.coreos.com/operators [error opening dir]

0 directories, 0 files

$  tree -a must-gather-general/*cluster-scoped-resources/operators.coreos.com/operators
must-gather-general/*cluster-scoped-resources/operators.coreos.com/operators [error opening dir]

0 directories, 0 files

$ tree -a must-gather-general/*namespaces/openshift-marketplace/operators.coreos.com/catalogsources
must-gather-general/*namespaces/openshift-marketplace/operators.coreos.com/catalogsources [error opening dir]

0 directories, 0 files

[2] must-gather calling the script directly: -- /usr/bin/gather_olm

oc adm must-gather \
    --dest-dir=must-gather-general-olm \
    --image=quay.io/openshift/origin-must-gather:"${OCP_VERSION}" -- /usr/bin/gather_olm

$ tree -a must-gather-general-olm/*cluster-scoped-resources/operators.coreos.com/operators
must-gather-general-olm/*cluster-scoped-resources/operators.coreos.com/operators [error opening dir]

[3] running the same command on gather_olm

$ oc adm inspect --dest-dir=must-gather-olm -A olm
Wrote inspect data to must-gather-olm.

$ tree -a must-gather-olm/*cluster-scoped-resources/operators.coreos.com/operators
must-gather-olm/cluster-scoped-resources/operators.coreos.com/operators
├── 3scale-community-operator.3scale.yaml
├── 3scale-operator.3scale.yaml
├── cluster-logging.openshift-logging.yaml
├── compliance-operator.openshift-compliance.yaml
├── elasticsearch-operator.openshift-operators-redhat.yaml
└── web-terminal.openshift-operators.yaml

$ tree -a must-gather-olm/*namespaces/openshift-marketplace/operators.coreos.com/catalogsources
must-gather-olm/namespaces/openshift-marketplace/operators.coreos.com/catalogsources
├── certified-operators.yaml
├── community-operators.yaml
├── redhat-marketplace.yaml
└── redhat-operators.yaml

Questions

what is the best way to use most recent must-gathers in different Y stream?
Why OLM is not being collected running latest must-gather images?

I have already had the same issue described on #230 in my local development, and the olm script gather_olm is using the option --dest-dir and raising errors due to old cli.

network-plugin-specific gathering

#121 removed the gathering of some openshift-sdn-specific resources, since it broke must-gather under ovn-kubernetes.

It would be good if there was some way we could gather the HostSubnet objects under openshift-sdn, but not error out under ovn-kubernetes (or calico, or ...). Likewise, in the future, we will probably want to gather some ovn-kubernetes-specific data when using ovn-kubernetes (although that may involve running commands to get data out of ovn databases (via oc exec to the ovn pods?) rather than gathering additional API objects).

Obviously it's possible to do this with arbitrary shell-scripting, but what's the architecturally-correct way to do it? (We'd want this to be part of the default "oc adm must-gather" output.)

Question about creating my own image with custom scripts

I am interested to create an image with custom scripts to collect info from an OpenShift cluster. How can I get access to the doc mentioned below to get more familiar with this tool:

The directory structure, as well as specific details behind this tool can be found in this doc.

I requested access to the doc but my request hasn't been approved.

Run must-gather in 'merge-results' mode

When running must-gather via 'oc adm must-gather --image=...' , a new result folder is created.
Lots of static information (NS info, CRD's, Deployments, etc) is duplicated.
Need a way to runwith merge-results mode to have only the delta/aditional info.

Or an option to run must-gather with 'logs/events' mode - to produce only pod logs and events to be merged into a pre populated result folder.

must-gather log gathering window should not be infinite by default

Wrong repo :)

`must-gather` frequently fails on UPI vSphere tests

See https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_release/3593/rehearse-3593-pull-ci-openshift-installer-master-e2e-vsphere/14:

fail [github.com/openshift/origin/test/extended/cli/mustgather.go:73]: Unexpected error:
    <*errors.errorString | 0xc003384320>: {
        s: "expected files should not be empty: /tmp/test.oc-adm-must-gather.270681703/audit_logs/openshift-apiserver.audit_logs_listing",
    }
    expected files should not be empty: /tmp/test.oc-adm-must-gather.270681703/audit_logs/openshift-apiserver.audit_logs_listing
occurred

Must_Gather pod able to run on SchedulingDisabled node

I recently noticed that when I executed must-gather, the pod got scheduled on master node which seems quite expected in the first place since I disabled scheduling for the remaining two workers.

Later, I provided the node name while executing the must-gather, and it was scheduled on the worker node even though the worker was marked SchedulingDisabled.
e.g $ oc adm must-gather --node-name=$worker

Then, I checked # oc describe node output and could not locate the must-gather pod. Also, the pod's scheduler appears to be default-scheduler due to which I felt that it could have prevented pod scheduling on the SchedulingDisabled node.

Tested on the following version:

oc version

Client Version: 4.7.0-202105281408.p0.git.95881af-95881af
Server Version: 4.7.14
Kubernetes Version: v1.20.0+a0b09eb

Did I miss anything?
Is this behavior expected w.r.t design of must-gather?

must-gather takes too long to run

As part of our tekton (openshift pipeline) we have a must-gather task that runs if there are test case failures. Currently this task runs the following command oc adm must-gather --image=quay.io/opencloudio/must-gather:4.5.1 -- gather -n ibm-common-services,<namespace where product under test is installed to>. This command takes about 30 minutes to complete which greatly extends the completion time of our pipeline runs.

According to the article below by default the following modules are run: overview, system, failure, ocp, cloudpak. I was hoping by only running one mode(failure) it would run faster but it was only one minute faster.

https://www.ibm.com/support/knowledgecenter/SSHKN6/support/must-gather/1.x.x/overview.html

I ran oc adm must-gather and it took 27 minutes to complete
I ran oc adm must-gather -- gather -m failure and it took 26 minutes to complete

Are there any ways to shorten the time that must-gather takes to run?

Add the leases.coordination.k8s.io as one of the collected resources

The leases resource is mainly used for Node heartbeats and Leader election,
which both have a key role in some mechanisms of different operators, for exmple in the sriov-operator.
it makes sense to make ocp collect this as part of the must-gather.

must-gather does not collect full ImageStreamTags data

There is a big difference in ImageStreamTags items collected by must-gather and manual running of this command: oc get istag/python:latest -o yaml -n openshift

please take a look on attached files

must-gather_python.txt
direct_command_python.txt

In our case we need to have "release" and "version" labels to be collected

Errors don't align with reason for failure

Example:

$ bin/openshift-must-gather info clusteroperators
Error: the server doesn't have a resource type "clusteroperators"

is caused because the user's kubeconfig are not set and/or the active context is not authenticated with the cluster.

We should provide a better error and/or things to try (like logging in) if something like this happens.

Rysnc related errors

Getting below errors , while using in a IPI GCP private cluster env

.
.
rsync: connection unexpectedly closed (122395578 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [receiver=3.1.3]
rsync: connection unexpectedly closed (70267 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [generator=3.1.3]
.
.

Data collection is trying to use non-accessible host names.

Error: One or more errors ocurred gathering cluster data:

[one or more errors ocurred while gathering pod-specific data for namespace: openshift-apiserver-operator

one or more errors ocurred while gathering container data for pod openshift-apiserver-operator-77756dfc66-v57tb:

Get https://ip-10-0-4-202:10250/containerLogs/openshift-apiserver-operator/openshift-apiserver-operator-77756dfc66-v57tb/operator?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering pod-specific data for namespace: openshift-apiserver

[one or more errors ocurred while gathering container data for pod apiserver-bmfmt:

Get https://ip-10-0-4-202:10250/containerLogs/openshift-apiserver/apiserver-bmfmt/openshift-apiserver?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering container data for pod apiserver-q2wmq:

Get https://ip-10-0-17-67:10250/containerLogs/openshift-apiserver/apiserver-q2wmq/openshift-apiserver?timestamps=true: dial tcp: lookup ip-10-0-17-67 on 10.0.0.2:53: no such host], one or more errors ocurred while gathering pod-specific data for namespace: openshift-cluster-openshift-controller-manager-operator

one or more errors ocurred while gathering container data for pod openshift-cluster-openshift-controller-manager-operator-5dhl2xg:

Get https://ip-10-0-4-202:10250/containerLogs/openshift-cluster-openshift-controller-manager-operator/openshift-cluster-openshift-controller-manager-operator-5dhl2xg/operator?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering pod-specific data for namespace: openshift-cluster-samples-operator

one or more errors ocurred while gathering container data for pod cluster-samples-operator-7cf8d8cf66-rmdd5:

Get https://ip-10-0-4-202:10250/containerLogs/openshift-cluster-samples-operator/cluster-samples-operator-7cf8d8cf66-rmdd5/cluster-samples-operator?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering pod-specific data for namespace: openshift-dns

[one or more errors ocurred while gathering container data for pod dns-default-lklv8:

[Get https://ip-10-0-149-151:10250/containerLogs/openshift-dns/dns-default-lklv8/dns?timestamps=true: dial tcp: lookup ip-10-0-149-151 on 10.0.0.2:53: no such host, Get https://ip-10-0-149-151:10250/containerLogs/openshift-dns/dns-default-lklv8/dns-node-resolver?timestamps=true: dial tcp: lookup ip-10-0-149-151 on 10.0.0.2:53: no such host], one or more errors ocurred while gathering container data for pod dns-default-lplf9:

[Get https://ip-10-0-143-162:10250/containerLogs/openshift-dns/dns-default-lplf9/dns?timestamps=true: dial tcp: lookup ip-10-0-143-162 on 10.0.0.2:53: no such host, Get https://ip-10-0-143-162:10250/containerLogs/openshift-dns/dns-default-lplf9/dns-node-resolver?timestamps=true: dial tcp: lookup ip-10-0-143-162 on 10.0.0.2:53: no such host], one or more errors ocurred while gathering container data for pod dns-default-ltzhd:

[Get https://ip-10-0-4-202:10250/containerLogs/openshift-dns/dns-default-ltzhd/dns?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host, Get https://ip-10-0-4-202:10250/containerLogs/openshift-dns/dns-default-ltzhd/dns-node-resolver?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host], one or more errors ocurred while gathering container data for pod dns-default-mnq2w:

[Get https://ip-10-0-17-67:10250/containerLogs/openshift-dns/dns-default-mnq2w/dns?timestamps=true: dial tcp: lookup ip-10-0-17-67 on 10.0.0.2:53: no such host, Get https://ip-10-0-17-67:10250/containerLogs/openshift-dns/dns-default-mnq2w/dns-node-resolver?timestamps=true: dial tcp: lookup ip-10-0-17-67 on 10.0.0.2:53: no such host], one or more errors ocurred while gathering container data for pod dns-default-qcblt:

[Get https://ip-10-0-175-136:10250/containerLogs/openshift-dns/dns-default-qcblt/dns?timestamps=true: dial tcp: lookup ip-10-0-175-136 on 10.0.0.2:53: no such host, Get https://ip-10-0-175-136:10250/containerLogs/openshift-dns/dns-default-qcblt/dns-node-resolver?timestamps=true: dial tcp: lookup ip-10-0-175-136 on 10.0.0.2:53: no such host]], one or more errors ocurred while gathering pod-specific data for namespace: openshift-ingress-operator

one or more errors ocurred while gathering container data for pod ingress-operator-794cc7b555-62g87:

Get https://ip-10-0-175-136:10250/containerLogs/openshift-ingress-operator/ingress-operator-794cc7b555-62g87/ingress-operator?timestamps=true: dial tcp: lookup ip-10-0-175-136 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering pod-specific data for namespace: openshift-ingress

one or more errors ocurred while gathering container data for pod router-default-766766d8b6-zdkj9:

Get https://ip-10-0-175-136:10250/containerLogs/openshift-ingress/router-default-766766d8b6-zdkj9/router?timestamps=true: dial tcp: lookup ip-10-0-175-136 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering pod-specific data for namespace: openshift-kube-apiserver-operator

one or more errors ocurred while gathering container data for pod openshift-kube-apiserver-operator-58b8c455c5-p9drx:

: error dialing backend: dial tcp: lookup ip-10-0-38-26 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering pod-specific data for namespace: openshift-kube-apiserver

[one or more errors ocurred while gathering container data for pod openshift-kube-apiserver-ip-10-0-17-67.ec2.internal:

Get https://ip-10-0-17-67:10250/containerLogs/openshift-kube-apiserver/openshift-kube-apiserver-ip-10-0-17-67.ec2.internal/openshift-kube-apiserver?timestamps=true: dial tcp: lookup ip-10-0-17-67 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering container data for pod openshift-kube-apiserver-ip-10-0-4-202.ec2.internal:

Get https://ip-10-0-4-202:10250/containerLogs/openshift-kube-apiserver/openshift-kube-apiserver-ip-10-0-4-202.ec2.internal/openshift-kube-apiserver?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host], one or more errors ocurred while gathering pod-specific data for namespace: openshift-kube-controller-manager-operator

one or more errors ocurred while gathering container data for pod openshift-kube-controller-manager-operator-66cbb459f8-8lhcj:

Get https://ip-10-0-4-202:10250/containerLogs/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator-66cbb459f8-8lhcj/operator?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering pod-specific data for namespace: openshift-kube-controller-manager

[one or more errors ocurred while gathering container data for pod openshift-kube-controller-manager-ip-10-0-17-67.ec2.internal:

Get https://ip-10-0-17-67:10250/containerLogs/openshift-kube-controller-manager/openshift-kube-controller-manager-ip-10-0-17-67.ec2.internal/kube-controller-manager?timestamps=true: dial tcp: lookup ip-10-0-17-67 on 10.0.0.2:53: no such host, one or more errors ocurred while gathering container data for pod openshift-kube-controller-manager-ip-10-0-4-202.ec2.internal:

Get https://ip-10-0-4-202:10250/containerLogs/openshift-kube-controller-manager/openshift-kube-controller-manager-ip-10-0-4-202.ec2.internal/kube-controller-manager?timestamps=true: dial tcp: lookup ip-10-0-4-202 on 10.0.0.2:53: no such host]]

[RHOCP][4.6] Include missing Core namespace

Hi guys,

Not sure if you aware of it, but since RHOCP4.6, some new namespaces have been added to the core, such as openshift-*-infra, openshift-oauth-apiserver.
And as those namespaces are now part of the RHOCP clusters, we need them as part of the must-gather to diagnose Red Hat Customers' issues.
I've been facing cases where I was looking at the must-gather, and then I had to request the inspect ns/.... for the missing namespace.

Could you please add the missing namespaces to the must-gather gather list?

Cheers,

Emit highlevel copy/pasteable summary information

I'd like for the console output of the oc adm must-gather command to include copy / pastable summary of the top pieces of data necessary. These should be the key pieces of information that we feel every single bug or case must include for highest level triage. Something like the following.

Output in must-gather-xyz.tar.gz

When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information.
Cluster Version: 4.5.11
Platform: AWS
Degraded Operators: openshift-rocketship

include CustomResourceDefinition in must-gather

As part of our tekton (openshift pipeline) we have a must-gather task that runs if there are test case failures. Currently this task runs the following command oc adm must-gather --image=quay.io/opencloudio/must-gather:4.5.1 -- gather -n ibm-common-services,<namespace where product under test is installed to>. We had a developer open up an issue against us saying certain kinds from CustomResourceDefinitions were not included in the must-gather zip. Is there a way to have these included?

How to collecte the info in namespaces gpu-operator-resources to Must-gather archive

Get the pods' log in project gpu-operator-resources

for i in `oc get  pods -n openshift-monitoring --no-headers=true | awk '{print $1}'`; 
do 
oc logs $i -n openshift-monitoring --all-containers=true
done

must-gather.log spams logs about egressfirewalls and egressqoses

Looking at the must-gather.log from my latest ci-test run, it seems like we throw INFO logs which make the log harder to read. See:

[must-gather-xxbbf] POD 2023-05-17T11:59:17.152066356Z Gathering data for ns/openshift-cluster-version...
[must-gather-xxbbf] POD 2023-05-17T11:59:19.555676602Z W0517 11:59:19.555610       8 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:19.558880211Z W0517 11:59:19.558851       8 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:19.663594946Z Gathering data for ns/default...
[must-gather-xxbbf] POD 2023-05-17T11:59:19.796052158Z W0517 11:59:19.796007       8 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:19.802501693Z W0517 11:59:19.802477       8 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:19.933839951Z Gathering data for ns/openshift...
[must-gather-xxbbf] POD 2023-05-17T11:59:20.179844728Z W0517 11:59:20.179806       8 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:20.182575941Z W0517 11:59:20.182553       8 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:20.279559456Z Gathering data for ns/kube-system...
[must-gather-xxbbf] POD 2023-05-17T11:59:20.408955535Z W0517 11:59:20.408913       8 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:20.411238487Z W0517 11:59:20.411218       8 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:20.678549529Z Gathering data for ns/openshift-etcd...
[must-gather-xxbbf] POD 2023-05-17T11:59:21.679718696Z W0517 11:59:21.679675       8 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:21.683752225Z W0517 11:59:21.683726       8 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:22.060183522Z Wrote inspect data to must-gather.
[must-gather-xxbbf] POD 2023-05-17T11:59:23.790812585Z Gathering data for ns/openshift-config...
[must-gather-xxbbf] POD 2023-05-17T11:59:23.928293099Z W0517 11:59:23.928253     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:23.932897522Z W0517 11:59:23.932869     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:24.044526544Z Gathering data for ns/openshift-config-managed...
[must-gather-xxbbf] POD 2023-05-17T11:59:24.313978398Z W0517 11:59:24.313931     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:24.319885490Z W0517 11:59:24.319850     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:24.411760399Z Gathering data for ns/openshift-authentication...
[must-gather-xxbbf] POD 2023-05-17T11:59:24.699325865Z W0517 11:59:24.699277     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:24.702712681Z W0517 11:59:24.702687     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:24.886439798Z Gathering data for ns/openshift-authentication-operator...
[must-gather-xxbbf] POD 2023-05-17T11:59:25.100911831Z W0517 11:59:25.100877     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:25.106497028Z W0517 11:59:25.106460     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
.
.
.
[must-gather-xxbbf] POD 2023-05-17T12:00:06.109510622Z Gathering data for ns/openshift-machine-config-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:06.616004749Z W0517 12:00:06.615962     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:06.619411887Z W0517 12:00:06.619376     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:07.024611216Z Gathering data for ns/openshift-kni-infra...
[must-gather-xxbbf] POD 2023-05-17T12:00:12.765883020Z W0517 12:00:12.765836     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:12.769646923Z W0517 12:00:12.769622     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:12.847601824Z Gathering data for ns/openshift-openstack-infra...
[must-gather-xxbbf] POD 2023-05-17T12:00:12.959084289Z W0517 12:00:12.959043     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:12.961865990Z W0517 12:00:12.961835     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.044327483Z Gathering data for ns/openshift-ovirt-infra...
[must-gather-xxbbf] POD 2023-05-17T12:00:13.165554607Z W0517 12:00:13.165507     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.168633601Z W0517 12:00:13.168601     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.255017243Z Gathering data for ns/openshift-vsphere-infra...
[must-gather-xxbbf] POD 2023-05-17T12:00:13.384294288Z W0517 12:00:13.384242     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.387208878Z W0517 12:00:13.387171     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.479537846Z Gathering data for ns/openshift-nutanix-infra...
[must-gather-xxbbf] POD 2023-05-17T12:00:13.619418134Z W0517 12:00:13.619377     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.635433684Z W0517 12:00:13.635400     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.736145710Z Gathering data for ns/openshift-marketplace...
[must-gather-xxbbf] POD 2023-05-17T12:00:14.031202793Z W0517 12:00:14.031171     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:14.034909709Z W0517 12:00:14.034879     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:14.237141996Z Gathering data for ns/openshift-user-workload-monitoring...
[must-gather-xxbbf] POD 2023-05-17T12:00:14.363284391Z W0517 12:00:14.363225     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:14.367420112Z W0517 12:00:14.367390     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:14.980714662Z Gathering data for ns/openshift-sdn...
[must-gather-xxbbf] POD 2023-05-17T12:00:15.717496223Z W0517 12:00:15.717454     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:15.720279047Z W0517 12:00:15.720254     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:16.021123774Z Gathering data for ns/openshift-host-network...
[must-gather-xxbbf] POD 2023-05-17T12:00:16.150152499Z W0517 12:00:16.150110     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:16.153603423Z W0517 12:00:16.153575     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:16.255697194Z Gathering data for ns/openshift-network-diagnostics...
[must-gather-xxbbf] POD 2023-05-17T12:00:16.513744169Z W0517 12:00:16.513703     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:16.516812516Z W0517 12:00:16.516630     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:16.742676795Z Gathering data for ns/openshift-network-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:17.719904200Z W0517 12:00:17.719867     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:17.722734390Z W0517 12:00:17.722701     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:17.856745666Z Gathering data for ns/openshift-cloud-network-config-controller...
[must-gather-xxbbf] POD 2023-05-17T12:00:17.971103700Z W0517 12:00:17.971046     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:17.977205771Z W0517 12:00:17.977095     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:18.120809852Z Gathering data for ns/openshift-apiserver-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:18.333571325Z W0517 12:00:18.333524     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:18.343028392Z W0517 12:00:18.342996     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:18.550916367Z Gathering data for ns/openshift-apiserver...
[must-gather-xxbbf] POD 2023-05-17T12:00:18.962479751Z W0517 12:00:18.962435     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:18.966718088Z W0517 12:00:18.966688     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:19.337594227Z Gathering data for ns/openshift-controller-manager-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:19.636247453Z W0517 12:00:19.636200     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:19.639258446Z W0517 12:00:19.639224     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:19.772732552Z Gathering data for ns/openshift-controller-manager...
[must-gather-xxbbf] POD 2023-05-17T12:00:20.012199454Z W0517 12:00:20.012149     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:20.016432685Z W0517 12:00:20.016389     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:20.168098552Z Gathering data for ns/openshift-route-controller-manager...
[must-gather-xxbbf] POD 2023-05-17T12:00:20.359217322Z W0517 12:00:20.359170     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:20.362151005Z W0517 12:00:20.362123     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:20.515942559Z Gathering data for ns/openshift-cluster-samples-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:20.740834006Z W0517 12:00:20.740216     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:20.753593652Z W0517 12:00:20.753547     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:21.223975051Z Gathering data for ns/openshift-operator-lifecycle-manager...
[must-gather-xxbbf] POD 2023-05-17T12:00:21.619559658Z W0517 12:00:21.619190     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:21.625162052Z W0517 12:00:21.625135     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:21.829182721Z Gathering data for ns/openshift-service-ca-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:21.975115463Z W0517 12:00:21.975067     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:21.981283096Z W0517 12:00:21.981246     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:22.086810692Z Gathering data for ns/openshift-service-ca...
[must-gather-xxbbf] POD 2023-05-17T12:00:22.241755336Z W0517 12:00:22.241703     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:22.246609074Z W0517 12:00:22.246575     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:22.352120347Z Gathering data for ns/openshift-cluster-csi-drivers...
[must-gather-xxbbf] POD 2023-05-17T12:00:22.454613572Z W0517 12:00:22.454565     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:22.457398208Z W0517 12:00:22.457362     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection

I file this issue to suggest muting the the server doesn't have.. logs to provide a cleaner log output for the must-gather.

Must-gather image does not support multi-arch

Currently I have made a custom must gather image at https://github.com/appsody/appsody-operator/tree/master/must-gather but the image is not supported on multiple architectures. It runs on linux but not linux-ppc64 or linux-s390. Please consider adding support for these architectures or insight on how to change the dockerfile at https://github.com/appsody/appsody-operator/blob/master/must-gather/Dockerfile.

Move config.openshift.io out to its own plugin

We should restructure the commands / plugins so that config.openshift.io is its own plugin.

Broken link in README

In the README, the link to the enhancement is broken.

The directory structure, as well as specific details behind this tool can be found in this enhancement.

must-gather hangs when given an invalid image given to --image

When I try to call must-gather with an invalid image (i.e. any image that causes the must-gather pod to have the status Init:InvalidImageName) the command doesn't give any error and does not terminate.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.