Collect must-gather with more details for specific components in OpenShift

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Must Gather

Issue

  • How to generate a must-gather report with more details for specific use cases.

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Refer to gathering data about specific features for additional information. In case the standard must-gather collection fails, please refer to data gathering when oc adm must-gather fails.

Data Collection based on time frame

It could be required in case the size of the must-gather is extremely big (note that must-gathers of few GB are usually OK). Refer to Is there any way to reduce the size of must-gathers? for additional information.

Data Collection Audit logs

Run must-gather to capture audit logs:

$ oc adm must-gather -- /usr/bin/gather_audit_logs

It is also possible to generate the default must-gather including the audit logs:

$ oc adm must-gather -- '/usr/bin/gather && /usr/bin/gather_audit_logs'

Also refer to how to collect audit logs from master nodes in Red Hat OpenShift Container Platform 4

Data Collection for Openshift Container Native Virtualization (CNV)

For OCP-4.12 (Based on RHEL8)

$ oc adm must-gather --image-stream=openshift/must-gather --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel8:v$(oc get csv -n openshift-cnv -l "operators.coreos.com/kubevirt-hyperconverged.openshift-cnv=" -o=jsonpath='{.items[*].spec.version}')

For OCP-4.13 and above (Rebase to RHEL9)

$ oc adm must-gather --image-stream=openshift/must-gather --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v$(oc get csv -n openshift-cnv -l "operators.coreos.com/kubevirt-hyperconverged.openshift-cnv=" -o=jsonpath='{.items[*].spec.version}')

Data Collection for Red Hat OpenShift GitOps

To gather more details for Red Hat OpenShift GitOps:

$ oc adm must-gather --image="registry.redhat.io/openshift-gitops-1/must-gather-rhel8:v$(oc get csv -A -l operators.coreos.com/openshift-gitops-operator.openshift-gitops-operator= -o jsonpath='{.items[*].spec.version}')"

Refer to collecting debugging data for Red Hat OpenShift GitOps for additional data.

Data Collection for Red Hat OpenShift Logging

To gather more details about OpenShift Logging:

$ oc adm must-gather --image=$(oc -n openshift-logging get deployment.apps/cluster-logging-operator -o jsonpath='{.spec.template.spec.containers[?(@.name == "cluster-logging-operator")].image}')

For Logging 5, refer to collecting cluster logging data. For Logging 6, refer to collecting logging data. Refer also to the upstream documentation (Content from github.com is not included.cluster-logging must-gather) for additional information.

Data Collection for Red Hat OpenShift AI

Refer to must-gather for Red Hat OpenShift AI.

Data Collection for OpenShift Networking (OpenShiftSDN and OVN Kubernetes)

To gather network data run this must-gather collection script (the network data is already collected by standard must-gather in versions 4.12 and 4.13):

$ oc adm must-gather -- /usr/bin/gather_network_logs

Refer to How to collect network logs from an OpenShift 4 cluster via must-gather? for additional information.

Data Collection for the OpenShift SR-IOV operator

For the SR-IOV operator, please refer to Providing data for troubleshooting and analyzing the SR-IOV Operator in Red Hat OpenShift Container Platform 4

Data Collection for OpenShift Serverless

Data collection for OpenShift Serverless.

$ oc adm must-gather --image-stream=openshift/must-gather --image=registry.redhat.io/openshift-serverless-1/serverless-must-gather-rhel8:[serverless_version]

Refer to about collecting OpenShift Serverless data for additional information.

Data Collection for Red Hat OpenShift Service Mesh

Data collection for Red Hat OpenShift Service Mesh.

$ oc adm must-gather --image-stream=openshift/must-gather --image=registry.redhat.io/openshift-service-mesh/istio-must-gather-rhel8:[servicemesh_version]

If the Service Mesh control plane is in a different namespace, add gather [namespace_name] at the end of the above command.
See about collecting service mesh data for additional information.

Data Collection for Migration Toolkit for Containers (MTC)

Data collection for migration-related workflow:

$ oc adm must-gather --image-stream=openshift/must-gather --image=registry.redhat.io/rhmtc/openshift-migration-must-gather-rhel8:[MTC_version]

The [MTC_version] tag should be in format v1.y. See Migration Toolkit for Containers - Using must-gather tool to collect data for additional information.

Data Collection for Red Hat OpenShift Data Foundation (ODF)

Refer to how to collect ODF must-gather data.

OpenShift APIs for Data Protection (OADP)

OpenShift APIs for Data Protection.

Regular OADP must-gather

oc adm must-gather --image=registry.redhat.io/oadp/oadp-mustgather-rhel8:[version]

OADP must-gather with metrics dump

oc adm must-gather --image=registry.redhat.io/oadp/oadp-mustgather-rhel8:[version] -- /usr/bin/gather_metrics_dump

Note: For the OADP must-gather, it needs to change the "[Version]" with the correct OADP version (e.g.: v1.1) in the above commands. Refer to the OADP documentation for using the must-gather tool.

Data collection for Local Storage Operator (LSO)

For OCP-4.14 or previous (Based on RHEL8)

oc adm must-gather --image-stream=openshift/must-gather --image=registry.redhat.io/openshift4/ose-local-storage-mustgather-rhel8:v$(oc get csv -A -l "operators.coreos.com/local-storage-operator.openshift-local-storage=" -o=jsonpath='{.items[*].spec.version}'|awk -F- '{print $1}')

For OCP-4.15 and above (Rebase to RHEL9)

oc adm must-gather --image-stream=openshift/must-gather --image=registry.redhat.io/openshift4/ose-local-storage-mustgather-rhel9:v$(oc get csv -A -l "operators.coreos.com/local-storage-operator.openshift-local-storage=" -o=jsonpath='{.items[*].spec.version}'|awk -F- '{print $1}')

Data Collection for Red Hat Advanced Cluster Management for Kubernetes (RHACM)

Refer to what information gather for troubleshooting Red Hat Advanced Cluster Management (RHACM)?.

Data Collection for Hosted Control Planes (HCP)

Refer to troubleshooting OpenShift Hosted Control Planes.

Data Collection for Red Hat Quay

Refer to describe the Quay environment and gather support data.

Gathering Data for openshift-compliance operator

$ oc adm must-gather --image=quay.io/compliance-operator/must-gather:latest

NOTE: There's a new image available for compliance-operator must-gather with more improvements, the image is: ghcr.io/complianceascode/must-gather-ocp and soon in the upcoming release of compliance operator (tentatively CO v1.6.0) this image would be available in downstream, so users would be able to consume it for disconnected environments as well.

Data Collection for the Performance Addon Operator

The latest tag will not work (at the time of this writing):

[root@openshift-jumpserver-0 ~]# oc adm must-gather --image=registry.redhat.io/openshift4/performance-addon-operator-must-gather-rhel8:latest
[must-gather      ] OUT Using must-gather plugin-in image: registry.redhat.io/openshift4/performance-addon-operator-must-gather-rhel8:latest
[must-gather      ] OUT namespace/openshift-must-gather-mfxtk created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-jpv6b created
[must-gather      ] OUT pod for plug-in image registry.redhat.io/openshift4/performance-addon-operator-must-gather-rhel8:latest created
[must-gather-qb5cq] OUT gather did not start: unable to pull image: ErrImagePull: rpc error: code = Unknown desc = Error reading manifest latest in registry.redhat.io/openshift4/performance-addon-operator-must-gather-rhel8: unsupported: This repository does not use the "latest" tag to track the most recent image and must be pulled with an explicit version or image reference. For more information, see: https://access.redhat.com/articles/4301321
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-jpv6b deleted
[must-gather      ] OUT namespace/openshift-must-gather-mfxtk deleted
error: gather did not start for pod must-gather-qb5cq: unable to pull image: ErrImagePull: rpc error: code = Unknown desc = Error reading manifest latest in registry.redhat.io/openshift4/performance-addon-operator-must-gather-rhel8: unsupported: This repository does not use the "latest" tag to track the most recent image and must be pulled with an explicit version or image reference. For more information, see: https://access.redhat.com/articles/4301321
[root@openshift-jumpserver-0 ~]#

List available tags for the Performance Addon Operator must-gather image instead:

[root@openshift-jumpserver-0 ~]# skopeo list-tags docker://registry.redhat.io/openshift4/performance-addon-operator-must-gather-rhel8
{
    "Repository": "registry.redhat.io/openshift4/performance-addon-operator-must-gather-rhel8",
    "Tags": [
        "v4.5.1",
        "v4.5.0",
        "v4.4.0-6",
        "v4.5.1-2",
        "v4.4.0",
        "v4.4.1",
        "v4.4",
        "v4.5",
        "v4.5.0-14",
        "v4.4.1-2",
        "latest"
    ]
}

Gather only data for the Performance Addon Operator (mostly to test data collection for the Performance Addon Operator, not very useful for gathering troubleshooting data):

oc adm must-gather --image=registry.redhat.io/openshift4/performance-addon-operator-must-gather-rhel8:v4.5.1

Gather data for additional components alongside the Performance Addon Operator (preferred option):

oc adm must-gather --image-stream=openshift/must-gather --image=registry.redhat.io/openshift4/performance-addon-operator-must-gather-rhel8:v4.5.1

Gathering additional data for multiple components

The --image and --image-stream parameters are cumulative, for example, one might run the following to gather data from both OpenShift Container Native Virtualization and the Performance Addon Operator, but nothing further:

oc adm must-gather --image=registry.redhat.io/openshift4/performance-addon-operator-must-gather-rhel8:v4.5.1  --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel8
[must-gather      ] OUT Using must-gather plugin-in image: registry.redhat.io/openshift4/performance-addon-operator-must-gather-rhel8:v4.5.1, registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel8
[must-gather      ] OUT namespace/openshift-must-gather-cvsh6 created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-mrxjb created
[must-gather      ] OUT pod for plug-in image registry.redhat.io/openshift4/performance-addon-operator-must-gather-rhel8:v4.5.1 created
[must-gather      ] OUT pod for plug-in image registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel8 created
(...)

[root@openshift-jumpserver-0 ~]# ls -l must-gather.local.581199523937468209
total 0
drwxrwxrwx. 7 root root 104 Oct 15 10:18 registry-redhat-io-container-native-virtualization-cnv-must-gather-rhel8-sha256-36c2ba9c269e78081062ce73703cbc4ca51cbfbe77c1e8ab7b36722704328d1d
drwxrwxrwx. 5 root root  84 Oct 15 10:15 registry-redhat-io-openshift4-performance-addon-operator-must-gather-rhel8-sha256-11bed9cb86fa9098f7df5cd3d103bd5e531f6cb1e1355dd8e2b18981cf1fc81a

An operator can add further --image-stream or --image parameters to gather even more data.

Root Cause

The must-gather sub-command can use different images or scripts inside the images to gather different or additional data. For example, using --image-stream=openshift/must-gather --image=[operator_must-gather_image] will collect the default must-gather plus the operator specific must-gather in the same directory.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.