How to collect the must-gather for Red Hat OpenShift AI?

Solution Verified - Updated

Environment

  • Red Hat OpenShift AI (RHOAI) Self-Managed
    • 2.24.x
    • 2.25.0
    • 3.0

Issue

  • How to collect the Must-Gather for Red Hat OpenShift AI?
  • What data is needed to toubleshoot Red Hat OpenShift AI issues?
  • What data is needed to toubleshoot Red Hat OpenShift AI Installation failure issues?
  • What data is needed to toubleshoot Red Hat OpenShift AI Upgrade failure issues?

Resolution

Prerequisites

  • cluster-admin access to the Red Hat OpenShift cluster.
  • oc Command Line Interface (CLI) tool

Usage
To initiate the must-gather process, execute the following commands

  • For v2.23 and earlier versions:
# oc adm must-gather --image=quay.io/modh/must-gather:rhoai-x.y

With x being the major release version and y the minor one. To verify the exact available tags (which use the rhoai- prefix), check the repository tags: This content is not included.quay.io/modh/must-gather.

For example, in order to run must gather against 2.22 release execute the following command:

# oc adm must-gather --image=quay.io/modh/must-gather:rhoai-2.22
  • For v2.24 and later versions:
# oc adm must-gather --image=registry.redhat.io/rhoai/odh-must-gather-rhel9:vX.Y

With X being the major release version and Y the minor one.

For example, in order to run must gather against 2.25 release execute the following command:

# oc adm must-gather --image=registry.redhat.io/rhoai/odh-must-gather-rhel9:v2.25
  • For v3.0 and later versions:
# oc adm must-gather --image=registry.redhat.io/rhoai/odh-must-gather-rhel9:v3.x

Example for RHOAI v3.0 you can run command.

# oc adm must-gather --image=registry.redhat.io/rhoai/odh-must-gather-rhel9:v3.0

Where is must-gather collected ?

When opening a support case please include the summary printed by must-gather command. Also, make sure to upload the must-gather.* directory which includes all the gathered information for the cluster in investigation.

  • By default the must gather is collected in current working directory on the system where command is executed. 

    # ls -l | grep must-gather

  • If destination directory is mentioned, the must gather will be collected under the respective directory. 

    # oc adm must-gather --image=registry.redhat.io/rhoai/odh-must-gather-rhel9:vX.Y --dest-dir=/local/directory

Tag Version

  • To display available tag versions from registry.redhat.io, run below command.

    # skopeo list-tags docker://registry.redhat.io/rhoai/odh-must-gather-rhel9

Diagnostic Steps

Data Collected

  • The must-gather script focuses on collecting data from the following namespaces by default:

    1. redhat-ods-operator
    2. rhods-notebooks
    3. redhat-ods-applications
    4. redhat-ods-monitoring
    5. rhoai-model-registries

  • Additionally, it gathers data from any namespace containing its related resources.

  • For more information how to use must-gather image for specific RHOAI release, it can be read in the project README.md file

  • See https://github.com/red-hat-data-services/must-gather/blob/rhoai-X.Y/README.md. (replace X.Y in URL with RHOAI version)

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.