How to gather logs and data for OpenShift Virtualization?

Solution Verified - Updated

Environment

  • Red Hat OpenShift Virtualization
  • Red Hat OpenShift Container Platform (RHOCP)

Issue

  • How to gather all the necessary logs to debug OpenShift Virtualization issues?
  • What logs should be provided when opening an OpenShift Virtualization support case?

Resolution

Automatically gathering the logs information

To help troubleshoot and debug OpenShift Virtualization issues, gathering the relevant logs is particularly important. Before starting to troubleshoot or opening a support case, the easiest and best way to gather the necessary logs is to use the must-gather tool provided by OpenShift.

OpenShift Virtualization provide its own must-gather image to collect specific information. To gather both OpenShift and OpenShift Virtualization logs at the same time (recommended), use the following command:

$ oc adm must-gather --image-stream=openshift/must-gather --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v$(oc get csv -n openshift-cnv -l "operators.coreos.com/kubevirt-hyperconverged.openshift-cnv=" -o=jsonpath='{.items[*].spec.version}')

$ tar -zvcf must-gather.tar.gz must-gather.local.[random-ID]/

It is also possible to only gather the specific OpenShift Virtualization logs (excluding the OpenShift logs) using the following command :

$ oc adm must-gather --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v$(oc get csv -n openshift-cnv -l "operators.coreos.com/kubevirt-hyperconverged.openshift-cnv=" -o=jsonpath='{.items[*].spec.version}')

Note: for OpenShift Virtualization versions older than 4.13, the image should be cnv-must-gather-rhel8.

For collecting data about specific VMs, please refer to collecting data about virtual machines, and also an inspect of the namespace in which the virtual machine(s) are running:

$ oc adm inspect ns/openshift-cnv ns/[vm_namespace]

$ tar -zvcf must-gather.tar.gz must-gather.local.[random-ID]/

Manually gathering the logs information

Sometimes, the must-gather tool could experience some issues gathering the relevant logs (cluster not healthy, timeout, etc). It is also useful to know how to gather specific logs information depending on what needs to be troubleshooted.

The main namespace for OpenShift Virtualization is the openshift-cnv namespace. To gather the information about this namespace, use the following command :

$ oc adm inspect ns/openshift-cnv

However, other information are useful when debugging an OpenShift Virtualization issue, so it might be worth to also include the output of the following commands:

  • Other OpenShift Virtualization relevant namespaces:

    $ oc adm inspect ns/cdi ns/kubevirt-hyperconverged ns/openshift-operator-lifecycle-manager ns/openshift-marketplace ns/cluster-network-addons ns/openshift-sdn ns/sriov-network-operator ns/kubevirt-web-ui

  • For OpenShift information :

    $ oc adm inspect -A clusterversion,clusteroperators,catalogsource,clusterserviceversion,nodes,machines

  • For Operators information :

    $ oc adm inspect -A olm

  • For Virtual Machines information :

    $ oc adm inspect -A virtualmachineinstances,virtualmachineinstancereplicasets,virtualmachineinstancepresets,virtualmachineinstancemigrations,v2vvmwares,virtualmachineimports

  • For storage information :

    $ oc adm inspect -A storageclasses,persistentvolumes,datavolumes

  • For network information :

    $ oc adm inspect -A networks.operator.openshift.io,nodenetworkstates,nodenetworkconfigurationenactments,nodenetworkconfigurationpolicies,Network-attachment-definitions,networkaddonsconfig

Root Cause

To help troubleshoot and debug OpenShift Virtualization issues, gathering the relevant logs is particularly important. Using the OpenShift Virtualization must-gather image is the recommended way to collect most of the relevant data. If VMs are failing, collecting information from the namespace in which the VMs are running will be also needed.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.