How to get serial console logs for VMs part of OpenShift Virtualization Hosted Control Plane clusters

Environment

• Red Hat OpenShift Container Platform 4.13, 4.14
  • Hosted Control Planes on OpenShift Virtualisation Provider

Issue

• How to collect logs from Virtual Machines that fail to join the hosted cluster?

Resolution

1. Copy the script below to a file on the bastion node (with This page is not included, but the link has been rewritten to point to the nearest parent document.oc and virtctl tools installed) and mark it as executable (chmod +x).

2. Once the hosted cluster object is created, execute the script as following:

$ script.sh <NAME OF GUEST CLUSTER>

3. The script will loop waiting for all VMs to start and collect their logs from the serial console, saving locally.

NOTES:

• If the hosted clusters namespace is not "clusters", you can override it by setting the HC_NAMESPACE environment variable.
• Avoid connecting manually again to the VMs consoles while the script is running
• The script only collects live logs, not past ones. So you may need to reproduce the problem to capture the relevant logs.

#!/bin/bash
HC_NAMESPACE="${HC_NAMESPACE:-clusters}"
NAME=$1

if [[ -z "${NAME}" ]]
then
    echo "Please specify the name of the guest cluster."
    exit 1
fi

VMNS="${HC_NAMESPACE}"-"${NAME}"
REPLICAS=$(oc get NodePool -n "${HC_NAMESPACE}" "${NAME}" -o=jsonpath='{.spec.replicas}')
PLATFORMTYPE=$(oc get NodePool -n "${HC_NAMESPACE}" "${NAME}" -o=jsonpath='{.spec.platform.type}')
INFRAID=$(oc get HostedCluster -n "${HC_NAMESPACE}" "${NAME}" -o=jsonpath='{.spec.infraID}')

if [[ "${PLATFORMTYPE}" != "KubeVirt" ]]; then
    echo "This tool is designed for the KubeVirt provider."
    exit 1
fi

if ! which tmux >/dev/null 2>&1;
then
    echo "this tool requires tmux, please install it."
    exit 1
fi

VMNAMES=()

while [[ ${#VMNAMES[@]} < ${REPLICAS}  ]]; do
  for VMNAME in $(oc get vmi -n "${VMNS}" -l hypershift.openshift.io/infra-id="${INFRAID}" -o name 2>/dev/null); do
    SVMNAME=${VMNAME/virtualmachineinstance.kubevirt.io\//}
    if ! [[ " ${VMNAMES[*]} " =~ ${SVMNAME} ]]; then
       VMNAMES+=("${SVMNAME}")
       tmux new-session -s "${SVMNAME}" -d "virtctl console --timeout 30 -n ${VMNS} ${SVMNAME} | tee -a ${VMNS}_${SVMNAME}.log"
       echo "logs for VM ${SVMNAME} will be appended to ${VMNS}_${SVMNAME}.log"
    fi
  done
  sleep 3
done

echo "Log collection will continue in background while the VMs are running."
echo "Please avoid trying to directly connect to VM console with 'virtctl console' to avoid hijacking open sessions:"
echo "you can instead use 'tmux attach -t <vmname>' to reach open session, this will not break file logging."

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.