OpenShift Virtualization operator version is higher than the supported OpenShift cluster version

Solution Unverified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • < 4.18
  • OpenShift Virtualization
    • < 4.18

Issue

Due to the incident documented in Red Hat Operator has version higher than the cluster version, the OpenShift Virtualization (kubevirt-hyperconverged) operator may have been upgraded to a version higher than what is supported by the current OpenShift Container Platform (RHOCP) cluster version.

Resolution

OCP 4.17 and OpenShift Virtualization 4.18

In this scenario, the supported option is to stay on OpenShift Virtualization 4.18. With this configuration, there is an issue with the Topology view in Developer view mode.

To solve this issue and use OpenShift Virtualization 4.18 on OCP 4.17, follow the steps below.

$ oc adm release info quay.io/openshift-release-dev/ocp-release:4.18.33-x86_64 | grep console

  console                                        sha256:ada2d1130808e4aaf425a9f236298cd9c93f1ca51d0147efb7a72cb9180b0657
  • Patch ClusterVersion overrides to put the console-operator deployment as unmanaged :
oc patch clusterversion version --type json -p '[{
  "op": "add",
  "path": "/spec/overrides",
  "value": [{
    "kind": "Deployment",
    "group": "apps",
    "name": "console-operator",
    "namespace": "openshift-console-operator",
    "unmanaged": true
  }]
}]'
  • Update the console image in console-operator deployment env var (which the sha extracted in a previous step) :
oc set env deployment/console-operator -n openshift-console-operator CONSOLE_IMAGE=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ada2d1130808e4aaf425a9f236298cd9c93f1ca51d0147efb7a72cb9180b0657
  • Confirm that new console pods are created with the updated image in openshift-console namespace :
oc get pods -n openshift-console
  • In order to continue receiving updates for OpenShift Virtualization 4.18.z while on OCP 4.17, these steps should be performed :
    • Create a custom CatalogSource pointing to redhat-operators v4.18:
 $ oc apply -f - <<EOF
   apiVersion: operators.coreos.com/v1alpha1
   kind: CatalogSource
   metadata:
     name: redhat-operators-v418
     namespace: openshift-marketplace
   spec:
     displayName: Red Hat Operators v4.18
     image: registry.redhat.io/redhat/redhat-operator-index:v4.18
     publisher: Red Hat
     sourceType: grpc
     updateStrategy:
       registryPoll:
         interval: 15m
 EOF

 catalogsource.operators.coreos.com/redhat-operators-v418 created
  • Patch the existing kubevirt-hyperconverged subscription (in openshift-cnv) namespace to the newly-created custom catalog source :
$ oc patch subscription kubevirt-hyperconverged -n openshift-cnv --type merge -p '{"spec":{"source":"redhat-operators-v418"}}'
  • Optional : disable automatic upgrades, so upgrade will be done only upon approval :
$ oc patch subscriptions.operators.coreos.com -n openshift-cnv kubevirt-hyperconverged --type merge -p '{"spec":{"installPlanApproval": "Manual"}}'
  • Once the cluster is ready to get upgraded to OCP 4.18 :
    • Remove the override to unblock OCP upgrade :
$ oc patch clusterversion version --type json -p '[{"op": "remove", "path": "/spec/overrides"}]'

  • Upgrade OpenShift Container Platform to 4.18 :

  • Once OCP upgrade is finished, re-patch the kubevirt-hyperconverged subscription back to the default redhat-operators catalog source (which is now pointing to v4.18) :

$ oc patch subscription kubevirt-hyperconverged -n openshift-cnv --type merge -p '{"spec":{"source":"redhat-operators"}}'
  • Delete the custom catalog source :
 $ oc delete catsrc redhat-operators-v418 -n openshift-marketplace

 catalogsource.operators.coreos.com "redhat-operators-v418" deleted from openshift-marketplace namespace

Root Cause

The incident incorrectly exposed the following OpenShift Virtualization operator channels to older, unsupported OCP versions:

  • OpenShift Virtualization 4.18 was visible on OCP 4.12 through 4.17

Diagnostic Steps

Run the diagnostic steps available in the Red Hat Operator has version higher than the cluster version knowledge base solution and validate that the kubevirt-hyperconverged operator has mismatching version with the OCP cluster.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.