Red Hat Operator has version higher than the cluster version with Service Telemetry Framework

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.17 and earlier
    • Service Telemetry Framework (STF)
  • Red Hat OpenShift on AWS (ROSA)
    • 4.17 and earlier
    • Service Telemetry Framework (STF)
  • OpenShift Dedicated (OSD)
    • 4.17 and earlier
    • Service Telemetry Framework (STF)
  • Azure Red Hat OpenShift (ARO)
    • 4.17 and earlier
    • Service Telemetry Framework (STF)

Issue

  • On 2026-02-03 a push to the catalogs of older OpenShift releases (v4.12 - v4.17) resulted in potentially incompatible packages being released for Service Telemetry Framework.

Related to Red Hat Operator has version higher than the cluster version. Please refer to that article for more information.

Resolution

Upon evaluation, we identified that environments should not have major issues. If an issue is identified, the course of action is to downgrade Service Telemetry Framework (STF)

Downgrading STF

This document describes how to downgrade STF to an older version in OpenShift using the Operator Lifecycle Manager (OLM).

Overview

The downgrade process involves:

  1. Backing up and removing existing STF
  2. Uninstalling the current STF and Smart Gateway Operator
  3. Cleanup and verify
  4. Installing the older version of the STF and Smart Gateway Operator
  5. Restoring STF

Prerequisites

  • OpenShift cluster with administrative access
  • oc CLI tool configured and authenticated
  • Existing STF installation via OLM

Step 1: Backup and remove STF CRs

NOTE: Before deleting STF, save it to a local YAML files so it can be restored after the operator downgrade. See step 1.2 below.

1.1 List all STF CRs in the cluster
oc get stf -A
1.2 Backup each STF CR to a local file

Export the STF manifest to a clean YAML file that can be reapplied; e.g.:

# Replace the 'default' argument with the actual
# CR name and 'service-telemetry' with the actual namespace
oc get stf default -n service-telemetry -o yaml | \
  yq eval 'del(.metadata.creationTimestamp, .metadata.generation, .metadata.resourceVersion, .metadata.uid, > .metadata.finalizers, .metadata.annotations."kubectl.kubernetes.io/last-applied-configuration", .status)' - \
  stf-cr-backup.yaml

NOTE: This command removes server-generated metadata and status fields that would prevent the backup from being reapplied successfully. If you do not have yq, the above command won't work and you will need to manually export the full yaml and remove that data from the yaml file manually.

Step 2: Uninstall the current STO and SGO

This step can be performed via the OpenShift console UI if preferred. The below instructions demonstrate the process via the oc CLI. Though, if the UI is used, some steps still require the use of the CLI. Please see Step 3 for an example.

Delete the service-telemetry project (and with it, all STF operational components):

oc delete project service-telemetry

NOTE: It is expected that installation instructions in the official guides have been followed. If STF was deployed using a different namespace, then extra steps would be needed: precisely, removing STO/SGO subscriptions and removing STO/SGO CSVs.

Step 3: Cleanup and verify

Remove the STO and SGO CRDs:

oc delete crd servicetelemetrys.infra.watch
oc delete crd smartgateways.smartgateway.infra.watch

NOTE: These steps must be performed via the CLI even if you used the OpenShift Console UI to uninstall the operator in Step 2.

Step 4: Install the older STF

4.1 Create a namespace to contain the STF components, e.g. service-telemetry
oc new-project service-telemetry
4.2 Create an OperatorGroup in the namespace so that you can schedule the Operator pods:
oc create -f - <<EOF
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: service-telemetry-operator-group
  namespace: service-telemetry
spec:
  targetNamespaces:
  - service-telemetry
EOF
4.3 Create the new Subscription for the target version

Create the subscription for the older operator version:

oc create -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: service-telemetry-operator
  namespace: service-telemetry
spec:
  channel: stable-1.5
  installPlanApproval: Automatic
  name: service-telemetry-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

NOTE: This subscription will get the latest available version in the stable-1.5 channel (which corresponds to the recommended versions).

4.4 Verify the downgraded operator installation

Wait for the installation to complete, then verify:

# Check the CSV version
oc get csv -n service-telemetry | grep service-telemetry-operator
service-telemetry-operator.v1.5.1723536534   Service Telemetry Operator               1.5.1723536534   service-telemetry-operator.v1.5.1721288166   Succeeded

# Check the operator pod is running
oc get pods -n service-telemetry | grep service-telemetry-operator
service-telemetry-operator-c5dfd545b-prmww   1/1     Running   0          10m

Step 5: Restore STF

5.1 Apply the backed-up STF

Apply the yaml backup file to re-create the STF:

oc apply -f stf-cr-backup.yaml
5.2 Verify STF deployment

Check that STF is deployed with the older operator:

# Check STF pods in the namespaces you expect them to be
oc get pods -n service-telemetry
5.3 Connect STF to OSP

Follow steps in Chapter 4. Configuring Red Hat OpenStack Platform director for Service Telemetry Framework to connect your new STF instance with OSP.

Diagnostic Steps

Check the currentCSV for STO and SGO on OCP 4.12 - OCP 4.17 deployments with the following commands

$ oc get packagemanifest service-telemetry-operator -o jsonpath='{.status.channels[*].currentCSV}'

service-telemetry-operator.v1.5.1757650583

$ oc get packagemanifest smart-gateway-operator -o jsonpath='{.status.channels[*].currentCSV}'
smart-gateway-operator.v5.0.1757650606

Product(s)
Components

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.