Secrets Store CSI Driver Operator has version higher than the cluster version (4.14, 4.15, 4.16, 4.17)

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (OCP) 4.14, 4.15, 4.16, 4.17
  • Secrets Store CSI Driver Operator

Issue

  • During a recent incident window (see Solution 7137887), a catalog release error caused OCP 4.18 operator content to be pushed to OCP 4.12–4.17 clusters.

  • The Secrets Store CSI Driver Operator is a Technology Preview feature in OCP 4.14–4.17, available only on the preview OLM channel. In OCP 4.18, the operator graduated to General Availability (GA) and moved exclusively to the stable OLM channel.

  • Because the preview channel has no entries in the 4.18 catalog, the channel mismatch (preview vs. stable) should have prevented any automatic upgrades for operators running on OCP 4.14–4.17, even if subscriptions were set to Automatic approval. However, if your Secrets Store CSI Driver Operator was manually upgraded or is otherwise running a version intended for OCP 4.18 on an older cluster (4.14–4.17), this article provides the steps to roll back the operator to the appropriate version for your OCP cluster.

Resolution

Downgrade the Secrets Store CSI Driver Operator

Follow these steps to remove the incorrect version and trigger a fresh installation of the supported version.

1. Perform a backup

  • Back up all the intended operator resources:
$ oc get secretproviderclasses -A -o yaml > spc-backup.yaml

$ oc get secretproviderclasspodstatuses -A -o yaml > spcps-backup.yaml

$ oc get clustercsidrivers secrets-store.csi.k8s.io -o yaml > ccsd-backup.yaml

$ oc get subscription secrets-store-csi-driver-operator -n openshift-cluster-csi-drivers -o yaml > subscription-backup.yaml
  • Backup all application pods that use the secrets-store.csi.k8s.io provider.

2. Delete the ClusterCSIDriver CR

$ oc delete clustercsidrivers secrets-store.csi.k8s.io

Wait for driver pods to terminate:

$ oc get pods -n openshift-cluster-csi-drivers

3. Remove existing subscription
First find out the existing subscription:

$ oc get subscription -n openshift-cluster-csi-drivers

And then delete it:

$ oc delete subscription secrets-store-csi-driver-operator -n openshift-cluster-csi-drivers

4. Find and delete the incorrect CSV

$ oc get csv -n openshift-cluster-csi-drivers
NAME                                                     DISPLAY                             VERSION               REPLACES                                                 PHASE
secrets-store-csi-driver-operator.v4.18.0-202601302238   Secrets Store CSI Driver Operator   4.18.0-202601302238   secrets-store-csi-driver-operator.v4.14.0-202601210114   Succeeded

Replace with the name we identified in the step above.

$ oc delete csv secrets-store-csi-driver-operator.v4.18.0-<VERSION-TAG> -n openshift-cluster-csi-drivers

From the above example, it would be:

$ oc delete csv secrets-store-csi-driver-operator.v4.18.0-202601302238 -n openshift-cluster-csi-drivers

5. Verify uninstall
Operator and Driver pods should be removed

$ oc get pods -n openshift-cluster-csi-drivers

6. Re-install the operator with the version supported on your cluster

Create a new Subscription with channel: preview and startingCSV pinned to the target version. The following are some cluster-wise examples.

Cluster VersionExample startingCSV
4.17secrets-store-csi-driver-operator.v4.17.0-202602100344
4.16secrets-store-csi-driver-operator.v4.16.0-202602091312
4.15secrets-store-csi-driver-operator.v4.15.0-202601221319
4.14secrets-store-csi-driver-operator.v4.14.0-202601210114

Create a file called subscription-downgrade.yaml with the following content and replace <TARGET_CSV> with the version matching your cluster:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: secrets-store-csi-driver-operator
  namespace: openshift-cluster-csi-drivers
spec:
  channel: preview
  name: secrets-store-csi-driver-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Manual
  startingCSV: <TARGET_CSV>

Apply it:

$ oc apply -f subscription-downgrade.yaml

Key points regarding the subscription settings:

  • channel: preview is required for 4.14-4.17 (the "stable" channel did not exist before 4.18).
  • installPlanApproval: Manual prevents OLM from auto-upgrading.
  • startingCSV pins the install to the exact target version.

7. Find and approve the pending InstallPlan

Find the pending InstallPlan in the operator namespace:

$ oc get installplan -n openshift-cluster-csi-drivers

Approve it (replace <INSTALL_PLAN_NAME> with the actual name retrieved above):

$ oc patch installplan <INSTALL_PLAN_NAME> \
-n openshift-cluster-csi-drivers \
--type merge -p '{"spec":{"approved":true}}'

8. Verify Operator pod
Verify that the Operator pod is running and ready:

$ oc get pods -n openshift-cluster-csi-drivers -l app=secrets-store-csi-driver-operator

Verify that the CSV matches your intended OCP version:

$ oc get csv -n openshift-cluster-csi-drivers

9. Recreate ClusterCSIDriver CR

$ export CLUSTER_CSI_DRIVER_NAME=secrets-store.csi.k8s.io

$ oc apply -f - <<EOF
apiVersion: operator.openshift.io/v1
kind: ClusterCSIDriver
metadata:
  name: ${CLUSTER_CSI_DRIVER_NAME}
spec:
  managementState: Managed
EOF

10. Verify Driver DaemonSet health
Ensure all node pods are running and ready:

$ oc get daemonset secrets-store-csi-driver-node -n openshift-cluster-csi-drivers

$ oc get pods -n openshift-cluster-csi-drivers -l app=secrets-store-csi-driver-node

References

Diagnostic Steps

You may be affected by this issue if you are running OpenShift version <v4.18 and have secrets-store-csi-driver-operator running on v4.18:

$ oc get csv -n openshift-cluster-csi-drivers
NAME                                                     DISPLAY                             VERSION               REPLACES                                                 PHASE
secrets-store-csi-driver-operator.v4.18.0-202601302238   Secrets Store CSI Driver Operator   4.18.0-202601302238   secrets-store-csi-driver-operator.v4.14.0-202601210114   Succeeded

In above example, you can see that CSV version is v4.18 and hence the cluster version also must be v4.18.


This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.