Secrets Store CSI Driver Operator has version higher than the cluster version (4.14, 4.15, 4.16, 4.17)
Environment
- Red Hat OpenShift Container Platform (OCP) 4.14, 4.15, 4.16, 4.17
- Secrets Store CSI Driver Operator
Issue
-
During a recent incident window (see Solution 7137887), a catalog release error caused OCP 4.18 operator content to be pushed to OCP 4.12–4.17 clusters.
-
The Secrets Store CSI Driver Operator is a Technology Preview feature in OCP 4.14–4.17, available only on the
previewOLM channel. In OCP 4.18, the operator graduated to General Availability (GA) and moved exclusively to thestableOLM channel. -
Because the
previewchannel has no entries in the 4.18 catalog, the channel mismatch (previewvs.stable) should have prevented any automatic upgrades for operators running on OCP 4.14–4.17, even if subscriptions were set to Automatic approval. However, if your Secrets Store CSI Driver Operator was manually upgraded or is otherwise running a version intended for OCP 4.18 on an older cluster (4.14–4.17), this article provides the steps to roll back the operator to the appropriate version for your OCP cluster.
Resolution
Downgrade the Secrets Store CSI Driver Operator
Follow these steps to remove the incorrect version and trigger a fresh installation of the supported version.
1. Perform a backup
- Back up all the intended operator resources:
$ oc get secretproviderclasses -A -o yaml > spc-backup.yaml
$ oc get secretproviderclasspodstatuses -A -o yaml > spcps-backup.yaml
$ oc get clustercsidrivers secrets-store.csi.k8s.io -o yaml > ccsd-backup.yaml
$ oc get subscription secrets-store-csi-driver-operator -n openshift-cluster-csi-drivers -o yaml > subscription-backup.yaml
- Backup all application pods that use the
secrets-store.csi.k8s.ioprovider.
2. Delete the ClusterCSIDriver CR
$ oc delete clustercsidrivers secrets-store.csi.k8s.io
Wait for driver pods to terminate:
$ oc get pods -n openshift-cluster-csi-drivers
3. Remove existing subscription
First find out the existing subscription:
$ oc get subscription -n openshift-cluster-csi-drivers
And then delete it:
$ oc delete subscription secrets-store-csi-driver-operator -n openshift-cluster-csi-drivers
4. Find and delete the incorrect CSV
$ oc get csv -n openshift-cluster-csi-drivers
NAME DISPLAY VERSION REPLACES PHASE
secrets-store-csi-driver-operator.v4.18.0-202601302238 Secrets Store CSI Driver Operator 4.18.0-202601302238 secrets-store-csi-driver-operator.v4.14.0-202601210114 Succeeded
Replace with the name we identified in the step above.
$ oc delete csv secrets-store-csi-driver-operator.v4.18.0-<VERSION-TAG> -n openshift-cluster-csi-drivers
From the above example, it would be:
$ oc delete csv secrets-store-csi-driver-operator.v4.18.0-202601302238 -n openshift-cluster-csi-drivers
5. Verify uninstall
Operator and Driver pods should be removed
$ oc get pods -n openshift-cluster-csi-drivers
6. Re-install the operator with the version supported on your cluster
Create a new Subscription with channel: preview and startingCSV pinned to the target version. The following are some cluster-wise examples.
| Cluster Version | Example startingCSV |
|---|---|
| 4.17 | secrets-store-csi-driver-operator.v4.17.0-202602100344 |
| 4.16 | secrets-store-csi-driver-operator.v4.16.0-202602091312 |
| 4.15 | secrets-store-csi-driver-operator.v4.15.0-202601221319 |
| 4.14 | secrets-store-csi-driver-operator.v4.14.0-202601210114 |
Create a file called subscription-downgrade.yaml with the following content and replace <TARGET_CSV> with the version matching your cluster:
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: secrets-store-csi-driver-operator
namespace: openshift-cluster-csi-drivers
spec:
channel: preview
name: secrets-store-csi-driver-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
installPlanApproval: Manual
startingCSV: <TARGET_CSV>
Apply it:
$ oc apply -f subscription-downgrade.yaml
Key points regarding the subscription settings:
channel: previewis required for 4.14-4.17 (the "stable" channel did not exist before 4.18).installPlanApproval: Manualprevents OLM from auto-upgrading.startingCSVpins the install to the exact target version.
7. Find and approve the pending InstallPlan
Find the pending InstallPlan in the operator namespace:
$ oc get installplan -n openshift-cluster-csi-drivers
Approve it (replace <INSTALL_PLAN_NAME> with the actual name retrieved above):
$ oc patch installplan <INSTALL_PLAN_NAME> \
-n openshift-cluster-csi-drivers \
--type merge -p '{"spec":{"approved":true}}'
8. Verify Operator pod
Verify that the Operator pod is running and ready:
$ oc get pods -n openshift-cluster-csi-drivers -l app=secrets-store-csi-driver-operator
Verify that the CSV matches your intended OCP version:
$ oc get csv -n openshift-cluster-csi-drivers
9. Recreate ClusterCSIDriver CR
$ export CLUSTER_CSI_DRIVER_NAME=secrets-store.csi.k8s.io
$ oc apply -f - <<EOF
apiVersion: operator.openshift.io/v1
kind: ClusterCSIDriver
metadata:
name: ${CLUSTER_CSI_DRIVER_NAME}
spec:
managementState: Managed
EOF
10. Verify Driver DaemonSet health
Ensure all node pods are running and ready:
$ oc get daemonset secrets-store-csi-driver-node -n openshift-cluster-csi-drivers
$ oc get pods -n openshift-cluster-csi-drivers -l app=secrets-store-csi-driver-node
References
Diagnostic Steps
You may be affected by this issue if you are running OpenShift version <v4.18 and have secrets-store-csi-driver-operator running on v4.18:
$ oc get csv -n openshift-cluster-csi-drivers
NAME DISPLAY VERSION REPLACES PHASE
secrets-store-csi-driver-operator.v4.18.0-202601302238 Secrets Store CSI Driver Operator 4.18.0-202601302238 secrets-store-csi-driver-operator.v4.14.0-202601210114 Succeeded
In above example, you can see that CSV version is v4.18 and hence the cluster version also must be v4.18.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.