Image registry support for Google Cloud Platform Workload Identity removed
With the GA of 4.10 Red Hat OpenShift Container Platform, you are allowed to perform a new install of 4.10 on GCP to enable GCP workload identity features.
Post the GA of these capabilities, bug were identified This content is not included.1, This content is not included.2 that causes content offloading, in the image registry to fail in multiple ways. As a result of these bugs Red Hat has decided to pull back support for this functionality with in the image registry and remove the code that enables this in the OpenShift 4.10.8+ versions, until it can be resolved.
Customers who installed OpenShift using any of the version between 4.10.3 and 4.10.7 and who also enabled GCP Workload Identity Features as part of that install will have an impacted upgrade experience post 4.10.8, as the image registry will not be able to use these these features.
- Note: With OpenShift Container Platform 4.10.21 updates have been shipped/released that re-enable the functionally that was disabled/removed between 4.10.8 and 4.10.20.
- If you are returning to this Knowledge base article after implementing one of the mitigation (documented below), re-enable this functionality (in 4.10.21+) after a cluster reinstall.
To use the image registry on/between an OpenShift Container Platform 4.10.8 and 4.10.21 cluster that uses GCP Workload Identity Features, you must (at a minimum) configure the image registry to use long-lived credentials instead.
- Note: Red Hat does not support the migration from a non-short lived credentials (the default install settings) environment to a short-lived credentials environment (the GCP Workload Identity Features). As a result, if you didn't follow the Using manual mode with GCP Workload Identity documentation as part of the install you are not affected by this issue.
For customers who are affected by these issues, Red Hat has 2 options that you can consider or pursue.
- Note: Red Hat does not support the migration from a non-short lived credentials (the default install settings) environment to a short-lived credentials environment (the GCP Workload Identity Features). As a result, if you choose to re-enable this functionality at a later date; another cluster reinstall will be required.
- New Cluster Installation (disabling GCP Workload Identity Features), and by proxy force the image registry to use long-lived credentials.
- Update the image registry configuration to use long-lived credentials instead, prior to upgrading.
New Cluster Installation without disabling GCP Workload Identity Features
To complete this option, follow the existing product documentation starting with:
Then any of the following install docs sets:
- Installing a cluster Quickly on GCP
- Installing a cluster on GCP with customizations
- Installing a cluster on GCP with network customizations
- Installing a cluster on GCP in a restricted network
- Installing a cluster on GCP into an existing VPC
- Installing a private cluster on GCP
- Installing a cluster on GCP using Deployment Manager templates
- Installing a cluster into a shared VPC on GCP using Deployment Manager templates
- Installing a cluster on GCP in a restricted network with user-provisioned infrastructure
Updating the image registry configuration to use long-lived credentials prior to 4.10.8 upgrades.
-
Content from cloud.google.com is not included.Obtain and initialize the gcloud CLI
-
Before Upgrading (or attempting to upgrade) to 4.10.8, get the email of the IAM service account that was created for the openshift image registry by
ccoctlgcloud iam service-accounts list --filter="displayName=<name>-openshift-image-registry-gcs"- where
nameis the name prefix for any cloud resources that are created.
- where
-
Create a long-lived credentials key file for the openshift image registry
gcloud iam service-accounts keys create /path/to/image/registy/service/account/keyfile --iam-account=<image-registry-service-account-email> -
Generate base64 encoded string for the key file created in the previous step
cat /path/to/image/registy/service/account/keyfile | base64 -w 0 -
Replace the contents of the data.service_account.json field of the installer-cloud-credentials secret in the openshift-image-registry namespace with the base64 encoded credentials key generated in the previous step.
oc patch secret installer-cloud-credentials -n openshift-image-registry --patch '{"data":{"service_account.json":"<base64_encoded_credentials_key>"}}' -
Proceed with a typical upgrade
oc adm upgrade --to=VERSION