Pods fail with ImagePullBackOff after upgrade from 4.14.x to 4.14.14 or 4.15 on Azure Stack Hub
Environment
- OpenShift Azure Stack Hub Only
- 4.14.[0-14]
Issue
- Pods using images that are stored in the default OpenShift internal registry fails to be pull:
112s Warning Failed pod/new-default-deploy-547ccb8549-9wll5 Failed to pull image "default-route-openshift-image-registry.apps.mycluster.aroapp.io/new-default-app/ubi8:latest": rpc error: code = Unknown desc = reading manifest latest in default-route-openshift-image-registry.apps.mycluster.aroapp.io/new-default-app/ubi8: manifest unknown
112s Normal Pulling pod/new-default-deploy-547ccb8549-9wll5 Pulling image "default-route-openshift-image-registry.apps.mycluster.aroapp.io/new-default-app/ubi8:latest"
112s Warning Failed pod/new-default-deploy-547ccb8549-9wll5 Error: ErrImagePull
98s Warning Failed pod/new-default-deploy-547ccb8549-9wll5 Error: ImagePullBackOff
84s Normal BackOff pod/new-default-deploy-547ccb8549-9wll5 Back-off pulling image "default-route-openshift-image-registry.apps.mycluster.aroapp.io/new-default-app/ubi8:latest"
Resolution
IMPORTANT:
The upgrade cases from 4.13 -> 4.14.[0-14] require manual intervention for customers using the openshift internal image registry to store images. Customers should update to 4.14.15+ or 4.15 and then follow this recovery procedure
NOTE: It is recommended that customers on Azure Stack Hub that use the internal registry avoid upgrading to 4.14.0 through 4.14.14, and skip directly to 4.14.15+ when upgrading, in which case they do not need to use this recovery procedure.
The below guides target customers who updated from 4.14.[0-14] to 4.14.15+ or 4.15.
PRE-REQUISITES
- You have followed Content from learn.microsoft.com is not included.Azure’s guide to login to your Azure Stack Hub environment
- You have oc installed and have access to the cluster which you need to fix
- You have az installed and have read/write access to the image registry’s storage account
ENVIRONMENT SETUP
> ACCOUNT_NAME=$(oc get configs.imageregistry/cluster -ojsonpath="{.status.storage.azure.accountName}")
> echo ${ACCOUNT_NAME}
imageregistrycilnzkj8z8h
> CONTAINER_NAME=$(oc get configs.imageregistry/cluster -ojsonpath="{.status.storage.azure.container}")
> echo ${CONTAINER_NAME}
ci-ln-zkl93c2-1d09d-q6fnp-image-registry-qepfvmlygeowhxpqpyown
> TOKEN_EXPIRY='2024-02-22T16:00:00Z' ### update with a date/time in the future ###
> ACCOUNT_KEY=$(az storage account keys list --account-name=${ACCOUNT_NAME} | jq -r '.[0].value')
> SAS_TOKEN=$(az storage container generate-sas --account-key=${ACCOUNT_KEY} --account-name=${ACCOUNT_NAME} --name=${CONTAINER_NAME} --output=tsv --expiry="${TOKEN_EXPIRY}" --permissions=rwdl)
> SLASH_DOCKER_URL=$(az storage blob url --account-name ${ACCOUNT_NAME} --container-name ${CONTAINER_NAME} --name='//' --output=tsv)?${SAS_TOKEN}
> echo ${SLASH_DOCKER_URL}
https://imageregistrycilnzkj8z8h.blob.core.windows.net/ci-ln-zkl93c2-1d09d-q6fnp-image-registry-qepfvmlygeowhxpqpyown//?se=2024-02-22T16%3A00%3A00Z&sp=racwdl&sv=2022-11-02&sr=c&sig=<redacted>
#### PLEASE NOTE! the double slash (//) in the command above is intentional and necessary! ###
> NO_SLASH_DOCKER_URL=$(az storage blob url --account-name ${ACCOUNT_NAME} --container-name ${CONTAINER_NAME} --name='docker' --output=tsv)?${SAS_TOKEN}
> echo ${NO_SLASH_DOCKER_URL}
https://imageregistrycilnzkj8z8h.blob.core.windows.net/ci-ln-zkl93c2-1d09d-q6fnp-image-registry-qepfvmlygeowhxpqpyown/docker?se=2024-02-22T16%3A00%3A00Z&sp=racwdl&sv=2022-11-02&sr=c&sig=<redacted>
Guide for customers who upgraded from 4.14.0 through 4.14.14 to 4.14.15+
Requirements:
- You have followed Content from learn.microsoft.com is not included.Azure’s guide to login to your Azure Stack Hub environment
- You must have access to the image registry storage account
- You must have azure’s az client installed on your machine
- You have executed the setup steps on the previous section
STEPS:
Use az copy to copy blobs from docker into /docker. You may optionally delete the docker directory.
> az storage copy -s "${NO_SLASH_DOCKER_URL}" -d "${SLASH_DOCKER_URL}" --recursive
### (optional) ###
> az storage remove --account-name=${ACCOUNT_NAME} --account-key=${ACCOUNT_KEY} --container=${CONTAINER_NAME} -n docker
Product(s)
Components
Category
Tags
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.