Operator index image gets CrashLoop with "cache requires rebuild: cache reports digest as xxx, but computed digest is yyy" in OpenShift

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform
    • 4.11 and later versions
  • oc mirror command

Issue

  • Operator index image gets CrashLoop with cache requires rebuild: cache reports digest as \"xxx\", but computed digest is \"yyy\" in OpenShift

    oc logs -f redhat-operator-index-t7svb
time="2023-04-20T05:24:15Z" level=fatal msg="cache requires rebuild: cache reports digest as \"xxx\", but computed digest is \"yyy\""

Resolution

  • If the cluster is connected to Internet and the index image is pulled from Red Hat registry.redhat.io, please open a support case to Red Hat.
  • If the cluster is disconnected to Internet and you used the oc-mirror to sync the images and build the index image, please make sure the latest oc-mirror tool is used.
  • If the cluster is disconnected to Internet and you are using the latest oc-mirror, then please make sure the index image in your own registry is not corrupted. To quickly verify this, the user could consider to simply compare the size of the index image on another connected server by pulling the corresponding index image. Normally if you see the index image size is smaller in your own registry, it means the index image in your own registry is corrupted and needs to be synced again.

Root Cause

  • index image has a pre injected digest value when the image is built to make sure the cache is consistent after the image is pulled.
  • In index Pod there is an initContainer which is responsible to copy the cache content from the image to the emptyDir volume.
  • As the container of the index Pod, when the opm container starts, the opm will calculate the cache digest which has been copied to the emptyDir. If the computed digest doesn't match the value that has been injected during the build, then the opm will report the error cache requires rebuild: cache reports digest as \"xxx\", but computed digest is \"yyy\"
  • For oc-mirror bugs have been fixed to improve cache build issue. Here are the links and versions in details:

4.13 - This content is not included.OCPBUGS-12257 Catalogs rebuilt by oc-mirror are in crashloop : cache is invalid .
4.14 - This content is not included.OCPBUGS-12236 Catalogs rebuilt by oc-mirror are in crashloop : cache is invalid.
4.16 - This content is not included.OCPBUGS-38035 A better fix has been merged for cache requires rebuild: cache reports digest as .

Diagnostic Steps

  • To comfirm the pre injected digest value
$ podman run -it --rm --entrypoint /bin/bash registry.redhat.io/redhat/redhat-operator-index:v4.16
bash-5.1$ cat /tmp/cache/digest
8290a7a67a3e5b2a
  • To compare the image on the worker node has the correct size of the cache and content with other server that can connect to Internet and pull the index image from registry.redhat.io
$ podman run -it --rm --entrypoint /bin/bash registry.redhat.io/redhat/redhat-operator-index:v4.16
bash-5.1$ du -s /tmp/cache/
1182368	/tmp/cache/
bash-5.1$ ls -lh /tmp/cache/
total 452K
drwxr-x---. 2 1001 root 380K Apr  9 20:47 cache
-rw-r-----. 1 1001 root   16 Apr  9 20:47 digest
bash-5.1$ ls -lh /tmp/cache/cache/ |wc -l
4758
SBR
Product(s)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.