[OCP] Disconnected cluster and file-based catalogs - how to avoid pruning of no longer listed operators?

Solution Verified - Updated

Environment

Issue

  • oc-mirror automatically prunes operator images from the registry if they are no longer listed in the catalog.
  • ImagePullBackOff for operator images after updating a mirrored catalog by running oc-mirror.
  • Not able to install older versions from a mirrored catalog created by oc-mirror.

Resolution

Primary solution

With the release of oc-mirror with OpenShift 4.12.11 (or later), the oc-mirror command introduced a command-line flag --skip-pruning to disable automatic pruning. The following procedure became obsolete but should be kept as a reference.

Please take a look at the official documentation of OpenShift 4.12 for further details on using oc-mirror.

Obsoleted solution (for reference)

  • Create a JSON file containing all operators, channel and versions of a catalog (substitute redhat-operator-index:v4.11 and related filenames with the actual operator catalog and version to be processed).
% opm render registry.redhat.io/redhat/redhat-operator-index:v4.11 > redhat-operators.json
  • Create directory for customized catalog contents and extract operator definitions from source file
% mkdir custom-redhat-operators
% jq 'select(.package == "quay-operator" or (.schema == "olm.package" and .name == "quay-operator" ))' redhat-operators.json > custom-redhat-operator/just-quay.json
% jq 'select(.package == "cluster-logging" or (.schema == "olm.package" and .name == "cluster-logging" ))' redhat-operators.json > custom-redhat-operator/just-cluster-logging.json
... (repeat for every other operator of choice)

Note: A list of all operator names contained in that catalog can be obtained by running the following query:

% jq -r '.package' redhat-operators.json | grep -v "null" | sort -u
  • Build customized catalog image
% opm validate custom-redhat-operator/
% opm generate dockerfile custom-redhat-operator
% podman build -t custom-redhat-operator:v4.11 -f custom-redhat-operator.Dockerfile
  • Verify custom catalog image contents
% podman run -it -p 50051:50051 localhost/custom-redhat-operator:v4.11
...

[open second terminal]
% grpcurl -plaintext localhost:50051 api.Registry/ListPackages
{
  "name": "cluster-logging"
}
{
  "name": "quay-operator"
}
  • Push custom-redhat-operator catalog image to custom registry
% podman tag custom-redhat-operator:v4.11 registry.example.com/operator-catalogs/custom-redhat-operator:v4.11
% podman push registry.example.com/operator-catalogs/custom-redhat-operator:v4.11

After the creation of that catalog, the image can be pushed to the custom registry and the images can be mirrored by following the procedure of OpenShift 4.10

% oc adm catalog mirror \
  registry.example.com/operator-catalogs/custom-redhat-operator:v4.11 \
  registry.example.com/operators \
  # additional options, as per documentation

This procedure should get obsolete, once This content is not included.CFE-739 got implemented and oc-mirror would allow administrators to disable the pruning mechanism.

Root Cause

  • Using opm to prune catalog images does no longer work for file-based catalags, introduced with OpenShift-4.11.
  • oc-mirror pruning operator images which are no longer listed in a catalog
  • Operator-catalogs not listing older versions

With the switch from SQLite based operator catalogs to file-based catalogs, the opm tool is no longer able to create custom catalogs from selected operators. oc-mirror would allow the selection of operators from file-based catalogs, but would automatically prune images of operators which are no longer listed in this catalog. Due to current design of some operator-build-pipelines, only the most recent version of an operator is added to the catalog. Therefore, oc-mirror would remove operator images which have been mirrored in previous runs in case there was an update of an operator.

SBR
Product(s)
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.