oc mirror : A Comprehensive tool for Image Mirroring

Updated

oc mirror is an OpenShift Client (oc) plugin that manages OpenShift release, operator catalog, helm charts, and associated container images for mirror registries that support OpenShift environments.

Index

How it Works

The oc Mirror plugin processes a configuration file called ImageSetConfiguration, and that defines the configurations that are required by the oc Mirror. The two main outputs of oc mirror are a mirror registry payload, called an imageset, and a metadata file used to store information about previously generated images.
The plugin can run in either stateless or stateful mode. In stateless mode, oc mirror will not retain the metadata for each imageset creation. However, the recommended approach is to run the oc mirror with a storageConfig defined within the configuration file to ensure OpenShift release and operator upgradeability. The ability to calculate and maintain upgrade paths for OpenShift releases and operators are the features that constitute oc mirror's lifecycle management capabilities. The capability includes intelligently mirroring OpenShift releases and operators while maintaining upgrade paths from all previous component versions within a mirror registry. All release and operator component versions within a mirror registry should have an upgrade path to any higher version of that component within the mirror registry.

Prerequisites to proceed with oc mirror

Must have a container image registry that supports Docker v2-2, such as Red Hat Quay. If there is no already existing solution for a container image registry, subscribers of OpenShift Container Platform are provided a mirror registry for Red Hat OpenShift. The mirror registry for Red Hat OpenShift is included with the subscription and is a small-scale container registry that can be used to mirror the required container images of OpenShift Container Platform in disconnected installations.

How to download oc mirror binary

Follow the steps which is mentioned in the Red Hat documentation.

How to setup oc mirror authentication

Follow the steps which is mentioned in the Red Hat documentation.

What is an ImageSetConfiguration and how to create a sample ImageSetConfiguration

The ImageSetConfiguration file defines which OpenShift Container Platform releases, operators, and other images to mirror, along with other configuration settings for the oc mirror plugin.
There is a section storageConfig in the ImageSetConfiguration file, and the storageConfig can be a local directory or a registry that supports Docker v2-2. The oc mirror plugin stores metadata in this storageConfig during image mirroring. Do not delete or modify the metadata that is generated by the oc mirror plugin. It is recommended to use the same storageConfig every time mirroring is performed for the same mirror registry.

Use below command to create a template for the ImageSetConfiguration and save it to a file called imageset-config.yaml:

# oc mirror init --registry example.com/mirror/oc-mirror-metadata > imageset-config.yaml

  • ImageSetConfiguration
    It defines which OpenShift Container Platform releases, operators, and other images to mirror, along with other configuration settings for the oc mirror plugin. Put differently, ImageSetConfiguration is a file that contains all the information about platform releases, operators, and other images to mirror. It is being passed to the oc mirror binary to mirror the necessary images. This article will have more examples later on.

  • ImageContentSourcePolicy
    ImageContentSourcePolicy holds cluster-wide information about how to handle registry mirror rules. When a pod tries to pull an image from an upstream registry, CRIO will be redirected to the mirror registry, which is defined in the ImageContentSourcePolicy, because it will have information about the mirror registry for each upstream registry. The contents of this CR will be written to the file /etc/containers/registry.conf of each node in the cluster once the CR has been created in the cluster. Please note that adding contents to this CR is not going to trigger node drains or reboots, but removing any content would. The ImageContentSourcePolicy will be considered only while pulling images using digest, not by tag. Below is an example of an ImageContentSourcePolicy:

    apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: release-0
spec:
  repositoryDigestMirrors:
  - mirrors:
    - <mirror registry>/ocp4/openshift/release-images
    source: quay.io/openshift-release-dev/ocp-release ---> any image pull from "quay.io/openshift-release-dev/ocp-release" will forwarded to "<mirror registry>/ocp4/openshift/release-images"
  - mirrors:
    - <mirror regitry>/ocp4/openshift/release
    source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

  • Content from olm.operatorframework.io is not included.CatalogSource
    A CatalogSource represents a store of metadata that OLM can query to discover and install operators and their dependencies. The mirror tool will create a custom catalogsource based on the mirror registry image that is being pushed during the mirror.

  • mapping.txt(file)
    mapping.txt is the mapping file that is created by the mirror tool to identify the location of the upstream images in the local mirror registry.

How is the oc mirror authentication is working

The authentication works based on the credentials (pull-secret) present in the location ~/.docker/config.json or $XDG_RUNTIME_DIR/containers/auth.json. The first priority goes to the file ~/.docker/config.json, and if the directory ~/.docker is not present, then the credentials will be read from $XDG_RUNTIME_DIR/containers/auth.json.

What is the importance of storageConfig parameter in ImageSetConfiguration

The plugin can run in either stateless or stateful mode. In stateless mode, oc mirror will not retain the metadata for each imageset creation. However, the recommended approach is to run oc-mirror with a storage backend defined within the configuration file to ensure OpenShift release and operator upgradeability. The ability to calculate and maintain upgrade paths for OpenShift releases and operators are the features that constitute oc mirror's lifecycle management capabilities. The capability includes intelligently mirroring OpenShift releases and operators while maintaining upgrade paths from all previous component versions within a mirror registry. All release and operator component versions within a mirror registry should have an upgrade path to any higher version of that component within the mirror registry.

How to list available versions of an operator using oc mirror

Please follow the Red Hat KCS for more information's.

How does image pruning work with oc mirror

Please follow the Red Hat KCS for more information's.

How to find the list of images that is mirrored by oc mirror

The mapping.txt file will contain all the images that it has mirrored, and the file will be auto-generated by the oc mirror process. The mirror registry location of the upstream images will be contained in the same file.

Preparing for mirroring

  • Steps to mirror openshift platform images without OperatorHub operators for a fresh installation.

  • Scenario-1 : Use below ImageSetConfiguration to mirror the most recent platform images from a certain channel:

     kind: ImageSetConfiguration
 apiVersion: mirror.openshift.io/v1alpha2
 archiveSize: 4                                                      
 storageConfig:                                                      
   registry:
     imageURL: <registry>:8443/mirror/oc-mirror-metadata                 
     skipTLS: false
mirror:
  platform:
    channels:
    - name: stable-4.14                                             
      type: ocp
    graph: true

  • Scenario-2 : Use below ImageSetConfiguration to mirror a specific version (4.14.1 in this case) from a certain channel:

     kind: ImageSetConfiguration
 apiVersion: mirror.openshift.io/v1alpha2
 archiveSize: 4                                                      
 storageConfig:                                                      
   registry:
     imageURL: <registry>:8443/mirror/oc-mirror-metadata                 
     skipTLS: false
mirror:
  platform:
    channels:
    - name: stable-4.14                                             
      type: ocp
      minVersion: 4.14.1
      maxversion: 4.14.1
    graph: true

  • Scenario-3 : Use below ImageSetConfiguration to mirror a range of versions (in this case, 4.14.1 to 4.14.4) from a certain channel:

 kind: ImageSetConfiguration
 apiVersion: mirror.openshift.io/v1alpha2
 archiveSize: 4                                                      
 storageConfig:                                                      
   registry:
     imageURL: <registry>:8443/mirror/oc-mirror-metadata                 
     skipTLS: false
mirror:
  platform:
    channels:
    - name: stable-4.14                                             
      type: ocp
      minVersion: 4.14.1
      maxversion: 4.14.4
    graph: true

  • Scenario-4 : Use below ImageSetConfiguration to mirror a specific version from multiple channels:

     kind: ImageSetConfiguration
 apiVersion: mirror.openshift.io/v1alpha2
 archiveSize: 4                                                      
 storageConfig:                                                      
   registry:
     imageURL: <registry>:8443/mirror/oc-mirror-metadata                 
     skipTLS: false
mirror:
  platform:
    channels:
    - name: stable-4.14                                             
      type: ocp
      minVersion: 4.14.1
      maxVersion: 4.14.1
    - name: stable-4.11
      type: ocp
      minVersion: 4.11.42
      maxversion: 4.11.42
    - name: stable-4.12
      type: ocp
      minVersion: 4.12.19
      maxversion: 4.12.19
    graph: true

    The ImageSetConfiguration samples shown above can be used if the upstream registry as well as the mirror registry are accessible from the bastion node where the oc mirror command is running, this is called partial disconnected setup. In every instance, the mirror command will be the same. See below:

    # oc mirror --config=./imageset-config.yaml docker://<registry>:8443 – Please note that give the repository name after the port number if wish to mirror the images to a specific repository.

    If the setup is totally disconnected and there is no access to the upstream and mirror registry from a single machine, then the images should mirror to the local disk from a machine that can access the upstream registry. After that, copy the images that are mirrored to the disk to another machine that has access to the mirror registry. Once it's copied, use oc mirror to push the images to the mirror registry. In this case, storageConfig and mirroring instructions need to be slightly changed. The remaining ImageSetConfiguration variables won't change. Take a look at the details below.

    • This is how the storageConfig will appear (metadata will be stored locally on the drive rather than in the registry in this scenario).

      storageConfig:                                                      
  local:
    path: /home/user/metadata

    • The commands for mirroring to local disk appear as follows:

      # oc mirror --config=./imageset-config.yaml file://<path_to_output_directory> – mirror to disk

      Transfer the mirror_seq1_000000.tar file created in the local directory to a machine with access to the mirror registry.

    • The commands for mirroring from local disk to registry appear as follows:

      # oc mirror --from=./mirror_seq1_000000.tar docker://registry.example:5000 – this should run from the machine which is having mirror registry access

    Note: In the aforementioned scenario, if the archiveSize value is defined in the ImageSetConfiguration file, mirroring to disk will produce numerous archive (*.tar) files in a sequential manner. For example, the archive generated by the mirror to disk procedure is called mirror_seq1_000000.tar. Use the directory containing the *.tar files instead of the archive file if it is divided into several *.tar files.

  • Steps to mirror openshift platform images along with desired operators for new installation.

    kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 4                                                      
storageConfig:                                                      
  registry:
    imageURL: <registry>:8443/mirror/oc-mirror-metadata                 
    skipTLS: false
mirror:
  platform:
    channels:
    - name: stable-4.14                                             
      type: ocp
      minVersion: 4.14.1
      maxversion: 4.14.1
    graph: true 
 operators:
   - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
     packages:
     - name: serverless-operator
       channels:
       - name: stable

Look at the Red Hat KCS different examples of ImageSetConfiguration to use with operator image mirroring

  • How to mirror openshift operator images to an already existing Openshift cluster in order to install operators.

There are two approaches that can be used to achieve the same goal:

  • Use the same ImageSetConfiguration that is used for platform installation and add the desired operator to the ImageSetConfiguration, but please remember to specifically mention the currently installed openshift version in the ImageSetConfiguration if it is already not present. Otherwise, there's a potential that the current opernshift version images will be pruned. Please refer to the example below.

    The process remains unchanged as above; the only thing that changes is adding the desired operators to the ImageSetConfiguration. Please refer to the sample below.

     ```
 kind: ImageSetConfiguration
 apiVersion: mirror.openshift.io/v1alpha2
 archiveSize: 4                                                      
 storageConfig:                                                      
   registry:
     imageURL: <registry>:8443/mirror/oc-mirror-metadata                 
     skipTLS: false
 mirror:
   platform:
     channels:
     - name: stable-4.14                                             
       type: ocp
       minVersion: 4.14.1  #### the current cluster version is 4.14.1, so specifically mentioned to avoid pruning of images.
       maxversion: 4.14.1  #### kept max version also as current running version as there is no requirement of any other versions
     graph: true 
 operators:
   - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
     packages:
     - name: serverless-operator  #### Planning to onboard serverless-operator to the existing cluster
       channels:
       - name: stable
 ```

  • Use a separate ImageSetConfiguration for operators alone. In this case, it will have two ImageSetConfiguration to manage, one for platform and another for operator, but remember to keep different metadata (storageConfig) in both ImageSetConfiguration.

    Please see the Red Hat KCS different examples of ImageSetConfiguration to use with operator image mirroring.

Best practices to be followed while mirroring platform and operator images for upgrade

oc mirror has an image pruning feature, and it will prune the images from the mirror registry that are no longer referenced in the ImageSetConfiguration. It is advisable to use the minimum version as the installed version and the maximum version as the desired version in the ImageSetConfiguration. View the following example.

  • Installed the cluster and quay-operator using below ImageSetConfiguration:

    apiVersion: mirror.openshift.io/v1alpha2
kind: ImageSetConfiguration
storageConfig:
  registry:
    imageURL: <registry>:8443/mirror/oc-mirror-metadata                 
    skipTLS: false
mirror:
  platform:
    channels:
    - name: stable-4.14                                             
      type: ocp
      minVersion: 4.14.1
      maxversion: 4.14.1
    graph: true 
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
      packages:
        - name: quay-operator
          channels:
         - name: stable-x

  • The most recent version of quay-operator x.1.1 that is available in the stable-x channel will be deployed.

  • Planning to upgrade the quay-operator to the most recent version, x.1.2, which is available in the stable-x channel, after a few months.

  • What would happen if the same ImageSetConfiguration used to mirror the images? Because oc mirror has a pruning feature with regard to current storageconfig, the operator images for the installed version (x.1.1) will be removed from the mirror registry and replaced with the most recent version (x.1.2) in the mirror registry (only if the same channel is updated with a new version, otherwise won't be any impact).

  • It is always preferable to maintain maxVersion as the desired version and minVersion as the installed version, as shown below, to prevent pruning of the current installed version.

    apiVersion: mirror.openshift.io/v1alpha2
kind: ImageSetConfiguration
storageConfig:
  registry:
    imageURL: <registry>:8443/mirror/oc-mirror-metadata                 
    skipTLS: false
mirror:
  platform:
    channels:
    - name: stable-4.14                                             
      type: ocp
      minVersion: 4.14.1  #### Current cluster version is 4.14.1, so specifically mention the version to avoid pruning of current images
      maxversion: 4.14.4  #### Planning to upgrade to the version 4.14.4
      shortestPath: true  #### this argument will be avoid mirroring intermediate version
    graph: true 
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
      packages:
        - name: quay-operator
          channels:
          - name: stable-x
            minVersion: x.1.1
            maxVersion: x.1.2

  • Apart from this oc mirror has another option to skip pruning by adding the flag --skip-pruning while doing the mirror.

What comes next

Following mirroring, the oc mirror will primarily generate two manifest files and one text file. This includes imageContentSourcePolicy,CatalogSource, and mapping.txt. The article's previous section already explains how to use these files. The CatalogSource will only be generated if there is an operator specified in the ImageSetConfiguration for mirroring; otherwise, it won't be generated for platform image mirroring. The specific files will be generated along with a directory with the name ./oc-mirror-workspace/results-xxxxxx/ in the current working directory. Install-config allows to configure the imageContentSourcePolicy for a new cluster installation. Once the cluster is up and running need to apply the created CatalogSource as well. If it's an existing cluster, then directly apply the generated files.

#  oc apply -f ./oc-mirror-workspace/results-xxxx/

Does the imageContentSourcePolicy deprecated

The ImageContentSourcePolicy (ICSP) object has been deprecated as of Openshift 4.13. It is no longer recommended to configure repository mirroring using an ImageContentSourcePolicy (ICSP) object. While deprecated functionality is currently supported and included in the OpenShift Container Platform, it is not advised for new deployments and will be removed in a later version of the product.

What is an ImageDigestMirrorSet..?

Since the ImageContentSourcePolicy (ICSP) is an out-of-date object (from Openshift version 4.13), the ImageDigestMirrorSet will take its place. This CR allows to pull images from a mirrored registry by using digest specifications.

What is an ImageTagMirrorSet..?

The tag based image pulls are not allowed by the deprecated ImageContentSourcePolicy (ICSP). The ImageTagMirrorSet CR enables the functionality to pull the images from the mirror registry using tags.

How to convert imagesoctentsourcepolicy to ImageDigestMirrorSet

Please follow Red Hat article to know more about how to convert deprecated ImageContentSourcePolicy to ImageDigestMirrorSet.

How to create a custom catalog name with oc mirror

Please follow the Red Hat KCS for more information's.

Getting started with oc-mirror v2

  • Why oc-mirror v2 ?

    oc-mirror v2 has been developed to address challenges faced in v1, ensure seamless experience for end users while using oc-mirror and also one of the use cases that are trying to address here is enclave support. Currently this is in TechPreview for Openshift 4.16 and current plans of today is to GA in a later version.

  • How to download oc-mirror v2 binary ?

    There is no separate binary for oc-mirror v2, it is the same as v1 and should follow the same process to download like v1.

  • Is there any changes in the ImageSetConfig from v1 to v2 ?

    Yes, the storageConfig option is completely removed and do not use that any more. If added it will be ignored, instead it uses local cache to serve images more efficiently. Also, the api version is v2alpha1.

  • How to invoke oc-mirror v2 ?

    The same oc-mirror binary can be used with parameter --v2 which will invoke oc-mirror v2.

    # oc-mirror --v2 -h

  • Is there any command to generate sample ImageSetConfig for oc-mirror v2 ?

    Currently there is no option to generate ImageSetConfig like v1 which is oc-mirror --init. However the plan is to have this feature included in future version. In the meanwhile use below ImageSetConfig as a sample one.

    kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
  platform:
    channels:
    - name: stable-4.16
      type: ocp
    graph: true
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.16
    packages:
    - name: advanced-cluster-management
  additionalImages:
  - name: registry.redhat.io/ubi8/ubi:latest
  - name: quay.io/openshifttest/hello-openshift@sha256:4200f438cf2e9446f6bcff9d67ceea1f69ed07a2f83363b7fb52529f7ddd8a83

  • Is dry-run option available in oc-mirror v2 ?

    Yes, --dry-run option is available for v2 as well. see below command for reference. When running dry-run on a mirror to disk workflow, oc-mirror v2 also checks if all the images within the image set are available in its cache. Any missing images are added to the missing.txt file. When a dry-run is performed before mirroring both missing.txt and mapping.txt will contain the same list of images.

    # oc-mirror --config /tmp/isc_dryrun.yaml file://dry-run --dry-run --v2

  • What is new with oc-mirror v2 that users will experience when working with it?

    • Unlike v1, v2 cannot do content discovery, however users could still do the content discovery using v1 commands and it is still valid for v2.

      Note : Command like oc-mirror list releases; oc-mirror list operators.

    • No oc-mirror-workspace will be generated.

      In oc-mirror v1 when custom resources like idms/catalogsource/updateService files are generated they are generated under oc-mirror-workspace whereas in v2 in a new folder called working-dir/cluster-resources and all resources are generated under this.

    • IDMS & ITMS generation.

      • IDMS, ITMS, CatalogSource & UpdateService will be generated under the workspace provided.
      • All images by tag's are added to ITMS while all images by digest are added to IDMS.
      • One ITMS and IDMS is generated, each file will contain several custom resources 1 per category of images mirrored. Unlike v1 it does not need to merge the IDMS files generated and these could be applied cumulatively, just make sure to update the metadata.name so that it does not throw Already exists error

    • Support Incremental Mirroring using date.

      If --since 2024-01-01 is specified oc-mirror v2 will mirror content from this date, if not will mirror everything new from the previously mirrored content. 

      # oc-mirror --v2 -c clid-9.yaml file:///home/ocmirroruser/usertest --since 2024-01-01

    • No Auto-pruning of images.

      If a user would like to delete some operator/images from the registry they will need to generate a delete-images.yaml file and use that yaml file to delete images from the registry. When a user deletes the images only manifests will be deleted from the registry and user/admin needs to run garbage collect on the registry to completely remove it from the registry. Manifests that are common across the releases or operators will not be deleted and only images that are untagged will be removed from the registry when garbage collect is run on the registry.

      • User will need to create a ImageSetConfig.yaml file with kind DeleteImageSetConfiguration with all the images that would like to delete. Example as shown below:
      apiVersion: mirror.openshift.io/v2alpha1
kind: DeleteImageSetConfiguration
delete:
  platform:
    channels:
      - name: stable-4.13 
        minVersion: 4.13.3
        maxVersion: 4.13.3
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.13
      packages:
      - name: 3scale-operator
  additionalImages: 
    - name: registry.redhat.io/ubi8/ubi-minimal:latest
      • The commands to be used for pruning the images using DeleteImageSetConfiguration as follows:
       # oc-mirror delete --config delete-isc.yaml --workspace file://ocmirror --v2 --generate docker://<mirror-registry>/<repository> -> Generates working-dir/delete/delete-images.yaml but does not delete anything
 # oc-mirror delete --v2  --delete-yaml-file working-dir/delete/delete-images.yaml docker://<mirror-registry>/<repository> -> Delete images (Untagging)
      • Run garbage collection at the registry end to completely remove the images.

  • Do oc-mirror v2 generate ICSP and CatalogSource after mirroring ?

    ISCP is deprecated . so IDMS, ITMS & CatalogSource will be generated.

  • How to review errors that happened during mirroring ?

    If mirroring fails, all the errors are displayed on the stdout and at the end oc-mirror v2 generates a text file under working-dir with name mirroring_errors_<date>_<time>.txt which has list of images that failed to mirror and why.

Common issues

Article Type