Installing OpenShift Virtualization on a restricted network cluster

When OpenShift Container Platform is installed on restricted networks, also known as a disconnected cluster¹, Operator Lifecycle Manager (OLM) can no longer use the default OperatorHub sources because they require full Internet connectivity. Cluster administrators can disable those default sources and create local mirrors so that OLM can install and manage Operators from the local sources instead.

The official tool “oc adm catalog build” can be used to create a catalog image containing all operators metadata of a given AppRegistry namespace.
The official version of Openshift Virtualization is contained in redhat-operators AppRegistry namespace. This namespace is also public and does not require any authentication.

Use the following command to create a mirrored catalog image in the local registry accessible by the disconnected cluster:

$ oc adm catalog build --appregistry-org $APP_REGISTRY \ 
--from=registry.redhat.io/openshift4/ose-operator-registry:v4.4 \
--filter-by-os="linux/amd64" \
--to=$LOCAL_REGISTRY:$PORT/olm/$APP_REGISTRY:v1 \
[--insecure] \
[--auth-token="${QUAY_TOKEN}"]

Example:

$ oc adm catalog build --appregistry-org redhat-operators \
--from=registry.redhat.io/openshift4/ose-operator-registry:v4.4 \
--to=my-registry.local:5000/olm/redhat-operators:v1 \
 --filter-by-os="linux/amd64"

Pre-released version

We provide partners the ability to consume pre-released versions of OpenShift Virtualization, using a dedicated quay.io repository. That repository contains a custom OLM bundle image listing all metadata and images related to OpenShift Virtualization operator.
Today the builds have only a very basic test coverage, but we are working on CI to ensure proper testing before the images get published; a floating tag will point to the most recent successfully tested hco-bundle-registry image in order to allow the user to deploy the most up to date and successfully tested OpenShift Virtualization 2.4.0 version. The image is available at:
quay.io/openshift-cnv/container-native-virtualization-hco-bundle-registry:pass-basic-sanity-24

A quay.io account has been created for customers and partners to consume the pre-released content - openshift-cnv+beta

1. Login to quay.io to get access to openshift-cnv organization in quay.io:

    $ podman login quay.io --username openshift-cnv+beta --password ${YOUR_GIVEN_PASSWORD}
   

2. Mirror the bundle image to your local registry:

    $ oc image mirror quay.io/openshift-cnv/container-native-virtualization-hco-bundle-registry:pass-basic-sanity-24 ${LOCAL_REGISTRY}:${PORT}/cnv/hco-bundle-registry:pass-basic-sanity-24
   

Mirroring the content of the catalog to your local registry

In order to mirror the images in the catalog to be accessible to the disconnected cluster, use the following official command:

$ oc adm catalog mirror ${LOCAL_REGISTRY}:${PORT}/olm/${APP_REGISTRY}:v1 ${LOCAL_REGISTRY}:${PORT} [--insecure] [-a $REG_CREDS]

For mirroring the pre-released images, use the bundle image you have mirrored in the previous step:

$ oc adm catalog mirror ${LOCAL_REGISTRY}:${PORT}/cnv/hco-bundle-registry:pass-basic-sanity-24 ${LOCAL_REGISTRY}:${PORT} [--insecure] [-a $REG_CREDS]

Notes:

• Use the --insecure flag if the local registry is not configured with TLS. In such case, the insecure local registry should be added to the custom resource image.config.openshift.io/cluster in spec.registrySources.insecureRegistries.² It can be patched by using the following command:

oc patch image.config.openshift.io/cluster --type json -p \
"[{\"op\": \"add\", \"path\": \"/spec/registrySources/insecureRegistries/-\", \"value\": \"$LOCAL_REGISTRY:$PORT\"}]"

• Use the -a argument and provide the credentials for the local registry if it requires authentication (“auth.json”).

Example:

$ oc adm catalog mirror my-registry.local:5000/olm/redhat-operators:v1 my-registry.local:5000

The command mirrors all images included in the provided AppRegistry, to the local registry and outputs two files: imageContentSourcePolicy.yaml and mapping.txt.

The imageContentSourcePolicy.yaml file defines an object that should be applied on the cluster and configures the nodes to translate between the original images and the mirrored images in the disconnected network that should be pulled. It applies a MachineConfig on all nodes, configuring the mirrors for each image in /etc/containers/registries.conf, which in turn reboots all nodes one by one.

$ oc apply -f ${APP_REGISTRY}_manifests/imageContentSourcePolicy.yaml

Disable the default (online) OperatorSource/CatalogSource on the disconnected cluster:

$ oc patch OperatorHub cluster --type json \
    -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'

Configure the mirrored catalog source:

Edit the following CatalogSource manifest and set spec.image to the mirrored catalog image on your local registry, and create it on the disconnected cluster:

cat <<EOF | oc create -f -
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: my-mirrored-catalog-source
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: <catalog image location>
  displayName: My Mirrored Catalog Source
  publisher: grpc
EOF

When the mirrored catalog source has been processed by the OLM, Openshift Virtualization operator is available in OperatorHub and can be installed on the cluster following the official documentation.³

Upgrade from a locally mirrored installation to an Online installation

If the cluster has an Internet connection, it is possible to upgrade to a regular online installation without interrupting the Openshift Virtualization workloads.

1. Delete the mirrored CatalogSource and the ImageContentSourcePolicy that were created in the installation process.
   These resources can be found by:

    $ oc get catsrc -n openshift-marketplace
    $ oc get ImageContentSourcePolicy
   

2. Re-enable default Operator Sources:

    $ oc patch OperatorHub cluster --type json \
        -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": false}]'
   

3. Verify that the package manifests was successfully processed by the OLM:

    $ oc get packagemanifest -n openshift-marketplace kubevirt-hyperconverged
   

4. Navigate to “Installed Operators” in “openshift-cnv” namespace, and uninstall OpenShift Virtualization operator.
   Note: Do not delete the HyperConverged operand.

5. Navigate to “OperatorHub”, search for OpenShift Virtualization and install the operator.

6. All pods in openshift-cnv namespace are now being rolled out to the upgraded online version.

7. Upgrade is finished when:

• All Pods in openshift-cnv are in ready state.

$ oc get pods -n openshift-cnv
NAME                                                  READY   STATUS    RESTARTS   AGE
bridge-marker-4rc7b                                   1/1     Running   0          3m3s
bridge-marker-csgqp                                   1/1     Running   0          3m3s
bridge-marker-j9tsz                                   1/1     Running   0          3m3s
bridge-marker-mtrvv                                   1/1     Running   0          3m3s
bridge-marker-pkx8x                                   1/1     Running   0          3m3s
bridge-marker-q8lf7                                   1/1     Running   0          3m3s
cdi-apiserver-57b469b699-8w975                        1/1     Running   0          4h39m
cdi-deployment-7f59c8d85-ttklc                        1/1     Running   0          4h39m
cdi-operator-844bfd44b8-mn9xm                         1/1     Running   0          3m22s
cdi-uploadproxy-66dd8d79cf-2fjcs                      1/1     Running   0          4h39m
cluster-network-addons-operator-7b7d67474b-w6wx5      1/1     Running   0          3m23s
 ---trunc----

• The phase of the CSV of OpenShift Virtualization operator is “Succeeded”.

$ oc get csv -n openshift-cnv
NAME                                      DISPLAY                    VERSION   REPLACES   PHASE
kubevirt-hyperconverged-operator.v2.4.0   OpenShift Virtualization   2.4.0                Succeeded

• The HyperConverged custom resource have the following conditions (oc get hco kubevirt-hyperconverged -n openshift-cnv):

Update a mirrored CatalogSource

In order to update the mirror Catalog Image, while remain in a disconnected cluster, follow same steps for creating the Catalog Image using oc adm catalog build, mirroring the images onto the local registry using oc adm catalog mirror and apply the new catalog source and generated imageContentSourcePolicy.yaml on the cluster.⁴