Procedure

1. The ClusterServiceVersion object is Operator Lifecycle Manager’s representation of a running Operator in the cluster. Find the Red Hat Quay Operator’s ClusterServiceVersion by using a Kubernetes UI or the kubectl/oc CLI tool. For example:

   $ oc get clusterserviceversions -n <namespace>

   Copy code to clipboard

2. Using the UI, oc edit, or another method, modify the ClusterServiceVersion object to include the environment variables outlined above to point to the override images:

   JSONPath: spec.install.spec.deployments[0].spec.template.spec.containers[0].env

   - name: RELATED_IMAGE_COMPONENT_QUAY
     value: quay.io/projectquay/quay@sha256:c35f5af964431673f4ff5c9e90bdf45f19e38b8742b5903d41c10cc7f6339a6d
   - name: RELATED_IMAGE_COMPONENT_CLAIR
     value: quay.io/projectquay/clair@sha256:70c99feceb4c0973540d22e740659cd8d616775d3ad1c1698ddf71d0221f3ce6
   - name: RELATED_IMAGE_COMPONENT_POSTGRES
     value: centos/postgresql-10-centos7@sha256:de1560cb35e5ec643e7b3a772ebaac8e3a7a2a8e8271d9e91ff023539b4dfb33
   - name: RELATED_IMAGE_COMPONENT_REDIS
     value: centos/redis-32-centos7@sha256:06dbb609484330ec6be6090109f1fa16e936afcf975d1cbc5fff3e6c7cae7542

   Copy code to clipboard

Procedure

1. Deploy a PostgreSQL instance for Red Hat Quay.

2. Log in to the database by entering the following command:

   psql -U <username> -h <hostname> -p <port> -d <database_name>

   Copy code to clipboard

3. Create a database for Red Hat Quay named quay. For example:

   CREATE DATABASE quay;

   Copy code to clipboard

4. Enable the pg_trgm extension inside the database:

   \c quay;
   CREATE EXTENSION IF NOT EXISTS pg_trgm;

   Copy code to clipboard

5. Deploy a Redis instance:

   Note

   • Deploying a Redis instance might be unnecessary if your cloud provider has its own service.
   • Deploying a Redis instance is required if you are leveraging Builders.
   1. Deploy a VM for Redis.
   2. Verify that the Redis instance is accessible from the clusters where Red Hat Quay is running.
   3. Port 6379/TCP must be open.

   4. Run Redis inside the instance:

      sudo dnf install -y podman
      podman run -d --name redis -p 6379:6379 redis

      Copy code to clipboard
6. Create two object storage backends, one for each cluster. Ideally, one object storage bucket is close to the first, or primary, cluster, and the other runs closer to the second, or secondary, cluster.
7. Deploy the clusters with the same config bundle, using environment variable overrides to select the appropriate storage backend for an individual cluster.
8. Configure a load balancer to provide a single entry point to the clusters.

Procedure

1. Create a config.yaml file that is shared between clusters. This config.yaml file contains the details for the common PostgreSQL, Redis and storage backends:

   Geo-replication config.yaml file

   SERVER_HOSTNAME: <georep.quayteam.org or any other name>
   DB_CONNECTION_ARGS:
     autorollback: true
     threadlocals: true
   DB_URI: postgresql://postgres:password@10.19.0.1:5432/quay
   BUILDLOGS_REDIS:
     host: 10.19.0.2
     port: 6379
   USER_EVENTS_REDIS:
     host: 10.19.0.2
     port: 6379
   DATABASE_SECRET_KEY: 0ce4f796-c295-415b-bf9d-b315114704b8
   DISTRIBUTED_STORAGE_CONFIG:
     usstorage:
       - GoogleCloudStorage
       - access_key: GOOGQGPGVMASAAMQABCDEFG
         bucket_name: georep-test-bucket-0
         secret_key: AYWfEaxX/u84XRA2vUX5C987654321
         storage_path: /quaygcp
     eustorage:
       - GoogleCloudStorage
       - access_key: GOOGQGPGVMASAAMQWERTYUIOP
         bucket_name: georep-test-bucket-1
         secret_key: AYWfEaxX/u84XRA2vUX5Cuj12345678
         storage_path: /quaygcp
   DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
     - usstorage
     - eustorage
   DISTRIBUTED_STORAGE_PREFERENCE:
     - usstorage
     - eustorage
   FEATURE_STORAGE_REPLICATION: true

   Copy code to clipboard

   where:

   SERVER_HOSTNAME

   Specifies the hostname of the global load balancer. Must match the hostname of the global load balancer.

2. Create the configBundleSecret custom resource (CR) by entering the following command:

   $ oc create secret generic --from-file config.yaml=./config.yaml georep-config-bundle

   Copy code to clipboard

3. In each of the clusters, set the configBundleSecret and use the QUAY_DISTRIBUTED_STORAGE_PREFERENCE environmental variable override to configure the appropriate storage for that cluster. For example:

   Note

   The config.yaml file between both deployments must match. If making a change to one cluster, it must also be changed in the other.

   US cluster QuayRegistry example

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: example-registry
     namespace: quay-enterprise
   spec:
     configBundleSecret: georep-config-bundle
     components:
       - kind: objectstorage
         managed: false
       - kind: route
         managed: true
       - kind: tls
         managed: false
       - kind: postgres
         managed: false
       - kind: clairpostgres
         managed: false
       - kind: redis
         managed: false
       - kind: quay
         managed: true
         overrides:
           env:
           - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
             value: usstorage
       - kind: mirror
         managed: true
         overrides:
           env:
           - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
             value: usstorage

   Copy code to clipboard

   Note

   Because SSL/TLS is unmanaged, and the route is managed, you must supply the certificates directly in the config bundle. For more information, see Configuring SSL/TLS and Routes.

   European cluster

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: example-registry
     namespace: quay-enterprise
   spec:
     configBundleSecret: georep-config-bundle
     components:
       - kind: objectstorage
         managed: false
       - kind: route
         managed: true
       - kind: tls
         managed: false
       - kind: postgres
         managed: false
       - kind: clairpostgres
         managed: false
       - kind: redis
         managed: false
       - kind: quay
         managed: true
         overrides:
           env:
           - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
             value: eustorage
       - kind: mirror
         managed: true
         overrides:
           env:
           - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
             value: eustorage

   Copy code to clipboard

   Note

   Because SSL/TLS is unmanaged, and the route is managed, you must supply the certificates directly in the config bundle. For more information, see Configuring SSL and TLS for Red Hat Quay.

Procedure

1. On System B and System C, scale down your Red Hat Quay registry. This is done by disabling auto scaling and overriding the replica county for Red Hat Quay, mirror workers, and Clair if it is managed. Note that you must keep the Red Hat Quay registry running on System A. Do not update the quayregistry.yaml file on System A.

   Use the following quayregistry.yaml file as a reference:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: registry
     namespace: ns
   spec:
     components:
       …
       - kind: horizontalpodautoscaler
         managed: false
       - kind: quay
         managed: true
         overrides:
           replicas: 0
       - kind: clair
         managed: true
         overrides:
           replicas: 0
       - kind: mirror
         managed: true
         overrides:
           replicas: 0
       …

   Copy code to clipboard

   where:

   managed: false

   Disables auto scaling of Quay, Clair and Mirroring workers

   overrides

   Sets the replica count to 0 for components accessing the database and objectstorage

2. Wait for the registry-quay-app, registry-quay-mirror, and registry-clair-app pods to disappear. Enter the following command to check their status:

   oc get pods -n <quay-namespace>

   Copy code to clipboard

   Example output

   quay-operator.v3.7.1-6f9d859bd-p5ftc               1/1     Running     0             12m
   quayregistry-clair-postgres-7487f5bd86-xnxpr       1/1     Running     1 (12m ago)   12m
   quayregistry-quay-app-upgrade-xq2v6                0/1     Completed   0             12m
   quayregistry-quay-redis-84f888776f-hhgms           1/1     Running     0             12m

   Copy code to clipboard

3. On System A, initiate a Red Hat Quay upgrade to the latest y-stream version. This is a manual process. For more information about upgrading installed Operators, see Upgrading installed Operators. For more information about Red Hat Quay upgrade paths, see Upgrading the Red Hat Quay Operator.
4. After the new Red Hat Quay registry is installed, the necessary upgrades on the cluster are automatically completed. Afterwards, new Red Hat Quay pods are started with the latest y-stream version. Additionally, new Quay pods are scheduled and started.

5. Confirm that the update has properly worked by navigating to the Red Hat Quay UI:

   1. In the OpenShift console, navigate to Operators → Installed Operators, and click the Registry Endpoint link.

      Important

      Do not execute the following step until the Red Hat Quay UI is available. Do not upgrade the Red Hat Quay registry on System B and on System C until the UI is available on System A.

6. Confirm that the update has properly worked on System A, initiate the Red Hat Quay upgrade on System B and on System C. The Operator upgrade results in an upgraded Red Hat Quay installation, and the pods are restarted.

   Note

   Because the database schema is correct for the new y-stream installation, the new pods on System B and on System C should quickly start.

7. After updating, revert the changes made in step 1 of this procedure by removing overrides for the components. For example:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: registry
     namespace: ns
   spec:
     components:
       …
       - kind: horizontalpodautoscaler
         managed: true
       - kind: quay
         managed: true
       - kind: clair
         managed: true
       - kind: mirror
         managed: true
       …

   Copy code to clipboard

   where:

   kind: horizontalpodautoscaler:: Set this resource to True if the horizontalpodautoscaler resource was set to True before the upgrade procedure, or if you want Red Hat Quay to scale in case of a resource shortage.

Procedure

1. Sync the blobs between all of your defined sites by running the following command:

   $ python -m util.backfillreplication

   Copy code to clipboard
   Warning

   Prior to removing storage engines from your Red Hat Quay config.yaml file, you must ensure that all blobs are synced between all defined sites.

   When running this command, replication jobs are created which are picked up by the replication worker. If blobs need to be replicated, the script returns UUIDs of blobs that are replicated. If you run this command multiple times, and the output from the return script is empty, it does not mean that the replication process is done; it means that no more blobs remain to be queued for replication. Customers should use appropriate judgement before proceeding, as the allotted time replication takes depends on the number of blobs detected.

   Alternatively, you could use a third party cloud tool, such as Microsoft Azure, to check the synchronization status.

   This step must be completed before proceeding.

2. In your Red Hat Quay config.yaml file for site usstorage, remove the DISTRIBUTED_STORAGE_CONFIG entry for the eustorage site.

3. Identify your Red Hat Quay application pods by entering the following command:

   $ oc get pod -n <quay_namespace>

   Copy code to clipboard

   Example output:

   quay390usstorage-quay-app-5779ddc886-2drh2
   quay390eustorage-quay-app-66969cd859-n2ssm

   Copy code to clipboard

4. Open an interactive shell session in the usstorage pod by entering the following command:

   $ oc rsh quay390usstorage-quay-app-5779ddc886-2drh2

   Copy code to clipboard

5. Permanently remove the eustorage site by entering the following command:

   Important

   The following action cannot be undone. Use with caution.

   sh-4.4$ python -m util.removelocation eustorage

   Copy code to clipboard

   Example output:

   WARNING: This is a destructive operation. Are you sure you want to remove eustorage from your storage locations? [y/n] y
   Deleted placement 30
   Deleted placement 31
   Deleted placement 32
   Deleted placement 33
   Deleted location eustorage

   Copy code to clipboard

Procedure

1. Enter the following command to obtain a list of Red Hat Quay pods:

   $ oc get pods -n <namespace>

   Copy code to clipboard

   Example output

   example-registry-clair-app-7dc7ff5844-4skw5           0/1     Error                    0             70d
   example-registry-clair-app-7dc7ff5844-nvn4f           1/1     Running                  0             31d
   example-registry-clair-app-7dc7ff5844-x4smw           0/1     ContainerStatusUnknown   6 (70d ago)   70d
   example-registry-clair-app-7dc7ff5844-xjnvt           1/1     Running                  0             60d
   example-registry-clair-postgres-547d75759-75c49       1/1     Running                  0             70d
   example-registry-quay-app-76c8f55467-52wjz            1/1     Running                  0             70d
   example-registry-quay-app-76c8f55467-hwz4c            1/1     Running                  0             70d
   example-registry-quay-app-upgrade-57ghs               0/1     Completed                1             70d
   example-registry-quay-database-7c55899f89-hmnm6       1/1     Running                  0             70d
   example-registry-quay-mirror-6cccbd76d-btsnb          1/1     Running                  0             70d
   example-registry-quay-mirror-6cccbd76d-x8g42          1/1     Running                  0             70d
   example-registry-quay-redis-85cbdf96bf-4vk5m          1/1     Running                  0             70d

   Copy code to clipboard

2. Open a remote shell session to the Quay container by entering the following command:

   $ oc rsh example-registry-quay-app-76c8f55467-52wjz

   Copy code to clipboard

3. Create the necessary service keys by entering the following command:

   sh-4.4$ python3 tools/generatekeypair.py quay-readonly

   Copy code to clipboard

   Example output

   Writing public key to quay-readonly.jwk
   Writing key ID to quay-readonly.kid
   Writing private key to quay-readonly.pem

   Copy code to clipboard

Procedure

1. Enter the following command to enter your Red Hat Quay database environment:

   $ podman exec -it postgresql-quay psql -U postgres -d quay

   Copy code to clipboard

2. Display the approval types and associated notes of the servicekeyapproval by entering the following command:

   quay=# select * from servicekeyapproval;

   Copy code to clipboard

   Example output:

    id | approver_id |          approval_type           |       approved_date        | notes
   ----+-------------+----------------------------------+----------------------------+-------
     1 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:48.181347 |
     2 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:55.808087 |
     3 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:04.27095  |
     4 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:05.46235  |
     5 |           1 | ServiceKeyApprovalType.SUPERUSER | 2024-05-07 04:05:10.296796 |
   ...

   Copy code to clipboard

3. Add the service key to your Red Hat Quay database by entering the following query:

   quay=# INSERT INTO servicekey
     (name, service, metadata, kid, jwk, created_date, expiration_date)
     VALUES ('quay-readonly',
              'quay',
              '{}',
              '<contents_of_.kid_file>',
              '<contents_of_.jwk_file>',
              '<created_date_of_read-only>',
              '<expiration_date_of_read-only>');

   Copy code to clipboard

   Example output:

   INSERT 0 1

   Copy code to clipboard

4. Next, add the key approval with the following query:

   quay=# INSERT INTO servicekeyapproval ('approval_type', 'approved_date', 'notes')
     VALUES ("ServiceKeyApprovalType.SUPERUSER", "CURRENT_DATE",
              <include_notes_here_on_why_this_is_being_added>);

   Copy code to clipboard

   Example output:

   INSERT 0 1

   Copy code to clipboard

5. Set the approval_id field on the created service key row to the id field from the created service key approval. You can use the following SELECT statements to get the necessary IDs:

   UPDATE servicekey
   SET approval_id = (SELECT id FROM servicekeyapproval WHERE approval_type = 'ServiceKeyApprovalType.SUPERUSER')
   WHERE name = 'quay-readonly';

   Copy code to clipboard

   UPDATE 1

   Copy code to clipboard

Procedure

1. Read the secret name of your Red Hat Quay on OpenShift Container Platform deployment by entering the following command:

   $ oc get deployment -o yaml <quay_main_app_deployment_name>

   Copy code to clipboard

2. Use the base64 command to encode the quay-readonly.kid and quay-readonly.pem files by entering the following commands:

   $ base64 -w0 quay-readonly.kid

   Copy code to clipboard

   Example output

   ZjUyNDFm...

   Copy code to clipboard

   $ base64 -w0 quay-readonly.pem

   Copy code to clipboard

   Example output

   <example_secret>...

   Copy code to clipboard

3. Obtain the current configuration bundle and secret by entering the following command. Save the output to a file called config.yaml:

   $ oc get secret quay-config-secret-name -o json | jq '.data."config.yaml"' | cut -d '"' -f2 | base64 -d -w0 > config.yaml

   Copy code to clipboard

4. Edit the config.yaml file and add the following information to enable read-only mode:

   # ...
   REGISTRY_STATE: readonly
   INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid'
   INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
   # ...

   Copy code to clipboard

5. Save the file and base64 encode it by entering the following command:

   $ base64 -w0 quay-config.yaml

   Copy code to clipboard

6. Scale down the Red Hat Quay Operator pods to 0 by entering the following command. This ensures that the Operator does not reconcile the secret after editing it.

   $ oc scale --replicas=0 deployment quay-operator -n openshift-operators

   Copy code to clipboard

7. Edit the secret to include the new content by entering the following command:

   $ oc edit secret quay-config-secret-name -n quay-namespace

   Copy code to clipboard

   # ...
   data:
     "quay-readonly.kid": "ZjUyNDFm..."
     "quay-readonly.pem": "<example_secret>..."
     "config.yaml": "QUNUSU9OX0xPR19..."
   # ...

   Copy code to clipboard

   With your Red Hat Quay on OpenShift Container Platform deployment on read-only mode, you can safely manage your registry’s operations and perform such actions as backup and restore.

Procedure

1. Edit the config.yaml file and remove the following information:

   # ...
   REGISTRY_STATE: readonly
   INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid'
   INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
   # ...

   Copy code to clipboard

2. Scale the Red Hat Quay Operator back up by entering the following command:

   $ oc scale --replicas=1 deployment quay-operator -n openshift-operators

   Copy code to clipboard

Procedure

1. To back the QuayRegistry custom resource by exporting it, enter the following command:

   $ oc get quayregistry <quay_registry_name> -n <quay_namespace> -o yaml > quay-registry.yaml

   Copy code to clipboard

2. Edit the resulting quayregistry.yaml and remove the status section and the following metadata fields:

     metadata.creationTimestamp
     metadata.finalizers
     metadata.generation
     metadata.resourceVersion
     metadata.uid

   Copy code to clipboard

3. Backup the managed keys secret by entering the following command:

   Note

   If you are running a version older than Red Hat Quay 3.7.0, this step can be skipped. Some secrets are automatically generated while deploying Red Hat Quay for the first time. These are stored in a secret called <quay_registry_name>-quay-registry-managed-secret-keys in the namespace of the QuayRegistry resource.

   $ oc get secret -n <quay_namespace> <quay_registry_name>-quay-registry-managed-secret-keys -o yaml > managed_secret_keys.yaml

   Copy code to clipboard

4. Edit the resulting managed_secret_keys.yaml file and remove the entry metadata.ownerReferences. Your managed_secret_keys.yaml file should look similar to the following:

   apiVersion: v1
   kind: Secret
   type: Opaque
   metadata:
     name: <quay_registry_name>-quay-registry-managed-secret-keys
     namespace: <quay_namespace>
   data:
     CONFIG_EDITOR_PW: <redacted>
     DATABASE_SECRET_KEY: <redacted>
     DB_ROOT_PW: <redacted>
     DB_URI: <redacted>
     SECRET_KEY: <redacted>
     SECURITY_SCANNER_V4_PSK: <redacted>

   Copy code to clipboard

   All information under the data property should remain the same.

5. Redirect the current Quay configuration file by entering the following command:

   $ oc get secret -n <quay-namespace>  $(oc get quayregistry <quay_registry_name> -n <quay_namespace>  -o jsonpath='{.spec.configBundleSecret}') -o yaml > config-bundle.yaml

   Copy code to clipboard

6. Backup the /conf/stack/config.yaml file mounted inside of the Quay pods:

   $ oc exec -it quay_pod_name -- cat /conf/stack/config.yaml > quay_config.yaml

   Copy code to clipboard

7. Obtain the Quay database name:

   $ oc -n <quay_namespace> rsh $(oc get pod -l app=quay -o NAME -n <quay_namespace> |head -n 1) cat /conf/stack/config.yaml|awk -F"/" '/^DB_URI/ {print $4}'

   Copy code to clipboard

   Example output

   quayregistry-quay-database

   Copy code to clipboard

Procedure

1. Scale down the Red Hat Quay deployment by disabling auto scaling and overriding the replica count for Red Hat Quay, mirror workers, and Clair (if managed). For example:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: registry
     namespace: ns
   spec:
     components:
       …
       - kind: horizontalpodautoscaler
         managed: false
       - kind: quay
         managed: true
         overrides:
           replicas: 0
       - kind: clair
         managed: true
         overrides:
           replicas: 0
       - kind: mirror
         managed: true
         overrides:
           replicas: 0
       …

   Copy code to clipboard

   where:

   managed: false

   Disables auto scaling of Quay, Clair and Mirroring workers.

   overrides

   Sets the replica count to 0 for components accessing the database and objectstorage.

2. Wait for the registry-quay-app, registry-quay-mirror and registry-clair-app pods (depending on which components you set to be managed by the Red Hat Quay Operator) to disappear. You can check their status by entering the following command:

   $ oc get pods -n <quay_namespace>

   Copy code to clipboard

   Example output:

   $ oc get pod

   Copy code to clipboard

   Example output

   quay-operator.v3.7.1-6f9d859bd-p5ftc               1/1     Running     0             12m
   quayregistry-clair-postgres-7487f5bd86-xnxpr       1/1     Running     1 (12m ago)   12m
   quayregistry-quay-app-upgrade-xq2v6                0/1     Completed   0             12m
   quayregistry-quay-database-859d5445ff-cqthr        1/1     Running     0             12m
   quayregistry-quay-redis-84f888776f-hhgms           1/1     Running     0             12m

   Copy code to clipboard

Procedure

1. Identify the Red Hat Quay PostgreSQL pod name by entering the following command:

   $ oc get pod -l quay-component=postgres -n <quay_namespace> -o jsonpath='{.items[0].metadata.name}'

   Copy code to clipboard

   Example output:

   quayregistry-quay-database-59f54bb7-58xs7

   Copy code to clipboard

2. Download a backup database by entering the following command:

   $ oc -n <quay_namespace> exec quayregistry-quay-database-59f54bb7-58xs7 -- /usr/bin/pg_dump -C quayregistry-quay-database  > backup.sql

   Copy code to clipboard

Procedure

1. Decode and export the AWS_ACCESS_KEY_ID by entering the following command:

   $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)

   Copy code to clipboard

2. Decode and export the AWS_SECRET_ACCESS_KEY_ID by entering the following command:

   $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)

   Copy code to clipboard

3. Create a new directory by entering the following command:

   $ mkdir blobs

   Copy code to clipboard

4. Copy all blobs to the directory by entering the following command:

   $ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}')  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}') ./blobs

   Copy code to clipboard

Procedure

1. Scale up the Red Hat Quay deployment by re-enabling auto scaling, if desired, and removing the replica overrides for Quay, mirror workers and Clair as applicable. For example:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: registry
     namespace: ns
   spec:
     components:
       …
       - kind: horizontalpodautoscaler
         managed: true
       - kind: quay
         managed: true
       - kind: clair
         managed: true
       - kind: mirror
         managed: true
       …

   Copy code to clipboard

   where:

   spec.components.horizontalpodautoscaler.managed

   Specifies re-enabling auto scaling of Quay, Clair and Mirroring workers again.

   spec.components.quay

   Specifies that replica overrides are removed again to scale the Quay components back up.

2. Check the status of the Red Hat Quay deployment by entering the following command:

   $ oc wait quayregistry registry --for=condition=Available=true -n <quay_namespace>

   Copy code to clipboard

   Example output:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     ...
     name: registry
     namespace: <quay-namespace>
     ...
   spec:
     ...
   status:
     - lastTransitionTime: '2022-06-20T05:31:17Z'
       lastUpdateTime: '2022-06-20T17:31:13Z'
       message: All components reporting as healthy
       reason: HealthChecksPassing
       status: 'True'
       type: Available

   Copy code to clipboard

Procedure

1. Restore the backed up Red Hat Quay configuration by entering the following command:

   $ oc create -f ./config-bundle.yaml

   Copy code to clipboard
   Important

   If you receive the error Error from server (AlreadyExists): error when creating "./config-bundle.yaml": secrets "config-bundle-secret" already exists, you must delete your existing resource with $ oc delete Secret config-bundle-secret -n <quay-namespace> and recreate it with $ oc create -f ./config-bundle.yaml.

2. Restore the generated keys from the backup by entering the following command:

   $ oc create -f ./managed-secret-keys.yaml

   Copy code to clipboard

3. Restore the QuayRegistry custom resource by entering the following command:

   $ oc create -f ./quay-registry.yaml

   Copy code to clipboard

4. Check the status of the Red Hat Quay deployment by entering the following command. Wait for it to be available:

   $ oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace>

   Copy code to clipboard

Procedure

1. Scale down the Red Hat Quay deployment by disabling auto scaling and overriding the replica count for Quay, mirror workers and Clair (if managed). For example:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: registry
     namespace: ns
   spec:
     components:
       …
       - kind: horizontalpodautoscaler
         managed: false
       - kind: quay
         managed: true
         overrides:
           replicas: 0
       - kind: clair
         managed: true
         overrides:
           replicas: 0
       - kind: mirror
         managed: true
         overrides:
           replicas: 0
       …

   Copy code to clipboard

   where:

   spec.components.horizontalpodautoscaler.managed

   Specifies that the component is not managed by the Red Hat Quay Operator.

   spec.components.quay.overrides

   Specifies the replica count for the component.

2. Wait for the registry-quay-app, registry-quay-mirror and registry-clair-app pods (depending on which components you set to be managed by Red Hat Quay Operator) to disappear. You can check their status by running the following command:

   $ oc get pods -n <quay-namespace>

   Copy code to clipboard

   Example output:

   registry-quay-config-editor-77847fc4f5-nsbbv   1/1     Running            0          9m1s
   registry-quay-database-66969cd859-n2ssm        1/1     Running            0          6d1h
   registry-quay-redis-7cc5f6c977-956g8           1/1     Running            0          5d21h

   Copy code to clipboard

Procedure

1. Identify your Quay database pod by entering the following command:

   $ oc get pod -l quay-component=postgres -n  <quay_namespace> -o jsonpath='{.items[0].metadata.name}'

   Copy code to clipboard

   Example output:

   quayregistry-quay-database-59f54bb7-58xs7

   Copy code to clipboard

2. Upload the backup by copying it from the local environment and into the pod by entering the following command:

   $ oc cp ./backup.sql -n <quay_namespace> registry-quay-database-66969cd859-n2ssm:/tmp/backup.sql

   Copy code to clipboard

3. Open a remote terminal to the database by entering the following command:

   $ oc rsh -n <quay_namespace> registry-quay-database-66969cd859-n2ssm

   Copy code to clipboard

4. Enter psql by entering the following command:

   bash-4.4$ psql

   Copy code to clipboard

5. You can list the database by entering the following command:

   postgres=# \l

   Copy code to clipboard

   Example output

                                                     List of databases
              Name            |           Owner            | Encoding |  Collate   |   Ctype    |   Access privileges
   ----------------------------+----------------------------+----------+------------+------------+-----------------------
   postgres                   | postgres                   | UTF8     | en_US.utf8 | en_US.utf8 |
   quayregistry-quay-database | quayregistry-quay-database | UTF8     | en_US.utf8 | en_US.utf8 |

   Copy code to clipboard

6. Drop the existing database by entering the following command:

   postgres=# DROP DATABASE "quayregistry-quay-database";

   Copy code to clipboard

   Example output

   DROP DATABASE

   Copy code to clipboard

7. Exit the postgres CLI by entering the following command:

   \q

   Copy code to clipboard

8. Redirect your PostgreSQL database to your backup database by entering the following command:

   sh-4.4$ psql < /tmp/backup.sql

   Copy code to clipboard

9. Exit bash by entering the following command:

   sh-4.4$ exit

   Copy code to clipboard

Procedure

1. Export the AWS_ACCESS_KEY_ID by entering the following command:

   $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)

   Copy code to clipboard

2. Export the AWS_SECRET_ACCESS_KEY by entering the following command:

   $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)

   Copy code to clipboard

3. Upload all blobs to the bucket by running the following command:

   $ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}') ./blobs  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}')

   Copy code to clipboard

Procedure

1. Enter the following command to query for detected vulnerabilities:

   $ oc get vuln --all-namespaces

   Copy code to clipboard

   Example output

   NAMESPACE     NAME              AGE
   default       sha256.ca90...    6m56s
   skynet        sha256.ca90...    9m37s

   Copy code to clipboard

2. Optional. To display details for a particular vulnerability, identify a specific vulnerability and its namespace, and use the oc describe command. The following example shows an active container whose image includes an RPM package with a vulnerability:

   $ oc describe vuln --namespace <namespace> sha256.ac50e3752...

   Copy code to clipboard