Troubleshooting Quay Geo-Replication

Updated 24 Jul 2023

Introduction

Geo-replication is a feature of Red Hat Quay that enables the synchronization of container images across multiple geographically distributed instances of Quay.
With Quay Geo-replication, organizations can create replicas or copies of their container image repositories in different geographical regions. This helps improve the availability and performance of container image distribution, particularly in scenarios where users or applications are spread across multiple regions.

Troubleshooting Steps

Check debug logs from all Quay pods/containers from all regions. Look for any relevant error messages or warnings related to storage replication:
```
$ oc logs quay-pod

$ podman logs quay-container
```

Check Geo-replication configurations from Quay config.yaml file. Make sure same configuration file is used across all the regions.

    
    $ oc exec -it quay-pod -- cat /conf/stack/config.yaml

    $ podman exec -it quay-container cat /conf/stack/config.yaml

Check if data is replicated in all backend buckets correctly using Content from docs.aws.amazon.com is not included.aws cli commands or Content from docs.aws.amazon.com is not included.s3
```
$ aws --profile quay_prod_s3 --endpoint=http://10.0.x.x:port s3 ls ocp-quay --recursive --human-readable --summarize
.
.
Total Objects: 17996
Total Size: 514.4 GiB
```
Check if all backend storages are up and running.

Amazon Web Service Storage (AWS):

Check the AWS S3 service health status on the Content from status.aws.amazon.com is not included.AWS Service Health Dashboard
Validate your access to S3 by listing objects in a known bucket using the AWS CLI or SDKs.
Google Cloud Storage (GCS):

Check the Content from status.cloud.google.com is not included.Google Cloud Status Dashboard for the status of the GCS service. Verify your access to GCS by listing objects in a known bucket using the Google Cloud SDK or GCS client libraries.
NooBaa:

Check the NooBaa management console or administrative interface for any health or status indicators. Ensure that the NooBaa services and related components are running and accessible. Verify access to NooBaa by listing objects in a known bucket using the NooBaa CLI or SDK.
OpenShift Container Storage (OCS):

Check the OpenShift Console or management interface for the status of the OCS components. Verify the availability of OCS S3 interface and services. Ensure that the OCS services are running and accessible. Validate access to OCS S3 by listing objects in a known bucket using the appropriate S3-compatible SDK or CLI.
Ceph:

Check the status of Ceph services, including Ceph monitors, OSDs, and RGWs. Validate that the Ceph cluster is healthy and operational. Verify access to Ceph object storage by listing objects in a known bucket using the appropriate Ceph object storage API or CLI.
Azure Blob Storage:

Check the Content from status.azure.com is not included.Azure Status Dashboard to see the health status of the Azure Blob Storage service. Validate your access to Azure Blob Storage by listing containers or objects using the Azure CLI or Azure SDKs.
OpenStack Swift:

Check the Content from status.openstack.org is not included.OpenStack Status page to verify the status of the OpenStack Swift service. Ensure that the Swift services (proxy server, container servers, object servers) are running and accessible. Validate your access to Swift by listing containers or objects using the appropriate Swift CLI or SDK.