Quay georeplication errors out with SSLError

Environment

• Red Hat Quay
  • 3.x
• Red Hat OpenShift Container Platform
  • 4.x

Issue

• After running the command $ podman push following error occurs in the quay logs:

storagereplication stdout | 2021-12-16 14:56:29,602 [144] [ERROR] [__main__] Failed to copy path `sha256/9f/9f9b90db7acda0f3f43e720ac9d54a7e623078fc7af6cf0c1d055410986d3f10` of image storage 0a014260-01a3-4a54-8dd6-784de7bf4feb to location dr
toragereplication stdout | Traceback (most recent call last):
storagereplication stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 336, in ssl_wrap_socket
storagereplication stdout |     context.load_verify_locations(ca_certs, ca_cert_dir)
storagereplication stdout | ssl.SSLError: [X509] PEM lib (_ssl.c:4265)
storagereplication stdout | During handling of the above exception, another exception occurred:
.
.
.
storagereplication stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/httpsession.py", line 338, in send
storagereplication stdout |     raise SSLError(endpoint_url=request.url, error=e)
storagereplication stdout | botocore.exceptions.SSLError: SSL validation failed for https://s3-openshift-storage.example.com/quay-bucket-dr [X509] PEM lib (_ssl.c:4265)
storagereplication stdout | 2021-12-16 14:56:29,603 [144] [WARNING] [workers.queueworker] An error occurred processing request: {"namespace_user_id": 1, "storage_id": "0a014260-01a3-4a54-8dd6-784de7bf4feb"}
storagereplication stdout | 2021-12-16 14:56:29,603 [144] [WARNING] [workers.queueworker] Job exception:

Resolution

• The solution is to remove certificates from the extra_ca_certs directory one by one until you find the duplicate. After each removal, restart Quay pod to see if the issue still persists.

Root Cause

• The error is usually seen from the fact that multiple certificates signing the same thing are added to Quay. This error is frequently seen on regular pushes and LDAP connectivity, even when outside sources are used (for example AWS storage buckets).

Diagnostic Steps

• Check georeplication configurations are properly set in quay's config.yaml file. Example output is shown below :

DISTRIBUTED_STORAGE_CONFIG:
  dr:
  - RHOCSStorage
  - access_key: xxxxxxxxxx
    bucket_name: quay-bucket1-sample
    hostname: s3-openshift-storage.example.com
    is_secure: true
    port: ""
    secret_key: xxxxxxxxxxxx
    storage_path: /datastorage/registry
  local_us:
  - RHOCSStorage
  - access_key: xxxxxxxxxxxxx
    bucket_name: quay-bucket2-sample
    hostname: s3.openshift-storage.svc.cluster.local
    is_secure: false
    port: "80"
    secret_key: xxxxxxxxxxx
    storage_path: /datastorage/registry
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
- local_us
- dr
DISTRIBUTED_STORAGE_PREFERENCE:
- local_us
- dr

• List both storage engines under both distributed storage preference and default storage location. This ensures replication happens regardless of which Data Centre the image is originally pushed.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.