Troubleshooting Quay Storage

Updated 24 Jul 2023

Introduction

Object storage is a type of data storage architecture that manages data as discrete units called objects. Unlike traditional file systems that organize data into hierarchical directories and files, object storage treats data as independent entities with unique identifiers. Each object contains the data itself, along with metadata that describes the object and enables efficient retrieval.
Quay uses object storage as the underlying storage mechanism for storing and managing container images. It stores container images as individual objects. Each container image is treated as an object, with its own unique identifier and associated metadata.

Troubleshooting Steps

Check what object storage is used and if it is officially supported by Quay.

$ oc get quayregistry quay-registry-name -o yaml

Enable Debug Mode for Quay.
Examine Storage related logs in Quay container/Pod for any relevant error messages or warnings related to object storage. Debug logs can provide insights into connectivity issues, authentication failures, or any other issues encountered when interacting with the object storage service.

  $ oc logs quay-pod-name &> quay.logs

  $ podman logs quay-container &> quay.logs

Check Object Storage Configuration from Quay config.yaml file:

  $ oc exec -it quay-pod-name -- cat /conf/stack/config.yaml

  $ podman exec -it quay-container cat /conf/stack/config.yaml

DISTRIBUTED_STORAGE_CONFIG:
  local_us:
  - RHOCSStorage
  - access_key: redacted
    bucket_name: lht-quay-datastore-68fff7b8-1b5e-46aa-8110-c4b7ead781f5
    hostname: s3.openshift-storage.svc.cluster.local
    is_secure: true
    port: 443
    secret_key: redacted
    storage_path: /datastorage/registry
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
  - local_us
DISTRIBUTED_STORAGE_PREFERENCE:
  - local_us

Ensure it is accurate and matches the settings provided by the object storage provider. Check the access credentials, endpoint URLs, bucket/container names, and any other relevant configuration parameters.

Verify Object Storage Connectivity: Ensure that the Quay has network connectivity to the object storage endpoint. Check network configurations to ensure there are no restrictions blocking the communication between Quay and the object storage endpoint.
- Curl from inside Quay container/pod to s3 endpoint:
```
$ curl -kv http://s3.hostname.com:port
```
- Check health checks from inside Quay pod:
```
$ curl http://localhost:8080/health/instance
$ curl http://localhost:8080/health/endtoend
```
- If FEATURE_STORAGE_PROXY is enabled check if its download URL is accessible. It is found in Quay debug logs
```
$ curl -vvv "https://QUAY_HOSTNAME/_storage_proxy/dhaWZKRjlyO......Kuhc=/https/quay.hostname.com/quay-test/datastorage/registry/sha256/0e/0e1d17a1687fa270ba4f52a85c0f0e7958e13d3ded5123c3851a8031a9e55681?AWSAccessKeyId=xxxx&Signature=xxxxxx4%3D&Expires=1676066703"
```
Test Object Storage Access: Try accessing the object storage service outside of Quay to determine if the issue is specific to Quay or the underlying object storage. You can use command-line tools like Content from docs.aws.amazon.com is not included.aws, Content from cloud.google.com is not included.gsutil, Content from github.com is not included.s3cmd provided by the object storage provider to perform basic operations such as listing buckets/containers or uploading/downloading objects. This helps isolate the problem.