Describe the Quay Environment and gather Support Data

Describe the environment and what the quay deployment entails

ENVIRONMENT:

Quay

Quay Version:               3.x.x?
Cloud environment:          AWS/AZURE/VMware/GCP/RHV/BARE-METAL
Deployment Type:            Container based/OCP operator deployed
Network environment:        Connected/Disconnected
Internet Proxy Usage:       Yes/No

OpenShift Mirror Registry (OMR)

Mirror Registry Version:    1.3.x/2.0.x
Deployment Type:            Container based
Host OS:                    RHEL/CentOS/Ubuntu
Network environment:        Disconnected/Air-gapped
oc-mirror version:          //oc-mirror version v1/v2
imageURL:                   <example.com/organization/repository>

Depending on the environment the items needed to do minimal troubleshooting will be different. To that end I will be splitting the article in multiple sections depending on the environment:

OpenShift deployment

To get the complete picture in an OpenShift Environment, files have to be collected after debug has been enabled, to turn on debugging follow this article:

Quay Debug Mode

• Once debug has been turned and the issue is recreated performing these commands should generate a troubleshooting bundle that can be uploaded to the case:

# oc get quayregistry <quayregistry_name> -n <quay-namespace> -o yaml > quayregistry.yaml

# oc rsh -n <quay-namespace> <quay-app-pod-name> cat /conf/stack/config.yaml >> quay-config.yaml

# oc adm inspect ns/[quay registry namespace]

# tar cvaf  quay-enterprise-bundle.tar.gz --add-file=quayregistry.yaml --add-file=quay-config.yaml inspect.local.<some-number>/

• Once an enterprise bundle has been collected to get additional inspects without the extra files simply use an namespace inspect:

# oc adm inspect ns/[quay registry namespace]

# tar cvaf quay-inspect.tar.gz inspect.local.<some-number>/

• Beyond this starting bundle these commands may be necessary when troubleshooting an OpenShift Environment:

# oc cluster-info dump -n <quay-namespace> > cluster-dump.log
# oc get secrets -n <quay-namespace> -o yaml > secrets.yaml
# oc get deployment -n <quay-namespace> -o yaml > deployment.yaml
# oc get -n <quay-namespace> ClusterServiceVersion > quay-version.txt

• Typically the quay operator lives in a different namespace from the quay app pods, ie openshift-operators, when we need logs from the operator here is the command to get it:

# oc adm inspect ns/[quay operator namespace]

# tar cvaf inspect-quay-operator.tar.gz inspect.local.<some-number>/

• Lastly there might be times that we need the actual postgres logs from the managed pod:

$ oc cp <quay-namespace>/<quay-database-pod>:/var/lib/pgsql/data/userdata/postgresql/logs /path/to/desired-directory-on-host

$ tar cvaf quay-database-logs.tar.gz /path/on/host/

##############################################################################################################

Container deployment

• To get the complete picture in a Container Environment, files have to be collected after debug has been enabled, to turn on debugging follow this article:

Quay Debug Mode

• Once debug has been turned and the issue is recreated performing these commands should generate a troubleshooting bundle that can be uploaded to the case:

# podman exec -it <quay-app-container-name> cat /conf/stack/config.yaml >> quay-config.yaml
# podman logs <quay-app-container-name> > quay-app-debug.log
# tar cvaf quay-enterprise-bundle.tar.gz --add-file=quay-config.yaml quay-app-debug.log

• There might be times that we need the actual postgres logs from the running container:

$ podman cp <quay-database-container>:/var/lib/pgsql/data/userdata/postgresql/logs/. /path/to/desired-directory-on-host/

$ tar cvaf quay-database-logs.tar.gz /path/on/host/

#################################################################################################################

Clair

• Depending on the environment getting the minimal clair troubleshooting bundle will require different commands for each environment. Before collecting these bundles it is imperative to turn debugging on for both quay and clair. To turn on quay debug please reference the article link posted above. To turn on clair debug please reference the Clair troubleshooting section in the documentation:

Enabling debug mode for Clair

• Once debug has been turned on and the issue has been recreated we can collect the quay-clair troubleshooting bundle. For OpenShift the commands are:

# oc rsh -n <quay-namespace> <clair-app-pod-name> cat /clair/config.yaml >> clair-config.yaml

# oc get quayregistry <quayregistry_name> -n <quay-namespace> -o yaml > quayregistry.yaml

# oc rsh -n <quay-namespace> <quay-app-pod-name> cat /conf/stack/config.yaml >> quay-config.yaml

# oc adm inspect ns/[quay registry namespace]

# tar cvaf quay-clair-enterprise-bundle.tar.gz --add-file=quayregistry.yaml --add-file=quay-config.yaml --add-file=clair-config.yaml inspect.local.<some-number>/

• For quay and clair running in a container environment, the commands are:

# podman exec -it <quay-app-container-name> cat /conf/stack/config.yaml >> quay-config.yaml
# podman exec -it <clair-app-pod-name> cat /clair/config.yaml >> clair-config.yaml
# podman logs <quay-app-container-name> > quay-app-debug.log
# podman logs <clair-app-container-name> > clair-app-debug.log
# tar cvaf quay-clair-enterprise-bundle.tar.gz --add-file=quay-config.yaml --add-file=clair-config.yaml --add-file=clair-app-debug.log quay-app-debug.log