In Quay, podman pull errors out with 502 bad gateway

Solution Verified - Updated 13 Jun 2024

Environment

Red Hat Quay
- 3.x

Issue

The command $ podman pull fails with the below error:

$ podman pull <registry>/<image>:<tag>
Trying to pull  <registry>/<image>:<tag>...
Error: error pulling image " <registry>/<image>:<tag>": unable to pull <registry>/<image>:<tag>: unable to pull image: Error parsing 
image configuration: Error fetching blob: invalid status code from registry 502 (Bad Gateway)

The following type of SSL errors related to botocore are seen in Quay pod logs while pulling images:

...
gunicorn-registry stdout | ssl.SSLError: [X509] PEM lib (_ssl.c:4265)
gunicorn-registry stdout | urllib3.exceptions.SSLError: [X509] PEM lib (_ssl.c:4265)
gunicorn-registry stdout | botocore.exceptions.SSLError: SSL validation failed for https://s3.openshift- 
storage.svc.cluster.local:443/quay-datastore-bcca0914-d071-474b-854a-2f0f4fe72491 [X509] PEM lib (_ssl.c:4265)
...

Resolution

This behaviour is a well known bug - This content is not included.PROJQUAY-3881 in 3.7 and older version. It has been resolved in Quay v3.7.1 and v3.8.0.
The workaround is to add a new line manually. Verify which certificate does not have the new line at the end, add it by going to the end of the file and pressing enter, saving, and then reuploading to Quay.

Root Cause

The certificate installation script cert_install.sh failed to parse certificates that did not have a new line at the end of the entry.

Diagnostic Steps

Check Quay Debug logs for below Traceback:

$ oc logs quay-app-pod

gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 336, in ssl_wrap_socket
gunicorn-registry stdout |     context.load_verify_locations(ca_certs, ca_cert_dir)
gunicorn-registry stdout | ssl.SSLError: [X509] PEM lib (_ssl.c:4264)
gunicorn-registry stdout | During handling of the above exception, another exception occurred:
gunicorn-registry stdout | Traceback (most recent call last):
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/httpsession.py", line 311, in send
gunicorn-registry stdout |     urllib_response = conn.urlopen(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 719, in urlopen
gunicorn-registry stdout |     retries = retries.increment(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 376, in increment
gunicorn-registry stdout |     raise six.reraise(type(error), error, _stacktrace)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 735, in reraise
gunicorn-registry stdout |     raise value
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen
gunicorn-registry stdout |     httplib_response = self._make_request(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 376, in _make_request
gunicorn-registry stdout |     self._validate_conn(conn)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
gunicorn-registry stdout |     conn.connect()
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 352, in connect
gunicorn-registry stdout |     self.sock = ssl_wrap_socket(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 338, in ssl_wrap_socket
gunicorn-registry stdout |     raise SSLError(e)
gunicorn-registry stdout | urllib3.exceptions.SSLError: [X509] PEM lib (_ssl.c:4264)
gunicorn-registry stdout | During handling of the above exception, another exception occurred:
gunicorn-registry stdout | Traceback (most recent call last):
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 200, in _do_get_response
gunicorn-registry stdout |     http_response = self._send(request)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 269, in _send
gunicorn-registry stdout |     return self.http_session.send(request)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/httpsession.py", line 338, in send
gunicorn-registry stdout |     raise SSLError(endpoint_url=request.url, error=e)
gunicorn-registry stdout | botocore.exceptions.SSLError: SSL validation failed for https://quay.example.com:9021/quay [X509] PEM lib (_ssl.c:4264)
gunicorn-registry stdout | 2022-05-11 12:47:36,433 [277] [DEBUG] [botocore.hooks] Event needs-retry.s3.HeadBucket: calling handler <botocore.retryhandler.RetryHandler object at 0x7f0f2b3d01c0>
gunicorn-registry stdout | 2022-05-11 12:47:36,433 [277] [DEBUG] [botocore.retryhandler] retry needed, retryable exception caught: SSL validation failed for https://quay.example.com:9021/quay [X509] PEM lib (_ssl.c:4264)
gunicorn-registry stdout | Traceback (most recent call last):
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 336, in ssl_wrap_socket
gunicorn-registry stdout |     context.load_verify_locations(ca_certs, ca_cert_dir)
gunicorn-registry stdout | ssl.SSLError: [X509] PEM lib (_ssl.c:4264)
gunicorn-registry stdout | During handling of the above exception, another exception occurred:
gunicorn-registry stdout | Traceback (most recent call last):
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/httpsession.py", line 311, in send
gunicorn-registry stdout |     urllib_response = conn.urlopen(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 719, in urlopen
gunicorn-registry stdout |     retries = retries.increment(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 376, in increment
gunicorn-registry stdout |     raise six.reraise(type(error), error, _stacktrace)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 735, in reraise
gunicorn-registry stdout |     raise value
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen
gunicorn-registry stdout |     httplib_response = self._make_request(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 376, in _make_request
gunicorn-registry stdout |     self._validate_conn(conn)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
gunicorn-registry stdout |     conn.connect()
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 352, in connect
gunicorn-registry stdout |     self.sock = ssl_wrap_socket(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 338, in ssl_wrap_socket
gunicorn-registry stdout |     raise SSLError(e)
gunicorn-registry stdout | urllib3.exceptions.SSLError: [X509] PEM lib (_ssl.c:4264)
gunicorn-registry stdout | During handling of the above exception, another exception occurred:
gunicorn-registry stdout | Traceback (most recent call last):
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/retryhandler.py", line 269, in _should_retry
gunicorn-registry stdout |     return self._checker(attempt_number, response, caught_exception)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/retryhandler.py", line 316, in __call__
gunicorn-registry stdout |     checker_response = checker(attempt_number, response,
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/retryhandler.py", line 222, in __call__
gunicorn-registry stdout |     return self._check_caught_exception(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
gunicorn-registry stdout |     raise caught_exception
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 200, in _do_get_response
gunicorn-registry stdout |     http_response = self._send(request)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 269, in _send
gunicorn-registry stdout |     return self.http_session.send(request)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/httpsession.py", line 338, in send
gunicorn-registry stdout |     raise SSLError(endpoint_url=request.url, error=e)
gunicorn-registry stdout | botocore.exceptions.SSLError: SSL validation failed for https://quay.example.com:9021/quay [X509] PEM lib (_ssl.c:4264)

Check python store .pem bundle, located at /usr/local/lib/python3.8/site-packages/certifi/cacert.pem, for newline missing at the end of each certificate.

-----BEGIN CERTIFICATE-----
MIIDfzCCAmegAwIBAgIBADANBgkqhkiG9w0BAQsFADB3MS0wKwYDVQQuEyQ5OWVh
...
P+HJ1lPpvWItsfwJIRJwGIWYKLRXzUi9gGaXd9jN3F+x9+aSJzP6SfBcYwKt5exM
XNIURF1JeFuj2lfdWfZ1aqnh5zbO9yw734qyZ5nhAL+teb0=
-----END CERTIFICATE----------BEGIN CERTIFICATE-----     <---------- here 
MIIDUTCCAjmgAwIBAgIIbb5SXoMm9n8wDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
...
gTv00Wg5O3bvlV4V0I2HP9787Hjd739mPsr6aS2Iqu0KOTx+ag==
-----END CERTIFICATE-----

SBR

Shift Container Registry

Product(s)

Red Hat Quay

Components

quay

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.