[Satellite 6]: Unable to synchronize container repositories

Environment

• Red Hat Satellite 6.x
• Container repositories
• Corporate wide TLS intercepting proxy in use

Issue

• Worker terminated abnormally while processing container repo download
• ERROR: Skipping requests to <registry-fqdn> due to repeated connection failures: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed shown during container-sync.

Resolution

To disable this additional level of security, each repository needs to be changed to have SSL verification disabled.

• To do so in the web-UI set Verify SSL to no (uncheck the checkbox)

  • Content -> Products -> Repository -> Verify SSL needs to be set to "no".

• Using the hammer command on command-line:

    # hammer repository update --verify-ssl-on-sync false <repo-id>
  

Note: Please keep in mind that by disabling the SSL verification Satellite will no longer be able to verify the content-source's authenticity.

For more KB articles/solutions related to Red Hat Satellite 6.x Repository Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Repository Issues.

Root Cause

• Satellite will verify every TLS certificate when contacting a upstream content-source. An intercepting TLS proxy will decrypt the passing content, re-encrypt the traffic by using it's own certificate, signed by a CA which is unknown to Satellite. As this is also considered a weakening of the authenticity of the content-source, there is no supported way to import this CA into Satellite and this is also not advised.
• As the issuer of this certificate could not be trusted, the worker will refuse to continue and thus does not download any further content.

Diagnostic Steps

In case a pulp worker is encountering such interception, the following may be seen in /var/log/messages:

Apr  9 10:41:05 satellite pulp: [ID - daemon.info] requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (6): registry-1.docker.io
Apr  9 10:41:05 satellite pulp: [ID - daemon.info] requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): production.cloudflare.docker.com
Apr  9 10:41:06 satellite pulp: [ID - daemon.err] nectar.downloaders.threaded:ERROR: Skipping requests to registry-1.docker.io due to repeated connection failures: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
Apr  9 10:41:06 satellite pulp: [ID - daemon.err] celery.worker.job:ERROR: (19574-50880) Task pulp.server.managers.repo.sync.sync[25bb5af6-7365-48a3-9a49-6258cdb3714d] raised unexpected: WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL).',)
Apr  9 10:41:06 satellite pulp: [ID - daemon.err] celery.worker.job:ERROR: (19574-50880) Traceback (most recent call last):
Apr  9 10:41:06 satellite pulp: [ID - daemon.err] celery.worker.job:ERROR: (19574-50880)   File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1171, in mark_as_worker_lost
Apr  9 10:41:06 satellite pulp: [ID - daemon.err] celery.worker.job:ERROR: (19574-50880)     human_status(exitcode)),
Apr  9 10:41:06 satellite pulp: [ID - daemon.err] celery.worker.job:ERROR: (19574-50880) WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL).

While registry-1.docker.io may have been whitelisted, the redirection to production.cloudflare.docker.com is not, and gets it's certificate replaced by the proxy. This is acknowledged by the SSL:CERTIFICATE_VERIFY_FAILED message.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.