Satellite 6 failed to sync repositories after request timed out

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.10+

Issue

  • After upgrading to 6.10, Satellite fails to sync some repositories with large files and displays the following error:

    Request timed out for https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/repodata/4ae4b0b9f3e8e0ed6007d17625c04e36a3b01ec9-other.xml.gz. Increasing the total_timeout value on the remote might help. (Katello::Errors::Pulp3Error)

  • Failed to sync Capsule. During the Capsule sync, you see some on demand rpm downloads timeout error in the Satellite /var/log/messages like below.

    Backing off download_wrapper(...) for 0.4s (pulpcore.exceptions.base.TimeoutException: Request timed out for https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/dotnet/1/os/Packages/r/rh-dotnet60-dotnet-sdk-6.0-source-built-artifacts-6.0.101-2.el7_9.x86_64.rpm. Increasing the total_timeout value on the remote might help.)

  • Syncing a large rpm in Sat6.16 immediate download policy is causing error:

May 11 22:09:08 sat616 pulpcore-worker-2[213593]: pulp [8c165287-17c4-48fa-b92c-d4f926a0de91]: backoff:INFO: Backing off download_wrapper(...) for 0.5s (pulpcore.exceptions.base.TimeoutException: Request timed out for https://cdn.redhat.com/content/dist/rhel8/8.8/x86_64/codeready-builder/os/Packages/m/ModemManager-glib-devel-1.20.2-1.el8.x86_64.rpm. Increasing the total_timeout value on the remote might help.)
May 11 22:09:08 sat616 pulpcore-worker-2[213593]: Backing off download_wrapper(...) for 0.8s (pulpcore.exceptions.base.TimeoutException: Request timed out for https://cdn.redhat.com/content/dist/rhel8/8.8/x86_64/codeready-builder/os/Packages/m/ModemManager-devel-1.20.2-1.el8.i686.rpm. Increasing the total_timeout value on the remote might help.)
May 11 22:09:08 sat616 pulpcore-worker-2[213593]: pulp [8c165287-17c4-48fa-b92c-d4f926a0de91]: backoff:INFO: Backing off download_wrapper(...) for 0.8s (pulpcore.exceptions.base.TimeoutException: Request timed out for https://cdn.redhat.com/content/dist/rhel8/8.8/x86_64/codeready-builder/os/Packages/m/ModemManager-devel-1.20.2-1.el8.i686.rpm. Increasing the total_timeout value on the remote might help.)
May 11 22:09:35 sat616 pulpcore-worker-2[213593]: Backing off download_wrapper(...) for 1.8s (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host cdn.redhat.com:443 ssl:default [None])

Resolution

In the Satellite web UI, increase the sync connection timeouts from 300 seconds (5 minutes) to 3600 seconds (1 hour)

  • Administer, Settings, Content tab, Sync Total Timeout, set 3600
  • Administer, Settings, Content tab, Sync Connect Timeout, set 3600

Retry sync as follows:

Satellite Next Steps

Try to sync the repositories again.

Capsule Next Steps

  • Run the following script on the Satellite to propagate the setting to Pulp immediately. The script will trigger many "Refresh repository" tasks in Satellite.

    foreman-rake katello:refresh_pulp_repo_details

  • Wait until all tasks are finished successfully in the web UI, Monitor, Tasks page

  • Trigger an optimized Capsule sync to sync the unfinished repositories

Root Cause

Network performance was inadequate and exceeded the default 5 mins (300 seconds) to download the files.

Since the repositories in Satellite are using "on demand" download policy which means rpms are not downloaded when syncing repositories from upstream. When Capsule/client requests the rpms, Satellite will have to download them from upstream and then forward them to the requesting Capsule/client. If Satellite failed to download the requested rpms due to network timeout then it will not able to provide the rpms to the Capsule/client.

For more KB articles/solutions related to Red Hat Satellite 6.x Repository Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Repository Issues.

Diagnostic Steps

Check production.log on the Satellite for Request timed out errors:

# grep -i 'Request timed out' /var/log/foreman/production.log

If you are using a proxy to connect to the internet from the Satellite, make sure you have defined the proxy correctly in /etc/rhsm/rhsm.conf and in the Satellite web UI:

  • Infrastructure, HTTP Proxies
  • Administer, Settings, Content tab, Default HTTP Proxy
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.