Leapp upgrade of a Satellite client repeatedly fails on timeout when fetching random packages

Solution Verified - Updated

Environment

Red Hat Satellite 6

Issue

  • doing a LEAPP upgrade of a Satellite client system
  • the upgrade repeatedly fails with a timeout error during fetching different packages
  • the repositories the system is failing to fetch packages from are properly synchronized to the Satellite
    • download policy of the repositories is the default On Demand

Resolution

Change the download policy of the affected repositories to Immediate and perform a regular synchronization of them. Then try a new LEAPP upgrade.

For more KB articles/solutions related to Red Hat Satellite 6.x LEAPP Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x LEAPP Issues

Root Cause

A repository with On Demand download policy stores on the Satellite just repository's metadata. Any request for fetching a package means Satellite downloads the package from CDN (or whatever the feed URL of the repo is) before it offers it to the requester.

During a LEAPP upgrade, hundreds of packages are requested in a bulk. Satellite is not tuned for such high-scaled activity, which ends up in timeouts in processing the requests.

As each (even timeouted) request means a package is downloaded from CDN and locally stored on the disk, requesting the same package next time during the next upgrade attempt would succeed - that is why fetching of random packages fail in different upgrade attempts.

Diagnostic Steps

Example of client logs with the timeouts:

Cannot download Packages/l/libattr-2.4.48-3.el8.x86_64.rpm: All mirrors were tried

(28/206): librhsm-0.0.3-4.el8.x86_64.rpm        379 kB/s |  33 kB     00:00
[MIRROR] python3-libcomps-0.1.11-5.el8.x86_64.rpm: Curl error (28): Timeout was reached for https://satellite.example.com/pulp/repos/MYORG/Library/content/dist/rhel8/8.4/x86_64/baseos/os/Packages/p/python3-libcomps-0.1.11-5.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
[MIRROR] python3-setuptools-wheel-39.2.0-6.el8.noarch.rpm: Curl error (28): Timeout was reached for https://satellite.example.com/pulp/repos/MYORG/Library/content/dist/rhel8/8.4/x86_64/baseos/os/Packages/p/python3-setuptools-wheel-39.2.0-6.el8.noarch.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
[MIRROR] libattr-2.4.48-3.el8.x86_64.rpm: Curl error (28): Timeout was reached for https://satellite.example.com/pulp/repos/MYORG/Library/content/dist/rhel8/8.4/x86_64/baseos/os/Packages/l/libattr-2.4.48-3.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
[FAILED] libattr-2.4.48-3.el8.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.