Synchronizing large repositories in Red Hat Satellite 6 fails with error Katello::Errors::Pulp3Error: Response payload is not completed

Environment

• Red Hat Satellite 6.10+

Issue

• In environments with limited network or system speeds, attempting to sync large repositories post-upgrade to Red Hat Satellite 6.10 or on Red Hat Satellite 6.11 fails with error:

  Errors:  A file located at the url https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/repodata/8c0a915e4726b04d8d2515a29c8ac07e97caf058-other.sqlite.bz2 failed validation due to checksum.
  

• Synchronizing a big repository to Satellite6

• the repository is accessible via a proxy

• the sync always fails with error "Response payload is not completed"

Resolution

• To remediate the issue, adjust the download_concurrency to 1 using below command:

  # hammer repository list                                        // This command lists the repository IDs.
  # hammer repository update --download-concurrency=1 --id=<ID>  // Get the Repo ID from the previous command and replace with the <ID>
  

• If this issue happens for a Custom repository , try changing the download policy to On demand and then sync the repository.

• To view the current download concurrency you can use :

  #  su - postgres -c "psql -d foreman -c \"SELECT rr.id, rr.name, rr.download_concurrency FROM          katello_root_repositories rr WHERE rr.id = REPO_ID;\""
  

  Replace REPO_ID with actual Repository ID

• For any reason, if you want to revert back the download concurrency to default, please try :

  # curl -sk -u admin:'<SATELLITE_ADMIN_PASSWORD>' -X PUT \
   -w '\nHTTP_STATUS:%{http_code}\n' \
   -H "Content-Type: application/json" \
   -d '{"download_concurrency": null}' \
   "https://satellite.example.com/katello/api/v2/repositories/REPO_ID"
  

  Replace REPO_ID with actual Repository ID

For more KB articles/solutions related to Red Hat Satellite 6.x Repository Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Repository Issues.

Root Cause

• Red Hat Engineering team is already aware of this bug and is currently tracking the same in This content is not included.Bug 2033174.

Diagnostic Steps

• Below error traceback is seen on Satellite:

  Dec 15 15:31:23 satellite.example.com pulpcore-worker-3: Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
  Dec 15 15:31:23 satellite.example.com pulpcore-worker-3: pulp [2cf788a6-b222-4af4-b000-3ceab535ec53]: backoff:INFO: Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
  Dec 15 15:31:23 satellite.example.com pulpcore-worker-3: Backing off download_wrapper(...) for 1.0s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
  Dec 15 15:31:23 satellite.example.com pulpcore-worker-3: pulp [2cf788a6-b222-4af4-b000-3ceab535ec53]: backoff:INFO: Backing off download_wrapper(...) for 1.0s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
  Dec 15 15:31:27 satellite.example.com pulpcore-worker-3: pulp [2cf788a6-b222-4af4-b000-3ceab535ec53]: pulpcore.tasking.pulpcore_worker:INFO: Task 7fc09c6a-d7a1-4dfd-a9ed-117f7eb207c5 failed (A file located at the url https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/repodata/18f74a403ac596dbdf0aecb35a63e1cf43ad6b8a-filelists.xml.gz failed validation due to checksum.)
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.