Synchronizing a custom repository fails with error "Package id from primary metadata (XXXX), does not match package id from filelists, other metadata (YYYY)" in Red Hat Satellite 6.10

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.10 and above

Issue

  • While trying to create and sync a third-party repository, The following error is visible:

    Package id from primary metadata (1c2d9bfe9af04576a79114f34ac66a494ad5c635c5fc10c1b02f31c896c184ab), does not match package id from filelists, other metadata (51d1a188f1dbed24e59eaec64b7410e8f1dddb69c083f4431332fbaaa352d0b7)

  • The same repository configured in Red Hat Satellite 6.9 or below (using Pulp 2), gets synchronized without any issues but It no longer works after upgrading to Satellite 6.10.

Resolution

As it can be observed in the Root Cause and Diagnostic Steps, The main problems are the content of the upstream repository metadata and the way Pulp 3 performs iterative parsing of the same.

  • To resolve this issue properly, Ensure that the upstream repository metadata is fixed.

  • As a workaround, the following steps can be performed on the Red Hat Satellite 6.10, to disable the iterative parsing feature of Pulp 3 and re-sync the repo.

    • Take a backup of the pulp settings.

      # cp -pr /etc/pulp/settings.py /etc/pulp/settings.py_orig

    • Disable the iterative parsing and restart pulpcore specific services.

             # echo -e "\nRPM_ITERATIVE_PARSING = False" >> /etc/pulp/settings.py

       # for i in $(foreman-maintain service status -b | grep "Displaying the following" -A1 | tr "," "\n" | grep pulpcore)
          do 
          systemctl restart  $i; 
          done

    • Verify the health of the Satellite server.

      # sleep 10 && hammer ping

    • Now, Just simply synchronize the affected repo and wait for the task to get completed successfully.

    • Once the repository is synchronized:

      • Revert back the changes made in the /etc/pulp/settings.py file.
      • Restart the exact same set of pulpcore services using the for loop as used earlier.

NOTE:

  • At this stage, the repository would be properly synced and usable by the clients of the concerned Red Hat Satellite 6.10.

  • The iterative parsing has been turned back on again so that It does not cause any performance issues during other repository-sync operations.

  • The affected repository should not be attempted to re-synchronize without fixing the repository metadata at the upstream source.

 

It is advised to reach out to This content is not included.Red Hat Technical Support in case of any further clarification is required about the suggested approach.

For more KB articles/solutions related to Red Hat Satellite 6.x Repository/Pulp3 related issues, please refer to the following articles:
Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 3.0-related Issues
Consolidated Troubleshooting Article for Red Hat Satellite 6.x Repository Issues

Root Cause

Following is the main error message to focus on i.e. 

Apr 25 19:25:59 satellite pulpcore-worker-2: pulp [9b89eca6-a160-424e-a249-53cc23e8e262]: pulpcore.tasking.pulpcore_worker:INFO: Task 27638017-b73f-431c-83bf-5c5631ff68ee failed (Package id from primary metadata (1c2d9bfe9af04576a79114f34ac66a494ad5c635c5fc10c1b02f31c896c184ab), does not match package id from filelists, other metadata (51d1a188f1dbed24e59eaec64b7410e8f1dddb69c083f4431332fbaaa352d0b7))

This means the content of primary.xml metadata is being verified against filelists.xml and there is a conflict present.

It can happen due to two reasons.

(A) Issue with the order of the package ids in both of the metadata files.

1c2d9bfe9af04576a79114f34ac66a494ad5c635c5fc10c1b02f31c896c184abis the first entry in the primary.xml.gz file but the last entry in the filelists.xml.gz file.

51d1a188f1dbed24e59eaec64b7410e8f1dddb69c083f4431332fbaaa352d0b7 is the last entry in the primary.xml.gz but the first entry in the filelists.xml.gz file.

It does not need to be a situation of first and last entry but It could be the fact that the expected package id is having any two different positions in those two files, but due to the nature of iterative parsing, Pulp will expect that the order of the packages and package ids in both of the files should be exactly the same.

(B) The same issue as the first reason but additionally the filelists.xml.gz file could end up having duplicate package entries causing conflict during iterative parsing of the data.

Diagnostic Steps

  • While synchronizing the custom repository, The following error traceback was logged in /var/log/messages of the Red Hat Satellite 6.10 server.

       Apr 25 19:25:58 satellite pulpcore-api: pulp [9b89eca6-a160-424e-a249-53cc23e8e262]:  - - [25/Apr/2022:13:55:58 +0000] "GET /pulp/api/v3/tasks/27638017-b73f-431c-83bf-5c5631ff68ee/ HTTP/1.1" 200 1035 "-" "OpenAPI-Generator/3.14.1/ruby"

   Apr 25 19:25:59 satellite pulpcore-worker-2: pulp [9b89eca6-a160-424e-a249-53cc23e8e262]: pulpcore.tasking.pulpcore_worker:INFO: Task 27638017-b73f-431c-83bf-5c5631ff68ee failed (Package id from primary metadata (1c2d9bfe9af04576a79114f34ac66a494ad5c635c5fc10c1b02f31c896c184ab), does not match package id from filelists, other metadata (51d1a188f1dbed24e59eaec64b7410e8f1dddb69c083f4431332fbaaa352d0b7))

   Apr 25 19:25:59 satellite pulpcore-worker-2: pulp [9b89eca6-a160-424e-a249-53cc23e8e262]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 339, in _perform_task
   Apr 25 19:25:59 satellite pulpcore-worker-2: result = func(*args, **kwargs)
   Apr 25 19:25:59 satellite pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 491, in synchronize
   Apr 25 19:25:59 satellite pulpcore-worker-2: version = dv.create()
   Apr 25 19:25:59 satellite pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
   Apr 25 19:25:59 satellite pulpcore-worker-2: loop.run_until_complete(pipeline)
   Apr 25 19:25:59 satellite pulpcore-worker-2: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
   Apr 25 19:25:59 satellite pulpcore-worker-2: return future.result()
   Apr 25 19:25:59 satellite pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
   Apr 25 19:25:59 satellite pulpcore-worker-2: await asyncio.gather(*futures)
   Apr 25 19:25:59 satellite pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
   Apr 25 19:25:59 satellite pulpcore-worker-2: await self.run()
   Apr 25 19:25:59 satellite pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 748, in run
   Apr 25 19:25:59 satellite pulpcore-worker-2: await self.parse_repository_metadata(repomd, repomd_files, file_extension)
   Apr 25 19:25:59 satellite pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 802, in parse_repository_metadata
   Apr 25 19:25:59 satellite pulpcore-worker-2: file_extension=file_extension,
   Apr 25 19:25:59 satellite pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 1098, in parse_packages
   Apr 25 19:25:59 satellite pulpcore-worker-2: for pkg in parser.parse_packages_iterative(file_extension, skip_srpms=skip_srpms):
   Apr 25 19:25:59 satellite pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/metadata_parsing.py", line 271, in parse_packages_iterative
   Apr 25 19:25:59 satellite pulpcore-worker-2: ).format(pkgid, pkgid_extra)
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.