How to verify and repair synced content in Red Hat Satellite 6.10 and later?

Solution Verified - Updated 6 May 2025

Environment

Red Hat Satellite 6.10 and above
Red Hat Satellite Capsule 6.10 and above
Pulp 3

Issue

Is there a way to perform checksum validation of synchronized contents?
Is there a way to repair/fix any synced content that has become corrupted?
One or more artifacts specific to certain RPMS are supposed to be already downloaded inside /var/lib/pulp but they are missing. Is there any way to re-download only these missing artifacts?

Resolution

The Red Hat Satellite 6.10 and above offers a feature called Verify Content Checksum which is backed by the Content from docs.pulpproject.org is not included.Repair feature of Pulp-3.

This Verify Content Checksum or Repair feature works in this way i.e.

Looks for missing or corrupted files that are supposed to be downloaded or present in the storage/filesystem already.
Attempts a redownload of these files from the upstream URL of the related repository and places them in their right location, if the sha256sum value of the file matches with the expected value in repository metadata.
It can work on Content from docs.pulpproject.org is not included.one repository or all repositories at once as well.
In case the upstream repository has stopped serving the corresponding files or these files were uploaded directly into the repository manually or were created by Pulp itself (i.e. repo metadata), the repair feature will be unable to fix those files.
The modulemd metadata files for repositories can't be fixed by this feature till Red Hat Satellite 6.11. The issue was reported via This content is not included.Bugzilla 2107701 and has been fixed in Red Hat Satellite 6.12.
The container\docker contents cannot be fully repaired as well due to the This content is not included.Bugzilla 2173671 and has been fixed in Red Hat Satellite 6.14+.

An important point to be noted is that unlike the Validate Content feature of Pulp 2, This repair feature does not download the entire repository content but only works on the missing or corrupted ones that are supposed to be already downloaded.

How does it work in Red Hat Satellite 6.10 and above?

If a repository is already suspected to be having corrupted or missing content, then

Identify the id and/or version_href of the concerned repository:

       # hammer --csv repository list --organization RedHat --search "name = Red_Hat_Satellite_Client_6_11_RHEL7" --fields Id,Name,Product | column -s, -t
       Id  Name                                 Product
       97  Red_Hat_Satellite_Client_6_11_RHEL7  Satellite_6_11_Custom
       
       or
       
       # echo "select main.id,root.name,main.version_href from katello_root_repositories root \
         left JOIN katello_repositories main on main.root_id=root.id where library_instance_id is null \
         and root.name = 'Red_Hat_Satellite_Client_6_11_RHEL7';" | su - postgres -c "psql -x foreman"

       id           | 97
       name         | Red_Hat_Satellite_Client_6_11_RHEL7
       version_href | /pulp/api/v3/repositories/rpm/rpm/ac39aa73-8f97-4ec2-ba2b-b35a1a94bd25/versions/1/

From the command line invoke the following hammer command against the id of the repository and observe the outcome.

# hammer repository synchronize --validate-contents true --id 97 --organization-id 1
[......................................................................................................] [100%]
Total steps: 4/4
.--------------------------------
Identify corrupted units: 1/1
Identify missing units: 1/1
Repair corrupted units: 2/2

This message means, There was one missing artifact and one corrupted artifact found in this repository and both have been repaired.

The same action can be mimicked directly using the Pulp API as well but for that, the usage of version_href will be required i.e.

## Run the repair task in pulp for the specific repository:

# curl -s --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem \
  -H "Content-Type: application/json" -X POST \
  https://<satellite-fqdn>/pulp/api/v3/repositories/rpm/rpm/ac39aa73-8f97-4ec2-ba2b-b35a1a94bd25/versions/1/repair/ | json_reformat
  
{
   "task": "/pulp/api/v3/tasks/32483cc5-5474-417b-9c35-eac758d80d1c/"
}

## Enquiry the state of the task in pulp and wait until all the steps are completed:

# curl -s --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem \
  https://<satellite-fqdn>/pulp/api/v3/tasks/32483cc5-5474-417b-9c35-eac758d80d1c/ | json_reformat 
  
{
...
    "progress_reports": [
        {
            "message": "Identify missing units",
            "code": "repair.missing",
            "state": "completed",
            "total": null,
            "done": 1,
            "suffix": null
        },
        {
            "message": "Identify corrupted units",
            "code": "repair.corrupted",
            "state": "completed",
            "total": null,
            "done": 1,
            "suffix": null
        },
        {
            "message": "Repair corrupted units",
            "code": "repair.repaired",
            "state": "completed",
            "total": null,
            "done": 2,
            "suffix": null
        }
...
}

If it's being suspected that, content from multiple repositories could have the same problem, then execute the repair action in pulp for all repositories at once.

## Run the repair task in pulp for all repos:

# curl -s --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem \
  -H "Content-Type: application/json" -X POST https://<satellite-fqdn>/pulp/api/v3/repair/ | json_reformat
{
    "task": "/pulp/api/v3/tasks/36d19d67-6472-410b-ac0c-f87880b23006/"
}

## Monitor the task in pulp and wait until all the steps are completed:

# curl -s --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem \
  https://<satellite-fqdn>/pulp/api/v3/tasks/36d19d67-6472-410b-ac0c-f87880b23006/ | json_reformat

Can the same repair action be performed in a Red Hat Satellite Capsule 6.10 and above?

The hammer capsule content verify-checksum command can be used to do this in Red Hat Satellite\Capsule 6.16.

   # hammer capsule content verify-checksum --help
   Usage:
       hammer capsule content verify-checksum [OPTIONS]

   Options:
    --async                                   Do not wait for the task
    --content-view[-id] VALUE/NUMBER          Name/id of the content view to limit verifying checksum on
    --environment[-id] VALUE/NUMBER           (--environment-id is deprecated: Use --lifecycle-environment-id instead)
    --id NUMBER                               Id of the capsule
    --lifecycle-environment[-id] VALUE/NUMBER Name/id of the environment to limit verifying checksum on
    --name VALUE                              Name to search by
    --organization VALUE                      Organization name
    --organization-id VALUE                   Organization ID
    --repository[-id] VALUE/NUMBER            Name/id of the repository to limit verifying checksum on
    -h, --help                                Print help

It is possible to execute the same repair action using the Pulp API for individual Capsule servers as well.

## Run the repair task in pulp for all repos but initiate it from Satellite server pointing to Capsule FQDN:

# curl -s --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem \
  -H "Content-Type: application/json" -X POST https://<Capsule-FQDN>/pulp/api/v3/repair/ | json_reformat
{
    "task": "/pulp/api/v3/tasks/36d19d67-6472-410b-ac0c-f87880b23006/"
}

## Monitor the task in pulp and wait until all the steps are completed:

# curl -s --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem \
  https://<Capsule-FQDN>/pulp/api/v3/tasks/36d19d67-6472-410b-ac0c-f87880b23006/ | json_reformat

It is to be noted that, the action can be executed for individual repositories synced on the capsule as well but the challenge is to find out the right version_href for the same. Hence it is always suggested to run the repair action on all the repositories synced in the external capsule server.

How to fix inconsistent repository metadata in Red Hat Satellite 6.10 and above?

Refer to the following solution article to understand how the repository metadata or content-view-version metadata can be repaired or regenerated.

In case of any further concerns to be reported or clarifications required, It is suggested to open a This content is not included.support case with Red Hat Technical Support.

For more KB articles/solutions related to Red Hat Satellite 6.x Repository Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Repository Issues.

Diagnostic Steps

On a client system connected with Red Hat Satellite or Capsule 6.10, the yum install/update command fails with the following error i.e.

[MIRROR] katello-host-tools-3.5.4-1.el8sat.noarch.rpm: Status code: 502 for https://satellite610.test.lan/pulp/content/RedHat/Library/RHEL8/content/dist/layered/rhel8/x86_64/sat-tools/6.10/os/Packages/k/katello-host-tools-3.5.4-1.el8sat.noarch.rpm

Around the same time, the source Red Hat Satellite or Capsule server shows the following traceback in the logs.

Jan 21 18:05:24 satellite610 pulpcore-content: [2022-01-21 12:35:24 +0000] [1579] [ERROR] Unhandled exception
Jan 21 18:05:24 satellite610 pulpcore-content: Traceback (most recent call last):
Jan 21 18:05:24 satellite610 pulpcore-content: File "/usr/lib64/python3.6/site-packages/aiohttp/web_protocol.py", line 485, in start
Jan 21 18:05:24 satellite610 pulpcore-content: resp, reset = await task
Jan 21 18:05:24 satellite610 pulpcore-content: File "/usr/lib64/python3.6/site-packages/aiohttp/web_protocol.py", line 440, in _handle_request
Jan 21 18:05:24 satellite610 pulpcore-content: reset = await self.finish_response(request, resp, start_time)
Jan 21 18:05:24 satellite610 pulpcore-content: File "/usr/lib64/python3.6/site-packages/aiohttp/web_protocol.py", line 591, in finish_response
Jan 21 18:05:24 satellite610 pulpcore-content: await prepare_meth(request)
Jan 21 18:05:24 satellite610 pulpcore-content: File "/usr/lib64/python3.6/site-packages/aiohttp/web_fileresponse.py", line 115, in prepare
Jan 21 18:05:24 satellite610 pulpcore-content: st = await loop.run_in_executor(None, filepath.stat)
Jan 21 18:05:24 satellite610 pulpcore-content: File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
Jan 21 18:05:24 satellite610 pulpcore-content: result = self.fn(*self.args, **self.kwargs)
Jan 21 18:05:24 satellite610 pulpcore-content: File "/usr/lib64/python3.6/pathlib.py", line 1158, in stat
Jan 21 18:05:24 satellite610 pulpcore-content: return self._accessor.stat(self)
Jan 21 18:05:24 satellite610 pulpcore-content: File "/usr/lib64/python3.6/pathlib.py", line 387, in wrapped
Jan 21 18:05:24 satellite610 pulpcore-content: return strfunc(str(pathobj), *args)
Jan 21 18:05:24 satellite610 pulpcore-content: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/bb/592c8ada2ee35346cbccebb3df7916f512e39995712389aa05a3d6ed35c60a'

Investigation of that artifact from the database reveals that the "artifact_id" field value is present and hence that artifact should have been downloaded inside /var/lib/pulp as well.

   # echo "select ca.pulp_id,cca.artifact_id,ca.file,cca.relative_path from core_artifact ca LEFT JOIN core_contentartifact \
      cca on cca.artifact_id = ca.pulp_id where cca.relative_path = 'katello-host-tools-3.5.4-1.el8sat.noarch.rpm' \
      or ca.file = 'artifact/bb/592c8ada2ee35346cbccebb3df7916f512e39995712389aa05a3d6ed35c60a';" \
      | su - postgres -c "psql -x pulpcore"

   pulp_id       | c55dab97-6f3b-4275-af43-7004019727d9
   artifact_id   | c55dab97-6f3b-4275-af43-7004019727d9
   file          | artifact/bb/592c8ada2ee35346cbccebb3df7916f512e39995712389aa05a3d6ed35c60a
   relative_path | katello-host-tools-3.5.4-1.el8sat.noarch.rpm


   # echo "select * from core_contentartifact where relative_path = 'katello-host-tools-3.5.4-1.el8sat.noarch.rpm';" \
       | su - postgres -c "psql -x pulpcore" 

   pulp_id           | c5eb81fc-5f3e-4228-88ba-18e9646707aa
   pulp_created      | 2022-04-12 19:01:44.457762+05:30
   pulp_last_updated | 2022-04-12 19:01:44.457779+05:30
   relative_path     | katello-host-tools-3.5.4-1.el8sat.noarch.rpm
   artifact_id       | c55dab97-6f3b-4275-af43-7004019727d9
   content_id        | fdf615ca-0eae-49cf-89f4-7c45cc73b904


   # echo "select pulp_id,pulp_created,pulp_last_updated,file,sha256 from core_artifact where \
       file = 'artifact/bb/592c8ada2ee35346cbccebb3df7916f512e39995712389aa05a3d6ed35c60a';" \
       | su - postgres -c "psql -x pulpcore"

   pulp_id           | c55dab97-6f3b-4275-af43-7004019727d9
   pulp_created      | 2022-04-12 19:01:44.123128+05:30
   pulp_last_updated | 2022-04-12 19:01:44.123161+05:30
   file              | artifact/bb/592c8ada2ee35346cbccebb3df7916f512e39995712389aa05a3d6ed35c60a
   sha256            | bb592c8ada2ee35346cbccebb3df7916f512e39995712389aa05a3d6ed35c60a

But the artifact itself is missing from the filesystem.

# file /var/lib/pulp/media/artifact/bb/592c8ada2ee35346cbccebb3df7916f512e39995712389aa05a3d6ed35c60a
/var/lib/pulp/media/artifact/bb/592c8ada2ee35346cbccebb3df7916f512e39995712389aa05a3d6ed35c60a: cannot open (No such file or directory)

In some scenarios, The artifact might be present but instead of having the expected file format, It will be corrupted.

### A good artifact will something like this:

# file /var/lib/pulp/media/artifact/bb/592c8ada2ee35346cbccebb3df7916f512e39995712389aa05a3d6ed35c60a
/var/lib/pulp/media/artifact/bb/592c8ada2ee35346cbccebb3df7916f512e39995712389aa05a3d6ed35c60a: RPM v3.0 bin noarch katello-host-tools-3.5.4-1.el8sat

### A corrupted artifact would not show the right file format or name:

# file /var/lib/pulp/media/artifact/bd/1dc608c1e98924d834ea0fd32bbc8c08c38550727eb8ee33af85a107495196
/var/lib/pulp/media/artifact/bd/1dc608c1e98924d834ea0fd32bbc8c08c38550727eb8ee33af85a107495196: ASCII text

SBR

SysMgmt

Product(s)

Red Hat Satellite

Components

katello
pulp

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.