Pulp2 to Pulp3 content migration fails with error "Some corrupted or missing content found" in Red Hat Satellite 6.9

Solution Verified - Updated 13 Jun 2024

Environment

Red Hat Satellite 6.9
- Pulp 2 to Pulp 3 migration

Issue

The satellite-maintain content prepare command fails with the following error:

   2022-06-15 08:02:08 +0200: Importing migrated content type erratum: 166000/165321
   2022-06-15 08:02:18 +0200: Importing migrated content type erratum: 166000/165321
   2022-06-15 08:02:28 +0200: Importing migrated content type package_group: 350/958

   Some corrupted or missing content found, run 'foreman-maintain content migration-stats' for more information.
  
   E, [2022-06-15 08:02:39+0200 #3374] ERROR -- : Failed executing preserve_output=true foreman-rake katello:pulp3_migration, exit status 255 (ForemanMaintain::Error::ExecutionError)
   /usr/share/gems/gems/foreman_maintain-0.8.30/lib/foreman_maintain/utils/command_runner.rb:54:in `execution_error'
   /usr/share/gems/gems/foreman_maintain-0.8.30/lib/foreman_maintain/concerns/system_helpers.rb:68:in `execute!'
   /usr/share/gems/gems/foreman_maintain-0.8.30/definitions/procedures/content/prepare.rb:14:in `run'
   /usr/share/gems/gems/foreman_maintain-0.8.30/lib/foreman_maintain/executable.rb:139:in `__run__'

The satellite-maintain content migration-stats command displays the following information about the error:

============Missing/Corrupted Content Summary================
WARNING: MISSING OR CORRUPTED CONTENT DETECTED
Corrupted or Missing Rpm: 1000/173142
Corrupted or missing content has been detected, you can examine the list of content in /tmp/unmigratable_content-20220109-11817-1oil4a6 and take action by either:
 1. Performing a 'Verify Checksum' sync under Advanced Sync Options, let it complete, and re-running the migration
 2. Deleting/disabling the affected repositories and running orphan cleanup (foreman-rake katello:delete_orphaned_content) and re-running the migration
 3. Manually correcting files on the filesystem in /var/lib/pulp/content/ and re-running the migration
 4. Mark currently corrupted or missing content as skipped (foreman-rake katello:approve_corrupted_migration_content).  This will skip migration of missing or corrupted content.

Resolution

The Corrupted or Missing RPMs are the contents that Pulp was unable to migrate. There are different reasons present that could influence this behavior and each has its own resolution.

Before moving forward, note down the name of the directory ( in this example /tmp/unmigratable_content-20220109-11817-1oil4a6 ) and then use the file inside the same to gather information about the affected repositories and content-views related to missing\corrupted RPMs. Check the article 6629271 for more information about how to extract the required information.

And ensure that the concerned satellite server has been updated to the latest z-stream of Red Hat Satellite 6.9.

Scenario 1: Some of the RPMs that are not migrated perhaps are present in the /var/lib/pulp file system but pulp cannot access them for some reason.

If the issue is with their permissions, then easily fix that using this command satellite-maintain prep-6.10-upgrade.
It's possible that antivirus programs like Kaspersky or McAfee are installed on the Red Hat Satellite server and preventing access to certain RPMs from /var/lib/pulp. It's good to either exclude /var/lib/pulp from the scanning of the antivirus programs or else disable all the services and processes related to the antivirus programs.

Scenario 2: RPMs not being able to migrate because they are On_Demand content but do not contain an entry in the lazy_content_catalog collection of pulp2 database

A proof of this statement would be to have this text inside /var/log/messages around the time of migration i.e.

On_demand content cannot be migrated without an entry in the lazy catalog
This is the most common reason due to why the On_Demand RPMs are not migrated properly. Following is the way to confirm how many such packages are present.
```
# grep "On_demand content cannot be migrated" /var/log/messages | grep "without an entry in the lazy catalog" | awk '{print $NF}' | sort -u  | wc -l
```
The count usually matches up with the total count of Corrupted or Missing Rpm.
The most effective way to solve this issue is explained below:
- Stop and disable all pulpcore specific services if they are still running.
```
# for i in stop disable
do
systemctl $i pulpcore-resource-manager pulpcore-api pulpcore-content  pulpcore-worker@1 pulpcore-worker@2  pulpcore-worker@3 pulpcore-worker@4
done
```
- The list of affected repositories related to un-migrated RPMs should have already been obtained using the initial instructions of this article.
- For each affected repository:
  - If it's a Red Hat provided repository, then perform a Complete Sync of the repository and ensure that the task finishes without any errors or warnings.
  - If it's a non Red Hat \ Third-party \ Custom repository, then also perform the same Complete Sync of the repository but prior to that ensure to confirm the following about each repository:
    
    If the Download Policy of the repository is set to On Demand, Then the Mirror on Sync has to be set Yes for the same repository.
    
    If Mirror on Sync is set to No and it's required to keep the same, Then the Download Policy of the repository needs to be set as Immediate.

Scenario 3: RPMs not being able to migrate because they may be part of multiple Importers objects of repositories but one of them does not have a upstream\feed URL.

A proof of this statement would be to have this text inside /var/log/messages around the time of migration i.e.

On_demand content cannot be migrated without a remote

Following is the way to confirm the count of such packages:

# grep "On_demand content cannot be migrated" /var/log/messages | grep "without a remote" | awk '{print $NF}' | sort -u  | wc -l

It's not an issue that's common to observe but if there are any found, then it would mostly be a non Red Hat \ Third-party \ Custom repository - identification of the repository is described at the beginning of the article. Then the following approach should help:
- If the repository is being maintained manually i.e. RPMs are manually uploaded in the repo, then:
  - It's expected to not have a Upstream\Feed URL of the repo.
  - Always set the Download Policy of the repository needs to Immediate.
  - Verify that the RPMs specific to the repository exists inside /var/lib/pulp/content/units/ using a find command.
- If the repository is expected to be synced from an upstream source then:
  - Ensure that it has the right Upstream\Feed URL mentioned.
  - Ensure that it is properly synchronized in the Satellite server.

Scenario 4: RPMs are not being able to migrate because they are marked as downloaded in the pulp2 database but they are missing from /var/lib/pulp. Check the This content is not included.Bugzilla 2003884 for more information.

This is the last scenario that Satellite would run into and not very common to occur unless some manual deletion attempts were done to free up the disk space of /var/lib/pulp.
When none of the other scenarios seems to be matching and it requires to verify if this is the issue or not then:
- Get the list of the unique RPMs from /tmp/unmigratable_content-*/Rpm file.
- Collect the list of RPMs stored in Pulp Database, their downloaded status, and full path on the filesystem.
```
# mongo pulp_database --eval "DBQuery.shellBatchSize = 10000000; db.units_rpm.find().forEach(function(R) {print(R.filename, R.downloaded, R._storage_path);})" &> rpminfo
```
  Example of one rpm record in the rpminfo file:
```
teamd-1.29-1.el8.x86_64.rpm true /var/lib/pulp/content/units/rpm/ea/50028c48c0937b063cdc2a0e019cff46424ce23f8d47ee7ffb06f739abb701/teamd-1.29-1.el8.x86_64.rpm
```
  So here, if teamd-1.29-1.el8.x86_64.rpm is an rpm which was part of unmigrated RPMs list then, based on database records, It is expected to be downloaded
  inside /var/lib/pulp/content/units/rpm/ea/50028c48c0937b063cdc2a0e019cff46424ce23f8d47ee7ffb06f739abb701/ path.
  
  But when checking inside that path, the rpm itself is missing.
- Just verify any five or ten un-migrated RPMs in this way and if all of them exhibit the same behaviour, then the one and only option to fix them would be to run Validate Content Sync of any one repository that contains that RPM.
  
  Keep in mind that, Validate Content Sync downloads the entire repository content inside /var/lib/pulp and hence the related filesystem needs to have enough free space available to store the downloaded data and to allow performing the migration later.
  
  Also, The disk space consumed by Validate Content Sync cannot be reclaimed back in any easy way (if there is a need for it) but It will definitely fix the issue.

Scenario 5: Some packages that are no longer part of Red Hat repositories, somehow ended up in the stored repository metadata.

A prime example of this issue can be found in this article.
If all the un-migrated RPMs are related to this issue, then simply skip them from migration:
```
# foreman-rake katello:approve_corrupted_migration_content
```

Scenario 6: Only a few RPMs ( perhaps < 500 or < 1000 ) are un-migrated, where some of the RPMs are from Red Hat and some of them are from Custom repositories.

If someone does not want to go through the hassle of fixing them in any way now and deal with them after the upgrade or none of the solutions mentioned above is helping, then those RPMs can be skipped from migration.

To skip them:
```
# foreman-rake katello:approve_corrupted_migration_content
```
NOTE: It is highly recommended to consult with Red Hat before deciding to perform this action.
These RPMs will be lost after the upgrade to Satellite 6.10. So to get them back after the upgrade to Satellite 6.10:
- Re-sync the related repositories properly.
- Re-publish the content views where those repositories are present and (optionally) promote the newer version to applicable lifecycle environments.

Scenario 7: There could be a scenario where the user has run into this issue but the /tmp/unmigratable_content-*/Rpm file is empty.

If the migration fails due to corrupted or missing RPMs but the resulted file i.e. /tmp/unmigratable_content-*/Rpm is empty, refer to the following article for the next steps to diagnose and solve the problem.

Finally perform the migration:

Once all the necessary actions have been taken, Re-try the content migration and observe:
```
# satellite-maintain prep-6.10-upgrade
# satellite-maintain content prepare
```

Reach out to This content is not included.Red Hat Technical Support in case of any further assistance or clarification will be required.

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 to Pulp 3.0 Migration Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0 to Pulp 3.0 Migration Issues

Root Cause

Corrupt or missing RPM on the Satellite pulp filesystem.
Mismatch in Download policy and Mirroring policy causing On_demand content to not get migrated.
Antivirus programs not allowing to access the RPMs during the migration.

SBR

SysMgmt

Product(s)

Red Hat Satellite

Components

pulp
upgrade

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.