Why does content migration to Pulp 3 fails due to corrupted or missing content but file /tmp/unmigratable_content-*/Rpm is empty in Red Hat Satellite 6.9?
Environment
- Red Hat Satellite 6.9.
Issue
-
Content migration fails on corrupted or missing RPM packages:
root@satellite# satellite-maintain content migration-stats [...] ============Missing/Corrupted Content Summary================ WARNING: MISSING OR CORRUPTED CONTENT DETECTED Corrupted or Missing Rpm: 1/140788 Corrupted or missing content has been detected, you can examine the list of content in /tmp/unmigratable_content-20220613-12921-kvpcs2 and take action by either: 1. Performing a 'Validate Content Sync' under Advanced Sync Options, let it complete, and re-running the migration 2. Deleting/disabling the affected repositories and running orphan cleanup (foreman-rake katello:delete_orphaned_content) and re-running the migration 3. Manually correcting files on the filesystem in /var/lib/pulp/content/ and re-running the migration 4. Mark currently corrupted or missing content as skipped (foreman-rake katello:approve_corrupted_migration_content). This will skip migration of missing or corrupted content. Any skipped content will be re-synced if it is still available from the upstream repository. -
Resulting file /tmp/unmigratable_content-*/Rpm does not list any RPM package:
root@satellite# cat /tmp/unmigratable_content-20220613-12921-kvpcs2/Rpm filename,id,Repository Name,Content View Name,Content View Version root@satellite# -
How can I find information about the missing or corrupted package(s)?
Resolution
Please run the following commands in your Satellite server to obtain a list of RPM packages that failed to migrate:
root@satellite# foreman-rake console
conf.echo = false
migrated_rpms = ::Katello::Rpm.where('pulp_id LIKE ?', '%/pulp/api/v3/content/rpm/packages/%').or(::Katello::Rpm.where.not(migrated_pulp3_href: nil))
all_rpms = ::Katello::Rpm.all
unmigrated_rpms = all_rpms - migrated_rpms
pp unmigrated_rpms
exit
Examine the resulting output looking for packages reported with attribute "missing_from_migration" set to "true", and make a note of their ID and filename:
#<Katello::Rpm:0x000000002bca5280
id: 127040,
^^^^^^
pulp_id: "33b94870-7a2f-4b36-a775-1310df7676f0",
created_at: Sat, 18 Jan 2020 17:40:15 UTC +00:00,
updated_at: Sat, 18 Jan 2020 17:40:15 UTC +00:00,
name: "compat-locales-sap",
version: "1.0.10",
release: "5.el8",
arch: "noarch",
epoch: "0",
filename: "compat-locales-sap-1.0.10-5.el8.noarch.rpm",
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sourcerpm: "compat-locales-sap-1.0.10-5.el8.src.rpm",
checksum: "feeb439d5153e766965461fdd586f61484f949ee91291aff149da21fef657b9f",
version_sortable: "01-1.01-0.02-10",
release_sortable: "01-5.$el.01-8",
summary: "Compatibility locales for SAP",
nvra: "compat-locales-sap-1.0.10-5.el8.noarch",
modular: false,
migrated_pulp3_href: nil,
evr:
"(0,\"{\"\"(1,)\"\",\"\"(0,)\"\",\"\"(10,)\"\"}\",\"{\"\"(5,)\"\",\"\"(0,el)\"\",\"\"(8,)\"\"}\")",
missing_from_migration: true,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ignore_missing_from_migration: false>]
Run the following command to query the Pulp database for more information about the package(s) replacing <package_name> with the filename obtained previously:
mongo pulp_database --eval "DBQuery.shellBatchSize = 10000000; db.units_rpm.find({'filename': '<package_name>'}, {filename: 1})" | grep -vi -e mongodb
For example:
root@satellite# mongo pulp_database --eval "DBQuery.shellBatchSize = 10000000; db.units_rpm.find({'filename': 'compat-locales-sap-1.0.10-5.el8.noarch.rpm'}, {filename: 1})" | grep -vi -e mongodb
{ "_id" : "33b94870-7a2f-4b36-a775-1310df7676f0", "filename" : "compat-locales-sap-1.0.10-5.el8.noarch.rpm" }
Run the following command to query the repository and content view version where the package is present replacing <unit_id> with the _id obtained previously:
for repo_id in $(mongo pulp_database --eval "DBQuery.shellBatchSize = 10000000; db.repo_content_units.find({'unit_id': '<unit_id>'},{'repo_id': 1})" | grep -vi -e mongodb | cut -d\" -f8); do mongo pulp_database --eval "DBQuery.shellBatchSize = 10000000; db.repos.find({'repo_id': '${repo_id}'}).pretty()" | grep -vi -e mongodb; done
For example:
root@satellite# for repo_id in $(mongo pulp_database --eval "DBQuery.shellBatchSize = 10000000; db.repo_content_units.find({'unit_id': '33b94870-7a2f-4b36-a775-1310df7676f0'},{'repo_id': 1})" | grep -vi -e mongodb | cut -d\" -f8); do mongo pulp_database --eval "DBQuery.shellBatchSize = 10000000; db.repos.find({'repo_id': '${repo_id}'}).pretty()" | grep -vi -e mongodb; done
{
"_id" : ObjectId("6062c930d6729f30184f1d49"),
"repo_id" : "1-my-cv-name-v5_0-9f5fbd88-a829-4fec-9c30-21bcd0f38a69",
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
"display_name" : "Red Hat Enterprise Linux 8 for x86_64 - AppStream RPMs x86_64 8",
"notes" : {
"_repo-type" : "rpm-repo"
},
"scratchpad" : {
},
"content_unit_counts" : {
"modulemd_defaults" : 44,
"erratum" : 1172,
"package_group" : 58,
"modulemd" : 309,
"package_environment" : 4,
"rpm" : 15833,
"yum_repo_metadata_file" : 1
},
"_ns" : "repos",
"last_unit_added" : ISODate("2022-05-10T02:10:11.612Z"),
"last_unit_removed" : ISODate("2021-03-30T06:55:43.340Z")
}
Note that the resulting output indicates that the package belongs to the RHEL 8 AppStream repository in version 5.0 of content view "my-cv-name".
For completeness, confirm these findings against the Foreman database running the following command where rpm_id is the id obtained in the very first command above:
su - postgres -c "psql -P expanded=auto foreman -c \"select katello_repositories.*, katello_root_repositories.name from katello_repository_rpms inner join katello_repositories on katello_repositories.id = katello_repository_rpms.repository_id inner join katello_root_repositories on katello_repositories.root_id = katello_root_repositories.id where rpm_id = <rpm_id>;\""
For example:
root@satellite# su - postgres -c "psql -P expanded=auto foreman -c \"select katello_repositories.*, katello_root_repositories.name from katello_repository_rpms inner join katello_repositories on katello_repositories.id = katello_repository_rpms.repository_id inner join katello_root_repositories on katello_repositories.root_id = katello_root_repositories.id where rpm_id = 127040;\""
-[ RECORD 1 ]-------------+--------------------------------------------------------------------
---------------
id | 2849
pulp_id | 1-my-cv-name-v5_0-9f5fbd88-a829-4fec-9c30-21bcd0f38a69
library_instance_id | 1404
content_view_version_id | 312
relative_path | ACME/content_views/my-cv-name/5.0/content/dist/rhel8/8/x86_64/appstream/os
environment_id |
saved_checksum_type | sha256
distribution_version |
distribution_arch |
distribution_bootable |
distribution_family |
distribution_variant |
container_repository_name |
root_id | 62
remote_href |
publication_href | /pulp/api/v3/publications/rpm/rpm/1eeb7795-ae7e-4ea8-a81e-d13b8c3c3038/
version_href | /pulp/api/v3/repositories/rpm/rpm/04af2288-23ba-4c5d-8f52-c2faa74c30e7/versions/1/
name | Red Hat Enterprise Linux 8 for x86_64 - AppStream RPMs x86_64 8
Once the content view versions that contain the offending package(s) have been identified, please:
- delete the content view version(s)
- remove content orphaned by the deleted content view version(s) by running the command described in article Removing a repository does not delete its content in /var/lib/pulp on Red Hat Satellite Server.
Resume the content migration:
root@satellite# satellite-maintain content prepare
And confirm that it ends successfully:
root@satellite# satellite-maintain content migration-stats
If the content migration still fails please raise a support case and share the actions applied, the information obtained by running the above commands and a sosreport of the Satellite alongside any other relevant details.
For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 to Pulp 3.0 Migration Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0 to Pulp 3.0 Migration Issues
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.