How to determine the repository to run Verify Checksum on for reported corrupted RPMs

Solution Verified - Updated

Environment

Satellite 6.9

Issue

Satellite 6.9 upgrade to 6.10 requires a content migration where all content on the Satellite is verified before importing into the Postgres database used by Pulp3. A message like the one below is seen when content on the filesystem does not match the checksum of the RPM in the database:

============Missing/Corrupted Content Summary================
WARNING: MISSING OR CORRUPTED CONTENT DETECTED
Corrupted or Missing Rpm: 1000/173142
Corrupted or missing content has been detected, you can examine the list of content in /tmp/unmigratable_content-20220109-11817-1oil4a6 and take action by either:
1. Performing a 'Verify Checksum' sync under Advanced Sync Options, let it complete, and re-running the migration
2. Deleting/disabling the affected repositories and running orphan cleanup (foreman-rake katello:delete_orphaned_content) and re-running the migration
3. Manually correcting files on the filesystem in /var/lib/pulp/content/ and re-running the migration
4. Mark currently corrupted or missing content as skipped (foreman-rake katello:approve_corrupted_migration_content).  This will skip migration of missing or corrupted content.

The resolution for this solution article is to help a user identify the repository that an RPM belongs to so that they may know which repositories to perform 'Verify Checksum' on.

Resolution

Red Hat Satellite 6.9.8 or below:

Copy the following python script to the Satellite and run using the directory path provided by the error message in the "Missing/Corrupted Content Summary". In the example above the path of the directory is /tmp/unmigratable_content-20220109-11817-1oil4a6.
Example:

# python corrupted_migration.py /tmp/unmigratable_content-20220109-11817-1oil4a6/Rpm

IMPORTANT: Be sure to add the filename 'Rpm' to the argument when running the command as seen in the example above.

corrupted_migration.py script:

import pymongo
import sys

RPM_FILE = sys.argv[1]

rpm=[]
with open(RPM_FILE, 'r') as tmp:
    for package in tmp:
        rpm.append(package.strip())
myclient=pymongo.MongoClient('mongodb://localhost:27017/')
mydb=myclient['pulp_database']
CORRUPTED_REPOS=[]
for package in rpm:
    mydoc = mydb['units_rpm'].find({"filename": package}, {"downloaded": 1})
    for x in mydoc:
        query2 = mydb['repo_content_units'].find({'unit_id': x['_id']},{'repo_id': 1})
        for y in query2:
            query3 = mydb['repos'].find({'repo_id': y['repo_id']},{'display_name': 1})
            for z in query3:
                if z['display_name'] not in CORRUPTED_REPOS:
                    CORRUPTED_REPOS.append(z['display_name'])

for name in CORRUPTED_REPOS:
        print(name)

 

Red Hat Satellite 6.9.9 or more:

No additional scripts are required and the file /tmp/unmigratable_content-20220109-11817-1oil4a6/Rpm would contain all the details to get started with the investigation.

Following are some examples of how to parse the required data from the file /tmp/unmigratable_content-20220109-11817-1oil4a6/Rpm:

  • Content Layout:

    # head -3 /tmp/unmigratable_content-20220109-11817-1oil4a6/Rpm | column -s, -t
filename                           id      Repository Name                                        Content View Name  Content View Version
firefox-60.2.0-1.el7_5.x86_64.rpm  189548  Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server  RHEL7.8            1.0
firefox-60.2.0-1.el7_5.x86_64.rpm  189548  Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server  RHEL7-12012021     1.0

  • List of unmigrated packages marked as missing or corrupted:

    # cat /tmp/unmigratable_content-20220109-11817-1oil4a6/Rpm | awk -F',' 'NR>1{ print $1}' | sort -u
firefox-60.2.0-1.el7_5.x86_64.rpm
rh-mariadb101-galera-25.3.12-12.el7.x86_64.rpm

  • List of repositories related to the affected RPMs:

    # cat /tmp/unmigratable_content-20220109-11817-1oil4a6/Rpm | awk -F',' 'NR>1{ print $3}' | sort -u
Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.5
Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7
Red Hat Enterprise Linux 7 Server RPMs x86_64 7.8
Red Hat Enterprise Linux 7 Server RPMs x86_64 7.9
Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server
Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 RHEL 7 Server EUS x86_64 7.3
Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 RHEL 7 Server EUS x86_64 7.7
Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server x86_64 7Server

  • List of content-views that contains the affected repositories:

    # cat /tmp/unmigratable_content-20220109-11817-1oil4a6/Rpm | awk -F',' 'NR>1{ print $4}' | sort -u | grep -v "Default Organization View"
RHEL7-09012021
RHEL7.7-EUS 
RHEL7.8
RHEL7-DMZ-Apr-2022
RHEL7-DMZ-Mar-2022

 

For more KB articles/solutions related to Red Hat Satellite 6.x Repository Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Repository Issues.

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 to Pulp 3.0 Migration Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0 to Pulp 3.0 Migration Issues

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.