Capsule sync fails with error RPM1008: Checksum type "sha1" is not available for all units in the repository.

Solution Verified - Updated

Environment

  • Red Hat Satellite 6

Issue

  • A Capsule sync fails with error(s) like:

      RPM1008: Checksum type "sha1" is not available for all units in the repository. Make sure those units have been downloaded

  • The problem persists even after republishing or re-promoting relevant Content View or after removing the pulp repository from the Capsule

Resolution

Solution 0: Ensure if published repository metadata of the problematic repo on Satellite dont contain RPMs with multiple different checksumtypes

In /var/lib/pulp/published/yum/master/yum_distributor/${repo}/*/repodata, check:

  • *primary.xml.gz file for presence of different <checksum pkgid="YES" type="sha256"> types
  • repomd.xml file for presence of checksum type or open-checksum type that differs from checksum pkgid in *primary.xml.gz

If there are more than one type inside primary.xml.gz or if repomd.xml differs from primary.xml.gz, let forcefully regenerate the repository metadata for all affected repos in the right ordering: run iteratively in foreman-rake console:

conf.echo = false
pulp_ids = [   # use proper pulp IDs, below are just examples
"ORG-Lifecycle-CV-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_6_Server_-_Extras_RPMs_x86_64",
"ORG-Library-RHEL7-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_-_Extras_RPMs_x86_64"]

repos = Katello::Repository.where(pulp_id: pulp_ids)
default_org_repos, cv_repos  = repos.partition(&:library_instance?)
composite_content_view_versions, content_view_versions = cv_repos.map(&:content_view_version).partition{ |cvv| cvv.content_view.composite? }

pp "Number of affected repos: #{repos.size}"
pp "Number of default org repos: #{default_org_repos.size}"
pp "Number of affected CV versions: #{content_view_versions.size}"
pp "Number of affected CCV versions: #{composite_content_view_versions.size}"

# Regenerate the Default Org Repos first. Wait until all tasks are finished in Web UI -> Monitor -> Tasks page before proceeding.
default_org_repos.each do |repo|
  ForemanTasks.async_task(Actions::Katello::Repository::MetadataGenerate, repo, force: true)
end

Now wait until all Satellite MetadataGenerate tasks complete, then continue:

# Regenerate the affected content views. Again, wait until all tasks are finished in Web UI -> Monitor -> Tasks page before proceeding.
content_view_versions.sort_by(&:content_view_id).each do |cvv|
  ForemanTasks.async_task(::Actions::Katello::ContentViewVersion::RepublishRepositories, cvv)
end

Again, wait until all Satellite MetadataGenerate tasks complete, then continue:

# Regenerate the affected composite content views. Yet again, wait until all tasks are finished in Web UI -> Monitor -> Tasks page before proceeding.
composite_content_view_versions.sort_by(&:content_view_id).each do |ccvv|
  ForemanTasks.async_task(::Actions::Katello::ContentViewVersion::RepublishRepositories, ccvv)
end
exit

And once again, wait until the latest bulk of MetadataGenerate` tasks complete. Then initiate a new Capsule sync.

Solution 1: We recommend you to attempt this solution first. Only attempt Solution 2 If this doesn't solve your issue.

  • SSH to the terminal of the problematic Capsule

  • For security reasons, we recommend you to take a snapshot of the Capsule before proceeding to the next step.

  • From the Capsule, run the following query:

      $ mongo pulp_database --eval 'db.repo_distributors.find({"distributor_type_id" : "yum_distributor", "config.checksum_type": {"$exists": true}}, {"_id": 0, "repo_id": 1, "config.checksum_type": 1}).count()'
  
  # Example output:
  MongoDB shell version v3.4.9
  connecting to: mongodb://127.0.0.1:27017/pulp_database
  MongoDB server version: 3.4.9
  120  <========= Affected repo count.

  • If the above query returns a count that is larger than 0 than run the following command from the Capsule to reset the "config.checksum_type" setting. If the above query returns 0 count than you need to proceed to the Solution 2.

      $ mongo pulp_database --eval 'db.repo_distributors.update({"distributor_type_id" : "yum_distributor", "config.checksum_type": {"$exists": true}}, {$unset: {"config.checksum_type": ""}}, {"multi": true})'
  
  # Example output:
  MongoDB shell version v3.4.9
  connecting to: mongodb://127.0.0.1:27017/pulp_database
  MongoDB server version: 3.4.9
  WriteResult({ "nMatched" : 120, "nUpserted" : 0, "nModified" : 120 })   <==== This should returns the number of matched and modified records

  • After that, trigger an Optimized Capsule sync through the Satellite Web UI and wait until the task is finished successfully.

  • If the issue persist, you need to proceed to the Solution 2.

Solution 2: Warning: the below procedure might not be bullet-proven yet, and some steps might require some checks / review from Red Hat side. Use with caution or under a supervision of Red Hat support.

Notice: since some commands are of huge length, the below snippets contain an extra empty line after each command to prevent copy&paste issues with lost or extra newlines.

The following procedure should be applied for each repository failed to be synchronized. To identify all those repos, click in WebUI to the Caps Sync task details, click to Dynflow console, expand all yellow or red steps and copy all repo_pulp_id: values. The procedure consists of several steps:

  1. Identify root repositories
  2. Find units checksumtype(s)
  3. Optionally unify the checksumtype - in all cloned repos(!)
  4. Update the katello repos and pulp repos' distributors
  5. Re-publish the repos
  6. Attempt another Capsule sync

Identify root repositories

  • apply for all repo_pulp_id: values (use them in the 1st line of the script):

      $ for repo in repo_pulp_id_1 repo_pulp_id_2 repo_pulp_id_3; do
  
  su - postgres -c "psql foreman -c \"COPY (SELECT pulp_id FROM katello_repositories WHERE ( relative_path like '%/Library/content/%' or relative_path like '%/Library/custom/%') AND root_id IN (SELECT root_id FROM katello_repositories WHERE pulp_id = '${repo}')) TO STDOUT;\""
  
  done | sort -u > root_repositories.txt
  
  $ cat root_repositories.txt

Find units checksumtypes

Each repository in root_repositories.txt must have just one checksumtype in below outputs:

$ rm -f checksumtypes.*
       
$ for repo_id in $(cat root_repositories.txt); do

    for type in $(mongo pulp_database --eval "db.repos.find({'repo_id': '$repo_id'}, {'content_unit_counts': 1, '_id': 0}).shellPrint()" | grep content_unit_counts | json_reformat | grep \" | cut -d\" -f2 | grep -v content_unit_counts); do
        
        for unit in $(mongo pulp_database --eval "DBQuery.shellBatchSize = 10000000; db.repo_content_units.find({'repo_id': '$repo_id', 'unit_type_id': '$type'}, {'unit_id': 1, '_id': 0}).shellPrint()" | grep unit_id | cut -d\" -f4); do
        
            mongo pulp_database --eval "DBQuery.shellBatchSize = 1; db.units_rpm.find({}, {'checksumtype': 1, '_id': 0}).pretty()" | grep ' : "' | awk -F " : \"" '{ print $2 }' | cut -d\" -f1; echo
        
        done | sort -u > checksumtypes.${repo_id}.${type}.txt
        
        if [ $(cat checksumtypes.${repo_id}.${type}.txt | wc -w) -gt 0 ]; then
        
            echo "$repo_id : $type : $(cat checksumtypes.${repo_id}.${type}.txt)"
        
        fi
        
    done
        
    cat checksumtypes.${repo_id}.*.txt | sort -u | grep -v ^$ > checksumtypes.repo.${repo_id}.txt
        
    echo
        
done

Each repository must have just one checksumtype and this must be used further. If more checksumtypes are noticed for a repo, apply the step "Unify the checksumtypes" first.

Optionally unify the checksumtype for a repo

This step must be applied only when a repository contains units of multiple checksum types. Sadly the only way to move forward here is to remove either unit type (and optionally add later on, not covered in this solution).

Assume a repo TEST has RPMs with checksumtype sha1 but SRPMs with sha256. We can unify the content checksumtypes just by removing the SRPMs from the repo:

$ repo_id=TEST

$ type=srpm

$ pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)

$ curl -i -H "Content-Type: application/json" -X POST -d "{\"criteria\": {\"type_ids\": [\"${type}\"]}}" -u admin:$pulpAdminPassword https://$(hostname -f)/pulp/api/v2/repositories/${repo_id}/actions/unassociate/

In case the repo belongs to a Content View (the repo ID contains a CV name), we must remove the SRPM units also from "CV versioned" repo:

$ vers_repo_id=$(su - postgres -c "psql foreman -c \"COPY (SELECT pulp_id FROM katello_repositories WHERE environment_id IS null AND (content_view_version_id,root_id) IN (SELECT content_view_version_id,root_id FROM katello_repositories WHERE pulp_id = '${repo_id}')) TO STDOUT;\"")

$ echo $vers_repo_id

$ curl -i -H "Content-Type: application/json" -X POST -d "{\"criteria\": {\"type_ids\": [\"${type}\"]}}" -u admin:$pulpAdminPassword https://$(hostname -f)/pulp/api/v2/repositories/${vers_repo_id}/actions/unassociate/

For each such updated repository, the repo must be deleted from the Capsule, so run on the Capsule:

$ repo_id=TEST

$ pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)

$ curl -ks -u admin:$pulpAdminPassword -X DELETE https://$(hostname -f)/pulp/api/v2/repositories/${repo_id}/

# ALSO delete the versioned repo id, as printed in "echo" command in previous code snippet - ONLY when relevant

$ vers_repo_id=TEST_VERSIONED_REPO_ID

$ curl -ks -u admin:$pulpAdminPassword -X DELETE https://$(hostname -f)/pulp/api/v2/repositories/${vers_repo_id}/

Update the katello repos and pulp repos' distributors

  • First, generate the list of cloned repos and their desired checksumtypes:
$ for f in checksumtypes.repo.*.txt; do

    rrepo=$(echo $f | cut -d'.' -f3)

    if [ $(cat $f | wc -w) -ne 1 ]; then

        echo "ERROR: root repo $rrepo has multiple checksumtypes ($(cat checksumtypes.repo.*.txt)), unify the repo first!"

        exit 1

    fi

    cksum=$(cat $f)

    for crepo in $(su - postgres -c "psql foreman -c \"COPY (SELECT pulp_id FROM katello_repositories WHERE root_id IN (SELECT root_id FROM katello_repositories WHERE pulp_id = '${rrepo}')) TO STDOUT;\""); do

        echo $crepo $cksum

    done 

done > fix_repo_checksums.txt
  • Now, apply that desired settings to both katello and pulp:
$ cat fix_repo_checksums.txt | while read repo cksum; do

    mongo pulp_database --eval "db.repo_distributors.update({ 'repo_id': '${repo}', 'distributor_type_id': 'yum_distributor' }, { \$set: { 'config.checksum_type': '${cksum}' } })"

done

$ cat fix_repo_checksums.txt | while read repo cksum; do

    echo "repo = Katello::Repository.find_by_pulp_id(\"${repo}\")"

    echo "repo.saved_checksum_type = \"${cksum}\""

    echo "repo.save!"

done > repos_update.rake

$ cat repos_update.rake | foreman-rake console

Re-publish the repos

  • Re-publish all updated repos. This can be done for all root repos + their clones concurrently:
$ pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)

$ cat fix_repo_checksums.txt | while read repo cksum; do

    curl -i -H "Content-Type: application/json" -X POST -d "{\"id\":\"${repo}\",\"override_config\":{\"force_full\":true}}" -u admin:${pulpAdminPassword} https://$(hostname -f)/pulp/api/v2/repositories/${repo}/actions/publish/

    echo    # pulp REST API responds without trailing '\n'

done
  • Wait until all publish tasks are complete - the left most numerical column of below command output must show just zeros:
$ qpid-stat --ssl-certificate=/etc/pki/pulp/qpid/client.crt  -b amqps://localhost:5671 -q | grep resource

Attempt another Capsule sync

Invoke a new (Optimised) Capsule sync. Optionally, one can synchronize just the repos that failed previously. If this option is chosen, then run the below commands with customised Capsule ID (2nd line) and list of failed repos (the repo_pulp_id: values list) in 3rd line:

$ echo "require 'ostruct'" > sync_tasks.rake

$ capsuleID=2

$ for repo in repo_pulp_id_1 repo_pulp_id_2 repo_pulp_id_3; do

    echo "ForemanTasks.async_task(::Actions::Pulp::Consumer::SyncCapsule, OpenStruct.new(:pulp_id => \"${repo}\"), OpenStruct.new(:id => ${capsuleID}), { :remove_missing => true})"

done >> sync_tasks.rake

$ cat sync_tasks.rake | foreman-rake console

If the Capsule sync fails again for some repository in some Content view, check if it really has units of one type - and if not so, unify them (re-run the whole procedure including the 2nd step of repo unification).

For more KB articles/solutions related to Red Hat Satellite 6.x Capsule Sync Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Capsule Sync Issues

Root Cause

  • Few possible scenarios happened, like:
    • to a repository with RPM packages of one checksumtype (say, sha256), an SRPM of a different checksumtype (say, sha1) was added
    • checksumtype of packages in upstream (on CDN or in 3rd party repo) was updated, while metadata of the repository on Satellite stick to the old checksumtype
  • As a result, Satellite holds a repository with multiple checksumtypes in either published errata metadata or RPMs or SRPMs or other units
    • such a repository fails to be synchronized to a Capsule, since pulp detects the checksumtypes mismatch during the sync
  • If that repository is contained in some Content View (CV), the mismatch is cloned alongside with cloning the root repository. So the unification of checksumtypes must be performed on a tree of cloned repositories, at the end.
  • If the mismatch is between published metadata and content, then setting published metadata type and re-publishing the repos will ensure the further Capsule sync will work - as it will fetch the whole repo "completely".
  • If the mismatch is between units of a repo (say SRPMs and RPMs have different checksumtypes), then the "easiest" way of resolving is to remove the units of "different" type from the repo(s) - an attempt to change checksumtype of the "offending" units might break other repositories the same way.

Diagnostic Steps

Capsule sync fails on syncing some particular repos with error:

RPM1008: Checksum type "sha1" is not available for all units in the repository. Make 
sure those units have been downloaded.

The particular checksumtype in the error string does not matter (much). It does not necessarily indicate either the "old" or "new" checksumtype of the affected repo.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.