Content migration from pulp2 to pulp3 fails with error Katello::Errors::Pulp3Error: Validation failed: resources missing in Red Hat Satellite 6.9

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.9

Issue

  • Execution of satellite-maintain content prepare fails with any of the following errors even before starting the actual migration process.

    ForemanTasks::TaskError: Task df64d1b7-3279-4684-b3af-5dcc3d5402ea: 
Katello::Errors::Pulp3Error: Validation failed: resources missing {'repositories_missing_distributors': ['1-EPEL-v1289_0-RE-EPEL-EPEL-7-x86_64']}
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/lib/katello/tasks/pulp3_migration.rake:43:in `block (2 levels) in <top (required)>'

     

    ForemanTasks::TaskError: Task cde6274a-38ec-4adf-9ee2-9eba814714b0: Katello::Errors::Pulp3Error: 
Validation failed: resources missing {'repositories': ['9039817e-7178-4654-af63-10086ffa83b3', '466279f4-d9fe-4250-b201-d35afe75c412', '6eda3c7b-e689-45c9-954a-b5eb0ff3e4b2', '561dc2e0-bc20-43ed-a5aa-0a8b80121cb4', 'dfe4d41e-af04-4c31-b061-195acd6dc6bf', '259bb346-ca50-431d-b723-abf198f452a5'], 
'repositories_missing_importers': ['9039817e-7178-4654-af63-10086ffa83b3', '466279f4-d9fe-4250-b201-d35afe75c412', '6eda3c7b-e689-45c9-954a-b5eb0ff3e4b2', '561dc2e0-bc20-43ed-a5aa-0a8b80121cb4', 'dfe4d41e-af04-4c31-b061-195acd6dc6bf', '259bb346-ca50-431d-b723-abf198f452a5'], 
repositories_missing_distributors': ['9039817e-7178-4654-af63-10086ffa83b3', '466279f4-d9fe-4250-b201-d35afe75c412', '6eda3c7b-e689-45c9-954a-b5eb0ff3e4b2', '561dc2e0-bc20-43ed-a5aa-0a8b80121cb4', 'dfe4d41e-af04-4c31-b061-195acd6dc6bf', '259bb346-ca50-431d-b723-abf198f452a5']}
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/lib/katello/tasks/pulp3_migration.rake:43:in `block (2 levels) in <top (required)>'

Resolution

  • This problem has been reported to the Red Hat Engineering team and is being tracked by the This content is not included.Bugzilla 2026188.

  • Rather than being a bug, This is more of a consistency detection check which fails and Two different approaches are to be considered for either of these reported behaviors.

    • If none of the pulp_id is present in repos, repo_importers and repo_distributors collection of mongodb but they are present in the foreman database, then

      • Ensure that no paused tasks are present related to those affected repositories. If there are any, try to resume the tasks and get them completed successfully.

      • Execute the following command to recreate the missing pulp entries in the Satellite server.

        # foreman-rake katello:correct_repositories COMMIT=true --trace
        • Ensure that none of the tasks related to the repo creation is getting stuck in paused step or ending up with any errors.

     

    • If the pulp_id is present in the foreman database but is missing from any one or two of the collections in mongodb, then

      • Identify the path of the repo from the katello_repositories.txt file.

      • That path should translate to which repo it is or which version of a content_view that repository belongs to.

      • If the path is indeed related to a broken version of content-view, then delete that affected version (considering it is not in use).

 

  • For any further assistance required, open up a This content is not included.support case with Red Hat with the following data attached to the case i.e.

    • A sosreport from the satellite server.
    • The .txt files are to be collected after executing the commands from the Diagnostic Steps section.

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 3.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 3.0-related Issues

Root Cause

  • The content prepare step has a precheck that validates the consistency of the repositories present in both foreman and pulp databases.

  • Under normal circumstances, it's expected that a pulp_id related to a repository that is present in the foreman database should also be part of repos, repo_importers, and repo_distributors collection within the pulp database (mongodb). But due to any reason if one or all of the collections are missing the entry of apulp_id whereas the same is present in the foreman database, then the error with resources missing is expected to be reported.

Diagnostic Steps

  • Collect the following data from the pulp and foreman databases of Red Hat Satellite server.

       # mongo pulp_database --eval "DBQuery.shellBatchSize = 10000000; db.repo_importers.find().pretty().shellPrint()" &> repo_importers_satellite.txt

   # mongo pulp_database --eval "DBQuery.shellBatchSize = 10000000; db.repo_distributors.find().pretty().shellPrint()" &> repo_distributors_satellite.txt

   # mongo pulp_database --eval "DBQuery.shellBatchSize = 10000000; db.repos.find().pretty().shellPrint()" &> repos_satellite.txt

   # su - postgres -c "psql foreman -c \"select * from katello_repositories;\"" &> katello_repositories.txt

 

  • Take any one of the reported pulp_id from the error and look for its presence in all the files.

       # grep 1-EPEL-v1289_0-RE-EPEL-EPEL-7-x86_64 *.txt

   repos_satellite.txt:	"repo_id" : "1-EPEL-v1289_0-RE-EPEL-EPEL-7-x86_64",

   katello_repositories.txt: 315195 | 1-EPEL-v1289_0-RE-EPEL-EPEL-7-x86_64   |                 192 |                   14026 | RE/content_views/EPEL/1289.0/custom/EPEL/EPEL-7-x86_64

   repo_importers_satellite.txt:	"repo_id" : "1-EPEL-v1289_0-RE-EPEL-EPEL-7-x86_64",
    • As it can be observed that two collections of mongodb have the entry of the pulp_id. But the repo_distributors_satellite.txt file does not contain it. So this justifies the first error message reported.

 

  • An example related to the second error message will be as below i.e.

       # grep 9039817e-7178-4654-af63-10086ffa83b3 *.txt

   katello_repositories.txt:  200 | 9039817e-7178-4654-af63-10086ffa83b3 |          |             2 | ITER/Library/custom/p-ccs-v6-ws/iter-rhel-x86_64-workstation-7u4-thirdparty-oracle-java
    • Here, since none of the files collected from the mongodb has any entries of the pulp_id, It's safe to assume that pulp_database is completely missing the information of the same repo whereas the foreman database has it.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.