[Satellite 6] 404 or ERROR: MissingResource disabling or enabling or viewing Products and Repositories.

Solution Verified - Updated 5 Aug 2024

Environment

Red Hat Satellite or Proxy 6

Issue

Unable to Enable or Disable a repository in the Red Hat Repositories page. If the option Force Unlock is used on a task that was attempting to disable or enable a repository then, the database is left in an inconsistent state. This cause various errors throughout Red Hat Satellite 6 including:
- Inability to render the Content -> Products UI page
- Inability to synchronize Products containing missing repositories.
- Inability to Publish and Promote Content Views containing missing repositories
- Inability to Delete or remove from an Environment some Content view containing a missing repository
- Inability to properly enable or disable repositories
- Inability to synchronize external Capsule Server.
- The hammer product list returns a HTTP 404 error
- The reindex fails with The following Katello::Repository items could not be indexed due to various reasons.

Resolution

Procedure for `Red Hat Satellite` 6.2.10 or newer

Red Hat Satellite is shipped with a rake script to (not only) fix missing repos. Run the below command on Red Hat Satellite Server :

foreman-rake katello:correct_repositories

After reviewing the command output, re-run it with COMMIT=true to really commit the suggested changes:

foreman-rake katello:correct_repositories COMMIT=true

Procedure for `Red Hat Satellite` 6.2.9 or older

The resolution requires the execution of a repair routine. The steps as follows need to be executed on the Satellite server:

1. Download the repair script and ensure proper permissions:

Script for Satellite 6.2.[0-9]

# curl http://people.redhat.com/~mmccune/repair_repos_62.rake > /usr/share/foreman/lib/tasks/repair_repos.rake 
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
# chmod 644 /usr/share/foreman/lib/tasks/repair_repos.rake

Script for `Red Hat Satellite` 6.1

#  curl http://people.redhat.com/~mmccune/repair_repos.rake > /usr/share/foreman/lib/tasks/repair_repos.rake 
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
# chmod 644 /usr/share/foreman/lib/tasks/repair_repos.rake

2. Execute the repair script to generate the list of repositories that need to be re-created:

# foreman-rake katello:generate_repositories_to_fix
             The following repositories are not present in pulp. A file has been generated at /tmp/foreman_pulp_ids_to_fix containing the pulp ids to be fixed.
              Please verify the list below are indeed repositories you want and fixed. Update /tmp/foreman_pulp_ids_to_fix if anything needs to be changed.
              Then run 'foreman-rake katello:generate_pulp_repositories' to generate their equivalent in pulp.
      

       name: Red Hat Subscription Asset Manager for RHEL 6 Server RPMs x86_64 6Server
       pulp_id: RH-EXTRAS-Test-Red_Hat_Enterprise_Linux_Server-Red_Hat_Subscription_Asset_Manager_for_RHEL_6_Server_RPMs_x86_64_6Server
       product: Red Hat Enterprise Linux Server
       organization: RH-EXTRAS-Test

3. Verify with Red Hat Support that this list is correct. Run the repair routine:

# foreman-rake katello:generate_pulp_repositories
Completed generating pulp repo for the following repository

       name: Red Hat Subscription Asset Manager for RHEL 6 Server RPMs x86_64 6Server
       pulp_id: RH-EXTRAS-Test-Red_Hat_Enterprise_Linux_Server-Red_Hat_Subscription_Asset_Manager_for_RHEL_6_Server_RPMs_x86_64_6Server
       product: Red Hat Enterprise Linux Server
       organization: RH-EXTRAS-Test
    
You need to manually sync this repository in the specified organization before proceeding.

4. Synchronize and publish any repositories listed from the above repair tool. Below steps trigger synchronization of all those repositories. It is expected the repositories with a feed from CDN or other resource will be properly synchronized, while

pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)

for repo in $(cat /tmp/foreman_pulp_ids_to_fix); do
  curl -i -H "Content-Type: application/json" -X POST -d "{\"id\":\"$repo\",\"override_config\":{\"validate\":true}}" -u admin:$pulpAdminPassword https://$(hostname -f)/pulp/api/v2/repositories/$repo/actions/sync/
  curl -i -H "Content-Type: application/json" POST -d "{\"id\":\"$repo\",\"override_config\":{\"force_full\":true}}" -u admin:$pulpAdminPassword https://$(hostname -f)/pulp/api/v2/repositories/$repo/actions/publish/
done

5. You also may need to delete and re-publish any Content Views that contained this repository. Contact Red Hat Support if you need assistance with this step. If you try to fix this by yourself, then:

wait til previous pulp tasks are running/pending and continue only if below command returns no task:

pulp-admin -u admin -p $pulpAdminPassword tasks list --state="running,waiting"

identify what Content View in what LifecycleEnvironment needs to be re-published / re-promoted. Check the /tmp/foreman_pulp_ids_to_fix for the repo list. If the structure of the repository name matches:

Organization-LifecycleEnvironment-ContentView-someNumber-*

then the ContentView (of given version per "someNomber") in relevant LifecycleEnvironment needs to be re-published / promoted to the same L.E. again.

For more KB articles/solutions related to Red Hat Satellite 6.x Repository Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Repository Issues.

Root Cause

Force Unlocked Tasks
Data mismatch

Diagnostic Steps

Either /var/log/messages or /var/log/foreman/production.log contains error with string Missing resource(s). In case of /var/log/messages, an example of such error:

May 26 11:06:30 capsule01 pulp: pulp.server.webservices.middleware.exception:ERROR: Missing resource(s): distributor=<some_repository>
May 26 11:06:30 capsule01 pulp: pulp.server.webservices.middleware.exception:ERROR: Traceback (most recent call last):
May 26 11:06:30 capsule01 pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in __call__
May 26 11:06:30 capsule01 pulp: pulp.server.webservices.middleware.exception:ERROR:     return self.app(environ, start_response)
May 26 11:06:30 capsule01 pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/postponed.py", line 42, in __call__
..
May 26 11:06:30 capsule01 pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/distributor.py", line 61, in get_distributor
May 26 11:06:30 capsule01 pulp: pulp.server.webservices.middleware.exception:ERROR:     raise MissingResource(distributor=distributor_id)

In case of /var/log/foreman/production.log, an example of such error:

[ERROR 2015-05-18 16:57:08 pulp_rest  #2361] RestClient.get "https://satellite01.example.com/pulp/api/v2/repositories/<repository_name>//history/sync/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "Authorization"=>"OAuth oauth_consumer_key=\"katello\", oauth_nonce=\"5qPT4PzV1KdArKjQPxflT1nAwWUYDPSR7UdOWjkQQ\", oauth_signature=\"YHQdzb%2FvgG1WtKSvOmUGirGwjho%3D\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"1431986228\", oauth_version=\"1.0\"", "accept"=>"application/json", "content_type"=>"application/json", "pulp-user"=>"satadmin-43e7f278"
 | \n# => 404 NotFound | application/json 578 bytes
 |
  Rendered /opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/views/katello/api/v2/products/index.json.rabl within katello/api/v2/layouts/collection (6886.3ms)
exception when talking to a remote client: 404 Resource Not Found RestClient::ResourceNotFound: 404 Resource Not Found
Body: {"http_request_method": "GET", "exception": null, "error_message": "Missing resource(s): resource_id=<repositor_name>", "_href": "/pulp/api/v2/repositories/<repository_name>/history/sync/", "http_status": 404, "error": {"code": "PLP0009", "data": {"resources": {"resource_id": "<repository_name>"}}, "description": "Missing resource(s): resource_id=<repository_name>", "sub_errors": []}, "traceback": null, "resources": {"resource_id": "<repository_name>"}}
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.1.0/lib/runcible/base.rb:97:in `block in get_response'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in `call'
..

Another possibility is task export contains Failed to copy metadata. See errors for more details. / Source Directory (%s), does not exist, cannot publish. and/or /var/log/messages having: FailedException: Unable to sync a repository that has no feed

SBR

SysMgmt

Product(s)

Red Hat Satellite

Components

foreman

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.