Red Hat Satellite 6 CDN Repository sync failed with RPM1004 Error retrieving metadata Forbidden.

Solution Verified - Updated

Environment

  • Red Hat Satellite 6

Issue

  • While syncing repository on Red Hat Satellite gives following error in /var/log/foreman/production.log:

    2018-01-04 20:28:59  [foreman-tasks/action] [E] RPM1004: Error retrieving metadata: Forbidden (Katello::Errors::PulpError)

Resolution

  1. Make sure that the allocation\manifest for Satellite is present on the This content is not included.Red Hat Customer Portal Subscriptions Allocations page and having valid subscriptions as a part of it.

    • If it does not exist, create one and export a new manifest.
    • If it exists but no subscriptions are present inside it, make sure to add them in the allocation.
    • If future-dated subscriptions have already been added to the manifest, the workaround in step 2 below is also needed once the subscriptions become active.

  2. As a first workaround, please proceed with a refresh of the Manifest:

    • On Satellite GUI --> Content --> Subscriptions --> Manage Manifest --> Refresh Manifest
    • When the refresh is completed, try to synchronize your repository again

  3. If the issues persist, try the following steps:

    Don't forget to change ACME to your Organization name.

    • For Satellite 6.5 or higher

            # foreman-rake console
      organization = Organization.find_by(name: 'ACME')
      repositories = ::Katello::Repository.in_default_view.in_product(::Katello::Product.redhat.in_org(organization))
repositories.each do |repo|
      task = ForemanTasks.async_task(Actions::Katello::Repository::RefreshRepository, repo)
      puts "task.id #{task.id}"
      end

    • For Satellite below 6.5

      # foreman-rake console
organization = Organization.find_by(name: 'ACME')
repositories = ::Katello::Repository.in_default_view.where(:product_id => ::Katello::Product.redhat.in_org(organization))
repositories.each do |repo|
  task = ForemanTasks.async_task(Actions::Katello::Repository::RefreshRepository, repo)
  puts "task.id #{task.id}"
end

  4. If the issues persist, you may proceed with the following steps:

    • Note: (Take Satellite snapshot before performing the recommended steps )

    • Run below command on Satellite Server:

      # su - postgres -c "psql -d foreman -c 'select label,count(label),state from foreman_tasks_tasks where state <> '\''stopped'\'' and label not in ('\''Actions::Katello::EventQueue::Monitor'\'', '\''Actions::Candlepin::ListenOnCandlepinEvents'\'') group by label,state;'"

      If this command displays any tasks in a paused state, remove them all with below command:

      • For Satellite 6.5 or higher*

        # foreman-maintain service stop --only dynflowd
# foreman-rake foreman_tasks:cleanup TASK_SEARCH='label ~ *' STATES=paused
# foreman-maintain service start --only dynflowd

      • *For Satellite below 6.5

        # katello-service stop --only foreman-tasks
# foreman-rake foreman_tasks:cleanup TASK_SEARCH='label ~ *' STATES=paused
# katello-service start --only foreman-tasks

      • Clean the orphaned Foreman task objects and update the database:

        # foreman-rake katello:delete_orphaned_content RAILS_ENV=production >/dev/null
# foreman-rake katello:clean_backend_objects
# foreman-rake katello:reimport
# foreman-maintain service restart
# satellite-installer --scenario satellite  --upgrade

For more KB articles/solutions related to Red Hat Satellite 6.x Repository Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Repository Issues.

Diagnostic Steps

  • Check the errors in /var/log/messages file on the Satellite server:

    8 10:36:08 u999sat6 pulp: celery.worker.job:INFO: Task 
pulp.server.managers.repo.sync.sync[b68a0d20-7da2-4f57-bafe-eb9deb621583] raised expected: 
PulpCodedException()
May  8 10:36:08 sat6 pulp: celery.worker.job:INFO: Task 
pulp.server.async.tasks._release_resource[eec4a821-508d-4804-9e3b-985457bd2386] 
succeeded in 0.00811915285885s: None
May  8 10:36:08 sat6 pulp: py.warnings:WARNING: (796254-24224) /usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py:806: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead
May  8 10:36:08 sat6 pulp: py.warnings:WARNING: (796254-24224)   
sync_result_collection.save(sync_result)
May  8 10:36:08 sat6 pulp: py.warnings:WARNING: (796254-24224)
May  8 10:36:08 sat6 pulp: pulp.server.async.tasks:INFO: Task failed : [93cd67e3-4209-4522-9aae-e14685a2f53e] : Error retrieving metadata: Forbidden

  • Check if the proxy is properly configured in the Json files in directory /etc/pulp/server/plugins.conf.d.

  • Error for the file /usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py:

    803         # Do an update instead of a save in case the importer has changed the scratchpad
804         model.Importer.objects(repo_id=repo_obj.repo_id).update(set__last_sync=sync_end_timestamp)
805         # Add a sync history entry for this run
806         sync_result_collection.save(sync_result)
807         # Ensure counts are updated
808         rebuild_content_unit_counts(repo_obj)
809 
810     fire_manager.fire_repo_sync_finished(sync_result)
811     if sync_result.result == RepoSyncResult.RESULT_FAILED:
812         raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))

  • Check error in the DynFlow logs for repository sync task :

    RestClient::InternalServerError- 500 Internal Server Error
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.