After upgrading to Red Hat Satellite 6.10, Content View publishing fails with error "404 not found"

Solution Verified - Updated

Environment

Red Hat Satellite 6.10
Red Hat Satellite 6.11

Issue

After upgrading Red Hat Satellite to the version 6.10, at certain times, publishing of Content Views and Composite Content Views fails with the following error 

Error message: the server returns an error HTTP status code: 400 Response headers: {"Date"=>"Wed, 09 Mar 2022 04:02:42 GMT", "Server"=>"gunicorn", "Content-Type"=>"application/json", "Vary"=>"Accept,Cookie", "Allow"=>"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "X-Frame-Options"=>"SAMEORIGIN", "Content-Length"=>"62", "Correlation-ID"=>"c628620e-a648-4200-9180-cd47427cec50", "Access-Control-Expose-Headers"=>"Correlation-ID", "Via"=>"1.1 satellite.example.com", "Connection"=>"close"} Response body: {"publication":["Invalid hyperlink - Object does not exist."]}

Error:
PulpRpmClient::ApiError
Error message: the server returns an error HTTP status code: 404 Response headers: {"Date"=>"Wed, 10 Mar 2022 14:52:03 GMT", "Server"=>"gunicorn", "Content-Type"=>"application/json", "Vary"=>"Accept,Cookie", "Allow"=>"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "X-Frame-Options"=>"SAMEORIGIN", "Content-Length"=>"62", "Correlation-ID"=>"c628620e-a648-4200-9180-cd47427cec50", "Access-Control-Expose-Headers"=>"Correlation-ID", "Via"=>"1.1 satellite.example.com", "Connection"=>"close"} Response body: {"detail":"Not found."}

Resolution

This is a known issue and has been escalated to the Red Hat Engineering team via This content is not included.Bugzilla 2047675 and has been fixed in Red Hat Satellite 6.11.

The function of bugfix is to prevent this issue from happening again but for a content-view that was already broken, It needs to be fixed manually as well.

Refer to the following steps to fix the existing content-views:

CAUTION: It's recommended to take the latest snapshot/backup of the affected system before proceeding further.

  • For Red Hat Satellite 6.10, Run the script as mentioned below in order to fix the broken Content Views.

    foreman-rake console <<< cat << EOF
conf.echo = false
Katello::ContentView.where(default: false).all.each do |cv|
  p "Checking content view #{cv.id} #{cv.name}"
  deleted_repo_refs = []
  cv.repository_references.each do |ref|
    begin 
      api = Katello::RepositoryTypeManager.repository_types[ref.root_repository.content_type].pulp3_api_class.new(SmartProxy.pulp_primary)
      api.repositories_api.read(ref.repository_href)
    rescue => e
      if e&.code == 404
        p "Repository #{ref.repository_href} is missing. Deleting it..."
        deleted_repo_refs << ref.repository_href
        ref.destroy!
      end
    end
  end
  if cv.versions.any?
    cv.versions.last.repositories.reject(&:archive?).each do |repo|
      version_href = repo.version_href
      if version_href.nil? || deleted_repo_refs.include?(version_href.gsub(/versions\/\d+\//, ""))
        p "Deleting repository: #{repo.id}, #{repo.pulp_id} (#{repo.relative_path})"
        repo.destroy!
      end
    end
  end
end
EOF

  • For Red Hat Satellite 6.11, Run the modified script as mentioned below:

    foreman-rake console <<< cat << EOF
Katello::ContentView.where(default: false).all.each do |cv|
 p "Checking content view #{cv.id} #{cv.name}"
 deleted_repo_refs = []
 cv.repository_references.each do |ref|
   begin
     api = Katello::RepositoryTypeManager.defined_repository_types[ref.root_repository.content_type].pulp3_api_class.new(SmartProxy.pulp_primary)
     api.repositories_api.read(ref.repository_href)
   rescue => e
     if e&.code == 404
       p "Repository #{ref.repository_href} is missing. Deleting it..."
       deleted_repo_refs << ref.repository_href
       ref.destroy!
     end
   end
 end
 if cv.versions.any?
   cv.versions.last.repositories.reject(&:archive?).each do |repo|
     version_href = repo.version_href
     if version_href.nil? || deleted_repo_refs.include?(version_href.gsub(/versions\/\d+\//, ""))
       p "Deleting repository: #{repo.id}, #{repo.pulp_id} (#{repo.relative_path})"
       repo.destroy!
     end
   end
 end
end
EOF

  • Clean up or unlock all the paused tasks related to Content View Publish action.

    # foreman-rake foreman_tasks:cleanup TASK_SEARCH='label = Actions::Katello::ContentView::Publish' STATES='paused' VERBOSE=true

  • If the foreman-rake script mentioned at first, was finished without showing any errors. Proceed with publishing a new version of all the broken Content Views first and then repeat the same for all broken Composite Content Views.

  • Once done, verify if the client systems connected with the Red Hat Satellite server can see and consume the contents now.

  • Collect a sosreport and Rollback to the snapshot if something seems broken and reach out to This content is not included.Red Hat Technical Support for further assistance.

For more KB articles/solutions related to Red Hat Satellite 6.x Content View Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Content View Issues

Root Cause

  • Content View and Composite Content Views are referring to the same repositories in Pulp (See below). Delete the composite content view will delete the referring Pulp repositories. Therefore, it breaks all the component content views.

       # foreman-rake console
   irb(main):001:0> pp Katello::ContentView.find_by_label("ccv_test").repository_references
   [#<Katello::Pulp3::RepositoryReference:0x0000000013f33320
     id: 25,
     repository_href:
      "/pulp/api/v3/repositories/rpm/rpm/eb4ce2fd-1056-4736-976d-9a7d629fa334/",    <<---- same href
     content_view_id: 17,
     root_repository_id: 2>]

   irb(main):002:0> pp Katello::ContentView.find_by_label("cv_test").repository_references
   [#<Katello::Pulp3::RepositoryReference:0x0000000013c99f88
     id: 7,
     repository_href:
      "/pulp/api/v3/repositories/rpm/rpm/eb4ce2fd-1056-4736-976d-9a7d629fa334/",     <<---- same href
     content_view_id: 2,
     root_repository_id: 2>]

  • This only affects the Content Views that are migrated to Red Hat Satellite 6.10 from Red Hat Satellite 6.9. Those newly created Content Views in Red Hat Satellite 6.10 don't have this issue.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.