Content View Publish Fails to Publish or is Stuck
Environment
Red Hat Satellite 6.8
Issue
- Publishing a new content view version errors before completing (e.g. errors at 87%), or content view publish gets stuck and doesn't finish
- The DistributorPublish subtask of the content view publish fails with
NoMethodError
undefined method `[]' for nil:NilClass
Resolution
- Determine if the content view publish is failing for a specific repository. Examine the subtasks for the content view publish and check for any subtasks that are in the errored state. Then expand that task and check the input and output of the tasks. The output should show something similar to:
NoMethodError
undefined method `[]' for nil:NilClass
---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.26/app/services/katello/pulp/repository.rb:199:in
`lookup_distributor_id'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.26/app/services/katello/pulp/repository.rb:193:in
`block in distributor_publish'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.26/app/services/katello/pulp/repository.rb:191:in
`each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.26/app/services/katello/pulp/repository.rb:191:in
`map'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.26/app/services/katello/pulp/repository.rb:191:in
`distributor_publish'"
- Then review the input and find the
source_repositorysection, and the name which will say which repository is failing:
source_repository:
id: 1240
name: Red_Hat_CodeReady_Linux_Builder_for_RHEL_8_x86_64_RPMs_8
- Remove all repositories found from the content view found from errored subtasks, and then publish a new version of the content view. Then add all of the repositories back to the content view and publish one more time. If the 2nd publish is successful, you can remove the publish that excluded the affected repositories.
For more KB articles/solutions related to Red Hat Satellite 6.x Content View Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Content View Issues
Root Cause
- One or more repositories is missing from an environment or version of a content view.
- This happens either by having a previous content view publish that did not complete successfully, or if the repository for a specific environment/version of the content view is manually removed from pulp.
Diagnostic Steps
- Review the foreman tasks for the content view publish by navigating to
Monitor => Tasksin the Satellite UI. Find the content view publish that failed, click on it, and then click theDynflow consolebutton. Click theRuntab and review if any of the subtasks are showing an error. - If any tasks are showing an error, click on them to expand the input and output values for the task. Find the output values and see if a similar error message exists:
NoMethodError
undefined method `[]' for nil:NilClass
---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.26/app/services/katello/pulp/repository.rb:199:in
`lookup_distributor_id'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.26/app/services/katello/pulp/repository.rb:193:in
`block in distributor_publish'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.26/app/services/katello/pulp/repository.rb:191:in
`each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.26/app/services/katello/pulp/repository.rb:191:in
`map'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.26/app/services/katello/pulp/repository.rb:191:in
`distributor_publish'"
- Then find the input and and see which repository is causing issues (this example shows the CodeReady Linux Builder repository):
source_repository:
id: 1240
name: Red_Hat_CodeReady_Linux_Builder_for_RHEL_8_x86_64_RPMs_8
- Also you can review the
/var/log/foreman/production.logon the Satellite server and find the time when you see the error from the foreman tasks. Please note, the foreman tasks are using the timezone UTC-0, and theproduction.loguses the timezone of the Satellite server, so the time values may differ. In the logs you should see an error about a missing resource, which indicates the organization id, the content view, the lifecycle environment, and the id of the repository in question. The following example shows "1-RHEL_x86_64-Library-9b969c1e-3194-4b10-8d36-2d8f2285de89":
2022-01-03T16:09:17 [I|kat|] GET: https://satellite.example.com/pulp/api/v2/repositories/1-RHEL_x86_64-Library-9b969c1e-3194-4b10-8d36-2d8f2285de89/?details=true: {"content_type"=>"application/json", "accept"=>"application/json"}
404 Not Found: {"http_request_method": "GET", "exception": null, "error_message": "Missing resource(s): repository=1-RHEL_x86_64-Library-9b969c1e-3194-4b10-8d36-2d8f2285de89", "_href": "/pulp/api/v2/repositories/1-RHEL_x86_64-Library-9b969c1e-3194-4b10-8d36-2d8f2285de89/?details=true", "http_status": 404, "error": {"code": "PLP0009", "data": {"resources": {"repository": "1-RHEL_x86_64-Library-9b969c1e-3194-4b10-8d36-2d8f2285de89"}}, "description": "Missing resource(s): repository=1-RHEL_x86_64-Library-9b969c1e-3194-4b10-8d36-2d8f2285de89", "sub_errors": []}, "traceback": null, "resources": {"repository": "1-RHEL_x86_64-Library-9b969c1e-3194-4b10-8d36-2d8f2285de89"
}}
Product(s)
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.