Red Hat Satellite generates inefficient queries consuming resources and taking long time to run when trying to cleanup orphans

Solution Verified - Updated

Environment

Red Hat Satellite 6.14.1

Issue

  • The task Actions::Katello::OrphanCleanup::RemoveOrphanedContentUnits runs automaticaly and is located in /etc/cron.d/katello.

  • The script for initiating the removal of orphaned content is scheduled in the crontab:

      00 22 * * 0 root    foreman-rake katello:delete_orphaned_content RAILS_ENV=production >/dev/null 2>&1

  • This script, which deletes the orphaned contents, generates inefficient queries, consumes resources, and takes a long time to run.

Resolution

  • In order to remediate this slowness, apply this patch:

      diff --git a/app/models/katello/concerns/pulp_database_unit.rb 
  b/app/models/katello/concerns/pulp_database_unit.rb
  index 675d28e315..ac48f0a369 100644
  --- a/app/models/katello/concerns/pulp_database_unit.rb
  +++ b/app/models/katello/concerns/pulp_database_unit.rb
  @@ -169,7 +169,7 @@ module Katello

      def orphaned
        if many_repository_associations
  -       where.not(:id => repository_association_class.select(unit_id_field))
  +        left_joins(repository_association.to_sym).where("#{repository_association_class.table_name}.# 
  {unit_id_field}" =>  nil)
        else
         where.not(:repository_id => ::Katello::Repository.select(:id))
        end

  • Apply the patching using the command patch:

      patch  /usr/share/gems/gems/katello-4*/app/models/katello/concerns/pulp_database_unit.rb  < 
  /var/tmp/2258876.patch.txt

  • Then restart the foreman services (can cause disruption on the Satellite, better if executed in a maintenance window), and try to delete the orphan task manually:

      foreman-maintain service restart
  foreman-rake katello:delete_orphaned_content RAILS_ENV=production >/dev/null 2>&1

  • Then monitor the task in the UI under "Monitor => Tasks" and type "orphan" in the search bar.

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 3.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 3.0-related Issues

Root Cause

  • Red Hat investigated this issue in bug report This content is not included.2258876 and delivered a fix in Satellite 6.15.0 through errata RHSA-2024:2010. If this issue still occurs in your environment after updating, open a support case in the Red Hat Customer Portal referring to this solution.

Diagnostic Steps

  • The following hammer command will export all tasks:

      foreman-rake foreman_tasks:export_tasks TASK_SEARCH='' 
  TASK_DAYS=3

  • Then using the command hammer task info --id= with the task ID, the item Action: Remove orphans can be monitored. Example:

      hammer task info --id=e1ec36a5-2468-4b94-8d6b-c5ce2bb5db60
  ID:          e1ec36a5-2468-4b94-8d6b-c5ce2bb5db60
  Action:      Remove orphans
  State:       stopped
  Result:      success
  Started at:  2024/03/12 11:10:40
  Ended at:    2024/03/13 23:41:41
  Duration:    131460.512781
  Owner:       foreman_admin
  Task errors:

  • Also the tasks can be monitored on the UI console under "Monitor => Tasks" and type "orphan" in the search bar.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.