Removing large numbers of Actions::Katello::Host tasks consuming database resources

Solution Unverified - Updated

Environment

Satellite 6.6 or older

Issue

  • Satellite 6 is performing poorly
  • Slow user interface actions
  • Slow operations in general
  • Tasks are not being processed in an expected timeframe
  • Disk space consumption by Postgres is high (/var/lib/pgsql)

If the Satellite 6 server has more than 10,000 tasks of type Actions::Katello::Host::* immediate removal as well as automatic cleanup is required to return the Satellite to optimal operations.

The majority are comprised of 4 types:

 Actions::Katello::Host::GenerateApplicability
 Actions::Katello::Host::UploadPackageProfile
 Actions::Katello::Host::UploadProfiles  
 Actions::Katello::Host::Update

You can check the count of stored Host tasks via a simple query:

#  echo "select count(label) from foreman_tasks_tasks where label like 'Actions::Katello::Host%';" | sudo -u postgres psql -d foreman
could not change directory to "/root"
 count 
-------
  120346
(1 row)

If this number is above 10,000 you should consider cleaning out your stopped tasks. These types of tasks often comprise over 90%+ of the tasks in Satellite's database if left unchecked. These tasks are small and should not be retained for any length of time as they generally don't hold any relevant data.

Resolution

For Satellite 6.7 or newer, the problematic Host tasks have been put aside of dynflow, hence the problem should not happen. The solution below is applicable to 6.6 or older.

There are 3 steps to cleaning out the old tasks:

1) Removing existing tasks:

Please execute this command from the Satellite shell to delete all uneeded Host tasks:

# foreman-rake foreman_tasks:cleanup TASK_SEARCH='label=Actions::Katello::Host::UploadPackageProfile or label=Actions::Katello::Host::Update or label=Actions::Katello::Host::GenerateApplicability or label=Actions::Katello::Host::UploadProfiles' AFTER='0d' STATES=stopped,paused VERBOSE=true

2) Setting policy to prune Host tasks with more frequency

Turn on auto-task cleanup:

# satellite-installer --foreman-plugin-tasks-automatic-cleanup true

3) Adjust policy for which tasks get cleaned up when automatic task cleanup runs. This policy will remove the top 4 task types that are more than 3 days old as well as remove ALL tasks stopped for more than 90 days. Note deleted tasks are archived in /var/lib/foreman/tasks-backup

Then edit: /etc/foreman/plugins/foreman-tasks.yaml and change the parameters to as follows under the cleanup section:

# Rules defined in this section by default don't operate
# on tasks specified in the actions section. This behavior
# can be overriden by setting the override_actions to true
   :rules:
     # Delete client API tasks after 3 days - These are often over 90% tasks in the database
     - :states: all
       :filter:  '(label = Actions::Katello::Host::Update or label = Actions::Katello::Host::GenerateApplicability or label = Actions::Katello::Host::UploadPackageProfile or label = Actions::Katello::Host::UploadProfiles)'
       :after: 3d
     # Delete everything (any action, any state) after 90 days
     - :states: all # Either list of state names or all
       :after: 90d
       :override_actions: true
SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.