How to perform task cleanup in Red Hat Satellite 6?

Solution Verified - Updated

Environment

  • Red Hat Satellite 6

Issue

  • While using Red Hat Satellite, a large number of tasks gets generated over time, which leads to big postgres database size and performance degradation. How to resolve this?

Resolution

  • NOTE: The most common tasks needing to be cleaned are of type Actions::Katello::Host tasks and specifically should be cleaned with the solution described here: Removing large numbers of Actions::Katello::Host tasks consuming database resources

  • The task can be cleaned up from the database using foreman-rake . It's recommended to use NOOP=true option that will list how many tasks would get cleaned and what exactly the search query would be.
    Example: foreman-rake foreman_tasks:cleanup NOOP=true

  • To clean all the tasks older than 30 days with verbosity and non-execution the below command can be used:

      foreman-rake foreman_tasks:cleanup TASK_SEARCH='label ~ *' AFTER='30d' VERBOSE=true NOOP=true
  # [noop] deleting all tasks matching filter label ~ * AND started_at < "2016-10-08 16:56:17" AND state = stopped
  # [noop] 14515 tasks would be deleted

  • With NOOP option enabled, no task is deleted. After tuning the search query to select all the unneeded tasks, the command can be executed without the NOOP option.

  • To get the list of tasks which are not in stopped state and could be running, paused or scheduled state, execute the following command:

       # su - postgres -c "psql -d foreman -c 'select label,count(label),state,result from foreman_tasks_tasks where state <> '\''stopped'\'' group by label,state,result ORDER BY label;'"

  • To clean tasks with specific label and state the below command can be used:

      foreman-rake foreman_tasks:cleanup TASK_SEARCH='label = <label name>' STATES='<state>' VERBOSE=true

  Example:

  foreman-rake foreman_tasks:cleanup TASK_SEARCH='label = Actions::Katello::Repository::Sync' STATES='running' VERBOSE=true

  • When specifying multiple states of a task run below command:

      foreman-rake foreman_tasks:cleanup TASK_SEARCH='label = Actions::Katello::Repository::Sync' STATES='running,pending,stopped' VERBOSE=true

  • Please note commonly mistaken syntax for filtering tasks with a certain result:

      foreman-rake foreman_tasks:cleanup TASK_SEARCH='label == a_label' RESULT='<result>' STATES='<state>'

    This is incorrect - there is no parameter RESULT. In order to filter per result, please use TASK_SEARCH=' result = <result>'

  • To clear any tasks that are in a stopped state but the result is pending:

      foreman-rake foreman_tasks:cleanup TASK_SEARCH='result == pending' STATES='stopped' VERBOSE=true

  • The tasks cleanup script accepts the same search syntax that is available in Monitor -> Tasks Web UI. A cleanup script that cleans all the virt-who tasks older than an hour is as below:

      foreman-rake foreman_tasks:cleanup TASK_SEARCH='label = "Actions::Katello::Host::Hypervisors"' AFTER='5h' VERBOSE=true

  • By default when task cleanup happens, The selected tasks are backed up before deleting them. This can consume a good amount of disk space over some time. To perform the task-cleanup without taking the backup of the selected tasks, refer to the following solution 3558991.

  • By default, only stopped tasks are cleaned, to avoid accidental deletion of tasks, preferred to resume tasks to avoid potential data corruption.

  • To enable automatic task cleanup in Red Hat Satellite 6 refer to the solution 3374401.

  • To list all the options available for the command, execute the following command:

      foreman-rake -D foreman_tasks:cleanup

Notice
The deletion of tasks itself can still keep some reserved disk space in the database. To finish the cleanup process, one is advised to also perform vacuuming of the database:

satellite-maintain service stop
systemctl start postgresql
su - postgres -c 'vacuumdb -f -d foreman'
satellite-maintain service start

The full vacuuming needs to be executed with exclusive access to the database, therefore stopping other Satellite services is required meanwhile. It is possible to call plain vacuum (without the -f / --full option), without the necessity to stop Satellite services. However, extra space is not returned to the operating system (in most cases); it's just kept available for reuse within the same table.

For more KB articles/solutions related to Red Hat Satellite 6.x Satellite Tasking System Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Satellite Tasking System Issues

Diagnostic Steps

Check the size of the PostgreSQL database.

  • For Satellite on RHEL8+ :

    # du -sh /var/lib/pgsql/
121.9G    /var/lib/pgsql/

  • For Satellite on RHEL7 :

    # du -sh /var/opt/rh/rh-postgresql12/lib/pgsql/
121.9G    /var/opt/rh/rh-postgresql12/lib/pgsql/

To identify the largest tasks, this command can be used:

su - postgres -c 'psql -d foreman -c\
     "SELECT foreman_tasks_tasks.label,
      count(foreman_tasks_tasks.id) tasks_total,
      count(dynflow_actions.id) actions_total
      FROM dynflow_actions
      LEFT JOIN foreman_tasks_tasks
      ON (foreman_tasks_tasks.external_id = dynflow_actions.execution_plan_uuid::varchar)
      GROUP BY foreman_tasks_tasks.label ORDER BY actions_total DESC LIMIT 30"'
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.