ListenOnCandlepinEvents task is gone, `dynflowd` service restart does not help

Environment

Red Hat Satellite 6.3 or 6.4

Issue

• no running task with label Actions::Candlepin::ListenOnCandlepinEvents is present
• no running task with label Actions::Katello::EventQueue::Monitor is present
• candlepin is optionally in Suspended mode (or will likely enter soon, as katello_event_queue gets filled)
• restarting service dynflowd (originally named as foreman-tasks) does not kick off a new instance of these tasks

Resolution

Run below workaround procedure to clean orphaned tasks and remove pending locks:

/sbin/service foreman-tasks stop
/sbin/foreman-rake foreman_tasks:cleanup:run TASK_SEARCH='label=Actions::Candlepin::ListenOnCandlepinEvents'   STATES="running,paused"
/sbin/foreman-rake foreman_tasks:cleanup:run TASK_SEARCH='label=Actions::Katello::EventQueue::Monitor'   STATES="running,paused"

#Delete the action locks
/bin/echo "delete from dynflow_coordinator_records where class = 'Dynflow::Coordinator::SingletonActionLock';"  | /bin/sudo -u postgres psql -d foreman

#start foreman-tasks
/bin/systemctl start foreman-tasks

For a permanent solution, wait until bugs This content is not included.bz1651367 and This content is not included.bz1642369 are fixed.

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

Diagnostic Steps

If below postgres query:

sudo su - postgres -c "psql -d foreman -c 'select label,count(label),state,result from foreman_tasks_tasks where state <> '\''stopped'\'' group by label,state,result ORDER BY label;'"

does not have a line:

 Actions::Candlepin::ListenOnCandlepinEvents |     1 | running   | pending

then this problem is probably hit. Especially if dynflowd service restart does not help.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.