Some stopped Satellite tasks still acquire locks, what prevents new tasks to run

Environment

Red Hat Satellite 6.16+

Issue

Some task (typically CV publish) got stuck or hit an error, so we decided to stop it (not cancel but stop it).

Despite the task is stopped, a new task operating over the same object (like a new CV publish) fails with "Required lock is already taken" error, pointing to the stopped task.

Resolution

As a workaround, halt the dynflow task that was not properly stopped on background. Use proper task ID (the UUID from the URI).

echo 'ForemanTasks::Task::DynflowTask.where(id: ["3aaa10b7-5521-4162-b279-328cc248c262"]).each(&:halt)' | foreman-rake console

As a final solution, wait until This content is not included.the underlying bug is fixed.

For more KB articles/solutions related to Red Hat Satellite 6.x Satellite Tasking System Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Satellite Tasking System Issues.

Root Cause

API endpoint to stop a task does stop foreman task but not the spawned dynflow task.

Since the dynflow task holds the locks, this prevents new tasks to work over the same objects.

Diagnostic Steps

A task fails with error

Required lock is already taken by other running tasks. Please inspect their state, fix their errors and resume them. Conflicts with tasks: - https://my.satellite.fqdn/foreman_tasks/tasks/3aaa10b7-5521-4162-b279-328cc248c262

Clicking on the task shows a stopped task (but with pending or error result, and not 100% complete).

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.