Unable to plan Ansible tasks with large inventories in Satellite

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.9 to 6.14

Issue

  • Planning an Ansible task with large inventories fails
  • Only a multiple of a hundred sub-tasks are successfully scheduled (100, 200, ...) and the rest end in N/A status
  • undefined method `wait' for nil:NilClass (NoMethodError) error message is found in /var/log/foreman/production.log
  • The following conditions are met:
    • A large inventory is used
    • Multiple dynflow worker processes are used
    • Concurrency control is implemented

Resolution

  • Red Hat investigated this issue in bug report This content is not included.Bugzilla 2068527 and delivered a fix in Satellite 6.15 through errata RHSA-2024:2010.

  • If this issue still occurs in your environment after updating, This content is not included.open a support case in the Red Hat Customer Portal referring to this solution.

  • If it's not possible to upgrade to the release version containing the fix, to improve the behavior:

    • Do not have multiple dynflow workers consuming the remote_execution queue

    • Configure a new system unit that will consume the remote_execution queue only for remote execution jobs

      • Red Hat Satellite 6.9:

        # cp -p /etc/foreman/dynflow/worker.yml /etc/foreman/dynflow/rex.yml
# sed -i.$(date +%Y%m%d) '/^  - remote_execution$/d' /etc/foreman/dynflow/worker.yml
# sed -i.$(date +%Y%m%d) 's#\(^:concurrency: \)\(.*$\)#\115#' /etc/foreman/dynflow/rex.yml
# sed -i '/^  - default$/d' /etc/foreman/dynflow/rex.yml
# systemctl enable --now dynflow-sidekiq@rex
# systemctl restart dynflow-sidekiq@*

      • Red Hat Satellite 6.10 to 6.14:

               # cat >/etc/foreman/dynflow/rex.yml << EOF
       /---
       :concurrency: 5
       :queues:
       - - remote_execution
          - 1
       EOF

       # sed -i 's/^\///' /etc/foreman/dynflow/rex.yml
       # sed -i.orig "/^- - remote_execution/,+1d" /etc/foreman/dynflow/worker-[[:digit:]].yml

       # systemctl restart dynflow-sidekiq@worker-*
       # systemctl enable --now dynflow-sidekiq@rex

    • Once done, retry the remote_execution jobs and observe their performance and behavior.

For more KB articles/solutions related to Red Hat Satellite 6.x Remote Execution Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Remote Execution Issues

Root Cause

  • A component of dynflow which handles concurrency control (concurrency level and time span fields in Satellite's job form) makes a wrong assumption and breaks as soon as there are multiple dynflow-sidekiq@ instances consuming a remote_execution queue

Diagnostic Steps

  • There are several dynflow-sidekiq@ instances consuming a remote_execution queue:
$ ls -al /etc/foreman/dynflow/
total 20
drwxrwxrwx. 2 root yank 4096 Feb 16 14:05 .
drwxrwxrwx. 4 root yank 4096 Nov 30  2021 ..
-rw-rw-rw-. 1 root foreman   51 Apr  7  2021 orchestrator.yml
lrwxrwxrwx. 1 root foreman   10 Feb 16 14:05 worker-1.yml -> worker.yml
lrwxrwxrwx. 1 root foreman   10 Feb 16 14:05 worker-2.yml -> worker.yml
-rw-rw-rw-. 1 root foreman   43 Feb 16 14:05 worker-hosts-queue.yml
-rw-rw-rw-. 1 root foreman   60 Feb 16 14:05 worker.yml
$ cat /etc/foreman/dynflow/worker.yml 
:concurrency: 10
:queues:
  - default
  - remote_execution
  • Look for a similar backtrace to this one in /var/log/foreman/production.log:
<TIMESTAMP> [E|bac|] undefined method `wait' for nil:NilClass (NoMethodError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:71:in `block (2 levels) in handle_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:69:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:69:in `block in handle_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:68:in `map'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:68:in `handle_plans'
 | [ concurrent-ruby ]
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:21:in `handle_plans!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_sub_plans.rb:92:in `trigger_with_concurrency_control'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_sub_plans.rb:82:in `trigger'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.2.3.1/app/lib/actions/remote_execution/run_hosts_job.rb:45:in `block in create_sub_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/delegation.rb:87:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.2.3.1/app/lib/actions/remote_execution/run_hosts_job.rb:40:in `map'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.2.3.1/app/lib/actions/remote_execution/run_hosts_job.rb:40:in `create_sub_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_sub_plans.rb:41:in `spawn_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_bulk_sub_plans.rb:71:in `spawn_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_bulk_sub_plans.rb:18:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_polling_sub_plans.rb:13:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.2.3.1/app/lib/actions/remote_execution/run_hosts_job.rb:97:in `run'
..
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/proxy_batch_triggering.rb:9:in `run'
..
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/watch_delegated_proxy_sub_tasks.rb:17:in `run'
..
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
...
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in `run_user_code'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 | [ sidekiq ]
 | [ concurrent-ruby ]
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.