How to add sidekiq workers in Satellite6?

Environment

• Red Hat Satellite 6.8 or newer

Issue

• Having performance / scalability problems with foreman tasking system (sidekiq since 6.8) whose workers are unable to process all work in time.
• How to increase the number of sidekiq workers to scale foreman tasking system?

Resolution

Two options are available:

• Increase number of threads by a dynflow worker (worker's concurrency). This has limited impact for values >5 due to ruby implementation of threads' concurrency.
• Increase number of workers, which is recommended.

Below examples to increase 1 worker to 3 while remaining 5 threads/concurrency of each.

Satellite 6.10 or newer

# satellite-installer --foreman-dynflow-worker-instances 3    # optionally, add --foreman-dynflow-worker-concurrency 10 to add more threads to each worker (default is 5)

Then check if there are three worker services:

# systemctl -a | grep dynflow-sidekiq@worker-[0-9]
  dynflow-sidekiq@worker-1.service                                                                               loaded    active   running   Foreman jobs daemon - worker-1 on sidekiq
  dynflow-sidekiq@worker-2.service                                                                               loaded    active   running   Foreman jobs daemon - worker-2 on sidekiq
  dynflow-sidekiq@worker-3.service                                                                               loaded    active   running   Foreman jobs daemon - worker-3 on sidekiq

Satellite 6.8 or 6.9

Review concurrency in worker's config:

# cat /etc/foreman/dynflow/worker.yml 
---
:concurrency: 5
:queues:
- - default
  - 1
- - remote_execution
  - 1
#

(the precise syntax varies over Satellite versions)

Now, add two more services:

# cd /etc/foreman/dynflow
# ln -s worker.yml worker-1.yml
# ln -s worker.yml worker-2.yml
# systemctl enable --now dynflow-sidekiq@worker-{1,2}

Check the services were enabled successfully:

# systemctl status dynflow-sidekiq@worker-1
● dynflow-sidekiq@worker-1.service - Foreman jobs daemon - worker-1 on sidekiq
   Loaded: loaded (/usr/lib/systemd/system/dynflow-sidekiq@.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2021-08-30 15:13:51 CEST; 1 day 1h ago
     Docs: https://theforeman.org
 Main PID: 15434 (sidekiq)
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/dynflow-sidekiq@worker-1.service
           └─15434 sidekiq 5.2.7  [5 of 5 busy]

Aug 31 16:55:30 my-sat007.example.com dynflow-sidekiq@worker-1[15434]: /usr/share/foreman/lib/proxy_api/resource.rb:72: warning: URI.escape is obsolete
..
# systemctl status dynflow-sidekiq@worker-2
● dynflow-sidekiq@worker-2.service - Foreman jobs daemon - worker-2 on sidekiq
   Loaded: loaded (/usr/lib/systemd/system/dynflow-sidekiq@.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-08-31 16:45:50 CEST; 15min ago
     Docs: https://theforeman.org
 Main PID: 19499 (sidekiq)
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/dynflow-sidekiq@worker-2.service
           └─19499 sidekiq 5.2.7  [5 of 5 busy]

Aug 31 16:55:34 my-sat007.example.com dynflow-sidekiq@worker-2[19499]: /usr/share/foreman/lib/proxy_api/resource.rb:72: warning: URI.escape is obsolete
..
#

For more KB articles/solutions related to Red Hat Satellite 6.x Satellite Tasking System Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Satellite Tasking System Issues.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.