Checking Red Hat Satellite 6 service status fails with error Unit name pulpcore-worker@.service is neither a valid invocation ID nor unit name is encountered

Solution Verified - Updated

Environment

  • Red Hat Satellite 6

Issue

  • Below error is encountered while checking the Red Hat Satellite 6 service status:

    Failed to get load state of pulpcore-worker@.service: Unit name pulpcore-worker@.service is neither a valid invocation ID nor unit name
    
  • The same error is encountered while trying to upgrade the Satellite server

       Check whether all services are running:                               [FAIL]
       Following services are not running: pulpcore-worker@.service
    
       Continue with step [Restart applicable services]?, [y(yes), n(no)] y
       Restart applicable services:
    
       Stopping the following service(s):
       pulpcore-worker@.service
       \ stopping pulpcore-worker@.service
       Failed to stop pulpcore-worker@.service: Unit name pulpcore-worker@.service is missing the instance name.
       See system logs and 'systemctl status pulpcore-worker@.service' for details.
       Failed to get load state of pulpcore-worker@.service: Unit name pulpcore-worker@.service is neither a valid invocation ID nor unit name.
       \ All services stopped
    

Resolution

  1. Follow the steps in the Diagnostic Steps section below.

  2. If the output from step (2) in the Diagnostic Steps section reports ony one pulpcore-worker@.service service as [FAIL] like this:

  # satellite-maintain service status -b
  Running Status Services

  Get status of applicable services:

  Displaying the following service(s):
  redis, postgresql, pulpcore-api, pulpcore-content, qdrouterd, qpidd, pulpcore-worker@.service, tomcat, dynflow- 
  sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker-1, foreman-proxy
  - displaying redis                                 [OK]
  - displaying postgresql                            [OK]
  - displaying pulpcore-api                          [OK]
  - displaying pulpcore-content                      [OK]
  - displaying qdrouterd                             [OK]
  - displaying qpidd                                 [OK]
  - displaying pulpcore-worker@.service              [FAIL]
  - displaying tomcat                                [OK]
  \ displaying dynflow-sidekiq@orchestrator          [OK]
  \ displaying foreman                               [OK]
  \ displaying httpd                                 [OK]
  \ displaying puppetserver                          [OK]
  \ displaying dynflow-sidekiq@worker-1              [OK]
  \ displaying foreman-proxy                         [OK]
  \ All services displayed                                              [FAIL]
  Some services are not running (pulpcore-worker@.service)

  Scenario [Status Services] failed.

and the output from step (3) in the Diagnostic Steps section displays only:

# ls -l /etc/systemd/system/multi-user.target.wants/pulpcore-worker*
lrwxrwxrwx. 1 root root 44 Oct  9 22:32 /etc/systemd/system/multi-user.target.wants/pulpcore-worker@.service -> /etc/systemd/system/pulpcore-worker@.service
lrwxrwxrwx. 1 root root 44 Sep  2  2021 /etc/systemd/system/multi-user.target.wants/pulp_workers.service -> /usr/lib/systemd/system/pulp_workers.service

then run the following commands:

  # for b in $(ls -1 /etc/systemd/system/multi-user.target.wants | grep ^pulpcore-worker); do echo - $b;unlink /etc/systemd/system/multi-user.target.wants/$b; done

  # satellite-installer
  1. Examine the output from the following command:
# satellite-maintain service status -b

For more KB articles/solutions related to Red Hat Satellite 6.x Installation/Upgrade/Update Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Installation/Upgrade/Update Issues.

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 3.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 3.0-related Issues

Root Cause

The satellite-maintain service status -b command was confused by wrong redundant symbolic link under the /etc/systemd/system/multi-user.target.wants/ directory:

lrwxrwxrwx. 1 root root 44 Oct  9 22:32 /etc/systemd/system/multi-user.target.wants/pulpcore-worker@.service -> /etc/systemd/system/pulpcore-worker@.service

Diagnostic Steps

  1. Check the pulpcore_worker_count in /etc/foreman-installer/scenarios.d/satellite-answers.yaml:
pulpcore_worker_count: n

where n is an integer.

Note: For capsule server check pulpcore_worker_count in /etc/foreman-installer/scenarios.d/capsule-answers.yaml

  1. Examine the output from the following command:
# satellite-maintain service status -b

There should be pulpcore-worker@1.service through pulpcore-worker@n.service services reporting [OK].

So, for instance, if pulpcore_worker_count in /etc/foreman-installer/scenarios.d/satellite-answers.yaml was 8, the output should be like this:

  Running Status Services

  Get status of applicable services: 

  Displaying the following service(s):
  redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker@1.service, pulpcore-worker@2.service, 
  pulpcore-worker@3.service, pulpcore-worker@4.service, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, 
  puppetserver, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
  / displaying redis                                 [OK]                         
  / displaying postgresql                            [OK]                         
  / displaying pulpcore-api                          [OK]                         
  / displaying pulpcore-content                      [OK]                         
  / displaying pulpcore-worker@1.service             [OK]                         
  / displaying pulpcore-worker@2.service             [OK]                         
  / displaying pulpcore-worker@3.service             [OK]                         
  / displaying pulpcore-worker@4.service             [OK]                         
  / displaying pulpcore-worker@5.service             [OK]                         
  / displaying pulpcore-worker@6.service             [OK]                         
  / displaying pulpcore-worker@7.service             [OK]                         
  / displaying pulpcore-worker@8.service             [OK]                         
  / displaying tomcat                                [OK]                         
  / displaying dynflow-sidekiq@orchestrator          [OK]                         
  / displaying foreman                               [OK]                         
  / displaying httpd                                 [OK]                         
  / displaying puppetserver                          [OK]                         
  - displaying dynflow-sidekiq@worker-1              [OK]                         
  - displaying dynflow-sidekiq@worker-hosts-queue-1  [OK]                         
  - displaying foreman-proxy                         [OK]                         
  - All services are running                                            [OK]      
  1. Examine the output of the following command:
# ls -l /etc/systemd/system/multi-user.target.wants/pulpcore-worker*

The output from the above command should display n lines like this:

lrwxrwxrwx. 1 root root 44 May  4  2023 /etc/systemd/system/multi-user.target.wants/pulpcore-worker@1.service -> /etc/systemd/system/pulpcore-worker@.service
lrwxrwxrwx. 1 root root 44 May  4  2023 /etc/systemd/system/multi-user.target.wants/pulpcore-worker@2.service -> /etc/systemd/system/pulpcore-worker@.service
.
.
lrwxrwxrwx. 1 root root 44 May  4  2023 /etc/systemd/system/multi-user.target.wants/pulpcore-worker@n.service -> /etc/systemd/system/pulpcore-worker@.service

So, for instance, if pulpcore_worker_count in /etc/foreman-installer/scenarios.d/satellite-answers.yaml was 8, the output should be like this:

lrwxrwxrwx. 1 root root 44 May  4  2023 /etc/systemd/system/multi-user.target.wants/pulpcore-worker@1.service -> /etc/systemd/system/pulpcore-worker@.service
lrwxrwxrwx. 1 root root 44 May  4  2023 /etc/systemd/system/multi-user.target.wants/pulpcore-worker@2.service -> /etc/systemd/system/pulpcore-worker@.service
lrwxrwxrwx. 1 root root 44 May  4  2023 /etc/systemd/system/multi-user.target.wants/pulpcore-worker@3.service -> /etc/systemd/system/pulpcore-worker@.service
lrwxrwxrwx. 1 root root 44 May  4  2023 /etc/systemd/system/multi-user.target.wants/pulpcore-worker@4.service -> /etc/systemd/system/pulpcore-worker@.service
lrwxrwxrwx. 1 root root 44 May  4  2023 /etc/systemd/system/multi-user.target.wants/pulpcore-worker@5.service -> /etc/systemd/system/pulpcore-worker@.service
lrwxrwxrwx. 1 root root 44 May  4  2023 /etc/systemd/system/multi-user.target.wants/pulpcore-worker@6.service -> /etc/systemd/system/pulpcore-worker@.service
lrwxrwxrwx. 1 root root 44 May  4  2023 /etc/systemd/system/multi-user.target.wants/pulpcore-worker@7.service -> /etc/systemd/system/pulpcore-worker@.service
lrwxrwxrwx. 1 root root 44 May  4  2023 /etc/systemd/system/multi-user.target.wants/pulpcore-worker@8.service -> /etc/systemd/system/pulpcore-worker@.service
  1. Below error can be observed in /var/log/foreman-maintain/foreman-maintain.log on Red Hat Satellite.
  Failed to get load state of pulpcore-worker@.service: Unit name pulpcore-worker@.service is neither a valid invocation ID nor unit name.
Failed to get properties: Unit name pulpcore-worker@.service is neither a valid invocation ID nor unit name.
SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.