The "hammer ping" command shows pulp and pulp_auth as failed even after upgrading to Red Hat Satellite 6.10

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.10

Issue

  • During the upgradation to Red Hat Satellite 6.10, even if the satellite-installer was executed successfully,

    • The post-upgradation steps performed by satellite-maintain detects an unexpected failure at the server-ping step :

      Running Checks after upgrading to Satellite 6.10
================================================================================
Clean old Kernel and initramfs files from tftp-boot:                  [OK]
\--------------------------------------------------------------------------------
Check number of fact names in database:                               [OK]
\--------------------------------------------------------------------------------
Check whether all services are running:                               [OK]
\--------------------------------------------------------------------------------
Check whether all services are running using the ping call:           [FAIL]
Some components are failing: pulp, pulp_auth

    • Manual execution of the hammer ping command shows something similar at the end of the output while all applicable components are showing ok status :

      # hammer ping
..
..
2 more service(s) failed, but not shown:
pulp, pulp_auth

Resolution

  • Considering that at this point, the UI of Red Hat Satellite 6.10 is accessible:

    • Click on Administer -> Organizations -> Mismatch Report -> Fix Mismatch
      then save\submit the changes.

    • Repeat the same steps for every Location from the Administer --> Locations page.

    • Finally, go to the Infrastructure --> Capsules page, Click open the name of Satellite server and click on the Refresh Features button.

  • Ensure that the 14th step from the upgrade guide has been completed successfully followed by a restart of all services specific to Red Hat Satellite 6.10.

    • In case, the command fails with Device or resource busy message, consider looking into this solution article.

  • Verify the health of the satellite server.

    # hammer ping
# satellite-maintain health check

 

For more KB articles/solutions related to Red Hat Satellite 6.x hammer Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x hammer-related Issues

Diagnostic Steps

  • Manual execution of the hammer ping command shows the following status.

       # hammer ping
   database:
       Status:          ok
       Server Response: Duration: 0ms
   katello_agent:
       Status:          ok
       message:         0 Processed, 0 Failed
       Server Response: Duration: 0ms
   candlepin:
       Status:          ok
       Server Response: Duration: 45ms
   candlepin_auth:
       Status:          ok
       Server Response: Duration: 45ms
   candlepin_events:
       Status:          ok
       message:         0 Processed, 0 Failed
       Server Response: Duration: 0ms
   katello_events:
       Status:          ok
       message:         0 Processed, 0 Failed
       Server Response: Duration: 0ms
   pulp3:
       Status:          ok
       Server Response: Duration: 294ms
   pulp3_content:
       Status:          ok
       Server Response: Duration: 58ms
   foreman_tasks:
       Status:          ok
       Server Response: Duration: 3ms

   2 more service(s) failed, but not shown:
   pulp, pulp_auth

  • The following details were captured in /var/log/foreman-maintain/foreman-maintain.log related to the failures. 

      I, [2021-12-10 10:19:51-0500 #7067]  INFO -- : --- Execution step 'Check whether all services are running using the ping call' [server-ping] started ---
  D, [2021-12-10 10:19:55-0500 #7067] DEBUG -- : Called /katello/api/ping
  D, [2021-12-10 10:19:55-0500 #7067] DEBUG -- : Response: 200,   {"status":"FAIL","services":{"candlepin":{"status":"ok","duration_ms":"35"},"candlepin_auth":{"status":"ok","duration_ms":"36"},"foreman_tasks":{"status":"ok","duration_ms":"8"},"katello_events":{"status":"ok","message":"0 Processed, 0 Failed","duration_ms":"1"},"candlepin_events":{"status":"ok","message":"0 Processed, 0 Failed","duration_ms":"0"},"pulp3":{"status":"ok","duration_ms":"61"},"pulp3_content":{"status":"ok","duration_ms":"57"},"katello_agent":{"status":"ok","message":"0 Processed, 0 Failed","duration_ms":"0"},"pulp":{"status":"FAIL","message":"404 Not Found"},"pulp_auth":{"status":"FAIL","message":"Skipped pulp_auth check after failed pulp check"}}}
  
  D, [2021-12-10 10:19:55-0500 #7067] DEBUG -- : Running command rpm -q 'katello' --queryformat="%{VERSION}" with stdin nil
  D, [2021-12-10 10:19:55-0500 #7067] DEBUG -- : output of the command:
4.1.1
  D, [2021-12-10 10:19:55-0500 #7067] DEBUG -- : Running command rpm -q 'katello-common' --queryformat="%{VERSION}" with stdin nil
  D, [2021-12-10 10:19:55-0500 #7067] DEBUG -- : output of the command:
4.1.1
  D, [2021-12-10 10:19:55-0500 #7067] DEBUG -- : Running command rpm -q 'katello-common' --queryformat="%{VERSION}" with stdin nil
  D, [2021-12-10 10:19:55-0500 #7067] DEBUG -- : output of the command:
4.1.1
  I, [2021-12-10 10:19:55-0500 #7067]  INFO -- : --- Execution step 'Check whether all services are running using the ping call' finished ---
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.