Red Hat Satellite upgrade to 6.16 failed and lost the session of the server due to discrepancy in postgresql version.

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.16

Issue

  • satellite-maintain upgrade run gets stuck in the following step:

      2025-08-06 05:01:09 [DEBUG ] [root] Warning: /etc/foreman-installer/custom-hiera.yaml: file does not contain a valid yaml hash

  2025-08-06 05:01:09 [INFO  ] [root] ... finished loading default values from puppet modules.
  2025-08-06 05:01:09 [INFO  ] [pre_values] Executing hooks in group pre_values
  2025-08-06 05:01:09 [INFO  ] [pre_values] All hooks in group pre_values finished
  2025-08-06 05:01:09 [DEBUG ] [root] Running installer with args [[]]
  2025-08-06 05:01:09 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
  2025-08-06 05:01:09 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
  2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/dns.rb
  2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/encoding.rb
  2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/hostname.rb
  2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/ipv6
  2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/noexec
  2025-08-06 05:01:10 [INFO  ] [pre_validations] Executing hooks in group pre_validations
  2025-08-06 05:01:10 [DEBUG ] [pre_validations] Hook /usr/share/foreman-installer/hooks/pre_validations/01-reset_data.rb returned nil
  2025-08-06 05:01:10 [DEBUG ] [pre_validations] Hook /usr/share/foreman-installer/hooks/pre_validations/12-check_certs_tar.rb returned nil

  • Post an unsuccessful upgrade, the satellite-installer is stuck at below step : 

      # satellite-installer -S satellite
      2025-02-05 15:14:59 [NOTICE] [root] Loading installer configuration. This will take some time.
      2025-02-05 15:15:03 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
      2025-02-05 15:15:03 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. 
      See --full-help for definitions.

    And logs similar to: /var/log/foreman-installer/satellite.log

      
 2025-08-06 05:01:09 [INFO  ] [root] ... finished loading default values from puppet modules.
 2025-08-06 05:01:09 [INFO  ] [pre_values] Executing hooks in group pre_values
 2025-08-06 05:01:09 [INFO  ] [pre_values] All hooks in group pre_values finished
 2025-08-06 05:01:09 [DEBUG ] [root] Running installer with args [[]]
 2025-08-06 05:01:09 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
 2025-08-06 05:01:09 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
 2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/dns.rb
 2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/encoding.rb
 2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/hostname.rb
 2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/ipv6
 2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/noexec
 2025-08-06 05:01:10 [INFO  ] [pre_validations] Executing hooks in group pre_validations
 2025-08-06 05:01:10 [DEBUG ] [pre_validations] Hook /usr/share/foreman-installer/hooks/pre_validations/01-reset_data.rb returned nil
 2025-08-06 05:01:10 [DEBUG ] [pre_validations] Hook /usr/share/foreman-installer/hooks/pre_validations/12-check_certs_tar.rb returned nil

  • Multiple services in failed state : 

         # satellite-maintain service status -b
         Running Status Services
         ===================================================
         Get status of applicable services:

         Displaying the following service(s):
          redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker@1.service, pulpcore-worker@2.service, pulpcore-worker@3.service, pulpcore-worker@4.service, pulpcore-worker@5.service, pulpcore-worker@6.service, pulpcore-worker@7.service, pulpcore-worker@8.service, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
           / displaying redis                                 [OK]
          / displaying postgresql                            [FAIL]
         - displaying pulpcore-api                          [FAIL]
         - displaying pulpcore-content                      [OK]
         - displaying pulpcore-worker@1.service             [FAIL]
         - displaying pulpcore-worker@2.service             [OK]
         - displaying pulpcore-worker@3.service             [FAIL]
         - displaying pulpcore-worker@4.service             [FAIL]
         - displaying pulpcore-worker@5.service             [FAIL]
         - displaying pulpcore-worker@6.service             [FAIL]
         - displaying pulpcore-worker@7.service             [OK]
         - displaying pulpcore-worker@8.service             [FAIL]
         - displaying tomcat                                [OK]
         - displaying dynflow-sidekiq@orchestrator          [FAIL]
         - displaying foreman                               [FAIL]
         - displaying httpd                                 [OK]
         - displaying dynflow-sidekiq@worker-1              [FAIL]
         - displaying dynflow-sidekiq@worker-hosts-queue-1  [FAIL]
         - displaying foreman-proxy                         [OK]
         - All services displayed                                              [FAIL]
         Some services are not running (postgresql, pulpcore-api, pulpcore-worker@1.service, pulpcore-worker@3.service, pulpcore-worker@4.service, pulpcore-worker@5.service, pulpcore-worker@6.service, pulpcore-worker@8.service, dynflow-sidekiq@orchestrator, foreman, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1)
         --------------------------------------------------------------------------------
          Scenario [Status Services] failed.

Resolution

  • Follow the below steps :

  1. Install the package postgresql-upgrade package :

     # dnf install postgresql-upgrade --disableplugin=foreman-protector

  2. Remove the option data_directory from postgresql configuration :

     # sed -i.orig -e "s|data_directory = '/var/lib/pgsql/data'||g" /var/lib/pgsql/data/postgresql.conf

  3. Perform the postgresql upgrade :

     # postgresql-setup --upgrade

  4. Start the postgresql :

     # systemctl start postgresql

  5. Check the postgresql status :

     # systemctl status -l postgresql

  6. Perform the satellite-installer :

     # satellite-installer

  7. Check if the environment is in maintenance mode :

     # satellite-maintain maintenance-mode status

    If it is enabled, please stop :

     # satellite-maintain maintenance-mode stop

For more KB articles/solutions related to Red Hat Satellite 6.x Installation/Upgrade/Update Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Installation/Upgrade/Update Issues.

Root Cause

  • The Red Hat Satellite server upgrade run process has upgraded the postgresql package version to 13, but the postgresql data version is still on 12.

Diagnostic Steps

  • postgresql package version points to version 13 :

     # rpm -qa postgresql
   postgresql-13.18-1.module+el8.10.0+22549+cf2ec3d9.x86_64     Thu Feb 13 12:49:23 2025
   postgresql-contrib-13.11-2.module+el8.9.0+19185+b1e56954.x86_64 Thu Feb 13 12:49:23 2025
   postgresql-evr-0.0.2-3.el8sat.noarch                        Tue Feb  4 13:24:55 2025
   postgresql-server-13.11-2.module+el8.9.0+19185+b1e56954.x86_64 Thu Feb 13 12:49:24 2025

  • The postgresql data version points to 12 : 

     # cat /var/lib/pgsql/data/PG_VERSION 
   12
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.