After leapp upgrade to RHEL 8.6 and reboot, the satellite-installer execution fails to start postgresql service in Red Hat Satellite 6.11

Solution Verified - Updated 13 Jun 2024

Environment

Red Hat Satellite 6.11
Red Hat Enterprise Linux 8.6

Issue

After upgrading the OS from RHEL 7.9 to RHEL 8.6 using leapp and reboot of the server, the leapp_resume activity fails at the backend to complete the satellite-installer execution.

Jul 14 14:05:28 satellite611 leapp3[7612]: ==> Processing phase `FirstBoot`
Jul 14 14:05:28 satellite611 leapp3[7612]: ====> * satellite_upgrader
Jul 14 14:05:28 satellite611 leapp3[7612]:        Execute installer in the freshly booted system, to finalize Satellite configuration
Jul 14 14:07:44 satellite611 leapp3[8122]: 2022-07-14 14:07:44.163 ERROR    PID: 8122 leapp.workflow.FirstBoot.satellite_upgrader: Could not run the installer, please inspect the logs in /var/log/foreman-installer!

None of the services related to satellite were found to be properly running and the postgresql service is in a failed state.

   * postgresql.service - PostgreSQL database server
      Loaded: loaded (/usr/lib/systemd/system/postgresql.service; disabled; vendor preset: disabled)
     Drop-In: /etc/systemd/system/postgresql.service.d
           `-postgresql.conf
      Active: failed (Result: exit-code) since Thu 2022-07-14 14:06:57 CDT; 14min ago

   Jul 14 14:06:57 satellite611.example.com systemd[1]: Starting PostgreSQL database server...
   Jul 14 14:06:57 satellite611.example.com systemd[1]: postgresql.service: Control process exited, code=exited status=1
   Jul 14 14:06:57 satellite611.example.com systemd[1]: postgresql.service: Failed with result 'exit-code'.
   Jul 14 14:06:57 satellite611.example.com systemd[1]: Failed to start PostgreSQL database server.

Resolution

This issue has been reported to the Red Hat Satellite Engineering team via This content is not included.Bugzilla 2104730 and is being actively worked on.

Meanwhile to solve this problem manually, Perform the following steps:

Check and fix the ownership of /var/lib/pgsql/data/ and the files and sub-directories inside the same.

# ls -l /var/lib/pgsql/data/
# chown -R postgres:postgres /var/lib/pgsql/data
# ls -l /var/lib/pgsql/data/

Manually execute the final installer execution that the leapp_resume service failed to complete.

# satellite-installer -S satellite --verbose
# satellite-maintain service restart
# sleep 20 && hammer ping

Reach out to This content is not included.Red Hat Technical Support in case of any further concerns or assistance required.

For more KB articles/solutions related to Red Hat Satellite 6.x LEAPP Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x LEAPP Issues

Diagnostic Steps

The /var/log/foreman-installer/satellite.log shows the following errors about the postgresql service:

   2022-07-14 14:06:56 [DEBUG ] [configure] /Stage[main]/Postgresql::Server::Service/Anchor[postgresql::server::service::begin]: Evaluated in 0.00 seconds
   2022-07-14 14:06:56 [DEBUG ] [configure] /Service[postgresqld]: Starting to evaluate the resource (1258 of 2111)
   2022-07-14 14:06:56 [DEBUG ] [configure] Executing: 'systemctl status postgresql'
   2022-07-14 14:06:56 [DEBUG ] [configure] Executing: '/bin/systemctl is-enabled -- postgresql'
   2022-07-14 14:06:56 [DEBUG ] [configure] Executing: '/bin/systemctl show --property=NeedDaemonReload -- postgresql'
   2022-07-14 14:06:56 [DEBUG ] [configure] Executing: '/bin/systemctl unmask -- postgresql'
   2022-07-14 14:06:57 [DEBUG ] [configure] Executing: '/bin/systemctl start -- postgresql'
   2022-07-14 14:06:57 [DEBUG ] [configure] Running journalctl command to get logs for systemd start failure: journalctl -n 50 --since '5 minutes ago' -u postgresql --no-p
ager
   2022-07-14 14:06:57 [DEBUG ] [configure] Executing: 'journalctl -n 50 --since '5 minutes ago' -u postgresql --no-pager'
   2022-07-14 14:06:57 [ERROR ] [configure] Systemd start for postgresql failed!
   2022-07-14 14:06:57 [ERROR ] [configure] journalctl log for postgresql:
   2022-07-14 14:06:57 [ERROR ] [configure] -- Logs begin at Thu 2022-07-14 14:04:18 CDT, end at Thu 2022-07-14 14:06:57 CDT. --
   2022-07-14 14:06:57 [ERROR ] [configure] Jul 14 14:06:57 satellite611.example.com systemd[1]: Starting PostgreSQL database server...
   2022-07-14 14:06:57 [ERROR ] [configure] Jul 14 14:06:57 satellite611.example.com systemd[1]: postgresql.service: Control process exited, code=exited status=1
   2022-07-14 14:06:57 [ERROR ] [configure] Jul 14 14:06:57 satellite611.example.com systemd[1]: postgresql.service: Failed with result 'exit-code'.
   2022-07-14 14:06:57 [ERROR ] [configure] Jul 14 14:06:57 satellite611.example.com systemd[1]: Failed to start PostgreSQL database server.
   2022-07-14 14:06:57 [ERROR ] [configure] /Stage[main]/Postgresql::Server::Service/Service[postgresqld]/ensure: change from 'stopped' to 'running' failed: Systemd start for postgresql failed!
   2022-07-14 14:06:57 [ERROR ] [configure] journalctl log for postgresql:
   2022-07-14 14:06:57 [ERROR ] [configure] -- Logs begin at Thu 2022-07-14 14:04:18 CDT, end at Thu 2022-07-14 14:06:57 CDT. --
   2022-07-14 14:06:57 [ERROR ] [configure] Jul 14 14:06:57 satellite611.example.com systemd[1]: Starting PostgreSQL database server...
   2022-07-14 14:06:57 [ERROR ] [configure] Jul 14 14:06:57 satellite611.example.com systemd[1]: postgresql.service: Control process exited, code=exited status=1
   2022-07-14 14:06:57 [ERROR ] [configure] Jul 14 14:06:57 satellite611.example.com systemd[1]: postgresql.service: Failed with result 'exit-code'.
   2022-07-14 14:06:57 [ERROR ] [configure] Jul 14 14:06:57 satellite611.example.com systemd[1]: Failed to start PostgreSQL database server.
   2022-07-14 14:06:57 [DEBUG ] [configure] Executing: 'systemctl status postgresql'
   2022-07-14 14:06:57 [DEBUG ] [configure] /Service[postgresqld]: Skipping restart; service is not running

Around the same time, the following details were logged inside the /var/log/messages file about the error indicating an issue with permissions or ownership.

Jul 14 14:06:57 satellite611 systemd[1]: Starting PostgreSQL database server...
Jul 14 14:06:57 satellite611 postgresql-check-db-dir[9321]: cat: /var/lib/pgsql/data/PG_VERSION: Permission denied
Jul 14 14:06:57 satellite611 postgresql-check-db-dir[9320]: An old version '' of the database format was found.
Jul 14 14:06:57 satellite611 postgresql-check-db-dir[9320]: You need to dump and reload before using PostgreSQL 12.11.
Jul 14 14:06:57 satellite611 postgresql-check-db-dir[9320]: See /usr/share/doc/postgresql/README.rpm-dist for more information.
Jul 14 14:06:57 satellite611 systemd[1]: postgresql.service: Control process exited, code=exited status=1
Jul 14 14:06:57 satellite611 systemd[1]: postgresql.service: Failed with result 'exit-code'.
Jul 14 14:06:57 satellite611 systemd[1]: Failed to start PostgreSQL database server.

SBR

SysMgmt

Product(s)

Red Hat Satellite

Components

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.