The postgresql service fails to start while restoring Red Hat Satellite backup with error Service[postgresqld]/ensure: change from 'stopped' to 'running' failed: Systemd start for postgresql failed!

Environment

• Red Hat Satellite 6.11
• Red Hat Enterprise Linux 8

Issue

• The postgresql service fails to start while restoring Red Hat Satellite 6.11 server on RHEL-8 with below error.

     # systemctl status postgresql
     ● postgresql.service - PostgreSQL database server
        Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)
       Drop-In: /etc/systemd/system/postgresql.service.d
                └─postgresql.conf
        Active: failed (Result: exit-code) since Mon 2022-08-01 00:00:00 PDT; 1min 1s ago
      Main PID: 123456 (code=exited, status=1/FAILURE)
  
     Aug 01 00:00:00 satellite systemd[1]: Starting PostgreSQL database server...
     Aug 01 00:00:00 satellite systemd[1]: postgresql.service: Main process exited, code=exited, status=1/FAILURE
     Aug 01 00:00:00 satellite systemd[1]: postgresql.service: Failed with result 'exit-code'.
     Aug 01 00:00:00 satellite systemd[1]: Failed to start PostgreSQL database server.
  

Resolution

• Remove the /var/lib/pgsql/data directory from Satellite machine on which backup is being restored.

  # rm -fr /var/lib/pgsql/data
  

• Increase the /var/lib/pgsql filesystem, so it does no get full while pgsql data is extracted during restore progress.

• Ensure /var/lib/pgsql/data/ directory has correct permissions and ownership.

  # ls -ld  /var/lib/pgsql/data/ 
  drwx------. 20 postgres postgres 4096 Aug 25 00:00 /var/lib/pgsql/data/
  

Root Cause

• The /var/lib/pgsql filesystem is getting full during the restore process.

Diagnostic Steps

• Below error is observed on /var/log/foreman-installer/satellite.log while running # satellite-installer command.

  [ERROR ] [configure] Unable to connect to PostgreSQL server! (:5432)
  [ERROR ] [configure] /Stage[main]/Postgresql::Server::Service/Postgresql_conn_validator[validate_service_is_running]/ensure: change from 'absent' to 'present' failed: Unable to connect to PostgreSQL server! (:5432)
  [ERROR ] [configure] /Stage[main]/Candlepin::Database::Postgresql/Postgresql::Server::Db[candlepin]/Postgresql::Server::Role[candlepin]/Postgresql_psql[CREATE ROLE candlepin ENCRYPTED PASSWORD ****]: Could not evaluate: Error evaluating 'unless' clause, returned pid 14678 exit 2: 'psql: error: FATAL:  cache lookup failed for relation 1247
  [ERROR ] [configure] '
  

• Extract the tarball of the pgsql data on the Satellite backup and ensure that the size of the partition dedicated to /var/lib/pgsql partition is greater than the size of the pgsql data directory from backup.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.