The httpd service failed to start at the end of the upgrade to Red Hat Satellite 6.10

Solution Verified - Updated 26 Apr 2022

Environment

Red Hat Satellite 6.10

Issue

At the end of the upgrade to Red Hat Satellite 6.10, The httpd service fails to start as it cannot load the mod_passenger.so library.

Jan  4 21:34:18 satellite610 systemd: Starting The Apache HTTP Server...
Jan  4 21:34:18 satellite610 httpd: httpd: Syntax error on line 40 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf.modules.d/zpassenger.load: Cannot load modules/mod_passenger.so into server: /etc/httpd/modules/mod_passenger.so: cannot open shared object file: No such file or directory
Jan  4 21:34:18 satellite610 systemd: httpd.service: main process exited, code=exited, status=1/FAILURE
Jan  4 21:34:18 satellite610 systemd: Failed to start The Apache HTTP Server.
Jan  4 21:34:18 satellite610 systemd: Unit httpd.service entered failed state.

Even if that has been worked around to complete the upgrade, the httpd service fails to start due to missing mod_wsgi.so library after the execution of satellite-maintain content remove-pulp2.

Jan  4 22:08:21 satellite610 httpd: httpd: Syntax error on line 40 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf.modules.d/wsgi.load: Cannot load modules/mod_wsgi.so into server: /etc/httpd/modules/mod_wsgi.so: cannot open shared object file: No such file or directory
Jan  4 22:08:21 satellite610 systemd: httpd.service: main process exited, code=exited, status=1/FAILURE
Jan  4 22:08:21 satellite610 systemd: Failed to start The Apache HTTP Server.
Jan  4 22:08:21 satellite610 systemd: Unit httpd.service entered failed state.
Jan  4 22:08:21 satellite610 systemd: httpd.service failed.

Resolution

This concern has been highlighted to the Red Hat Engineering team via Bugzilla This content is not included.2038577.
Take a tar of /etc/httpd and /usr/lib64/httpd/modules directories and keep that tar file in a safe location and proceed with the steps below to fix the httpd.service issue.

Solution 1 :

Verify if an entry of apache::purge_configs: false is present in the /etc/foreman-installer/custom-hiera.yaml file of Satellite server.
If Yes, then
- Edit the file /etc/foreman-installer/custom-hiera.yaml and comment out the apache::purge_configs: false entry.
- Retry the upgrade.

Solution 2 : Manual Approach

Check the status of httpd.service to confirm the reason of failure i.e. whether it is related to Syntax error on line 1 of /etc/httpd/conf.modules.d/zpassenger.load or not.
```
# systemctl status httpd -l
```

If Yes, then

Edit the file /etc/httpd/conf.modules.d/zpassenger.load and comment out the LoadModule line in that file.
Start the httpd.service manually and observe if that is getting started or not.
```
# systemctl start httpd
```
Re-run the upgrade and confirm that It's getting completed successfully.
Once the upgrade is successful, proceed with pulp2 data cleanup and restart of satellite services.
```
# satellite-maintain content remove-pulp2
# satellite-maintain service restart
```
- If the remove-pulp2 command fails with some Device or resource busy error, please refer to this article.

It is expected that the httpd.service will fail to start again but due to mod_wsgi.so library as the related This content is not included.package was removed as a part of remove-pulp2 action.

Proceed further with the following steps to get back httpd.service back up and running properly.

        # satellite-maintain service stop

        # rpm -ev --nodeps mod_wsgi mod_passenger

        # If any of these files are present, please delete the files.

          Under '/etc/httpd/conf.modules.d':
            10-wsgi.conf
            passenger.conf
            passenger_extra.conf
            wsgi.conf
            wsgi.load
            zpassenger.load

          Under '/etc/httpd/conf.d':
            01-status.conf
            03-crane.conf
            05-foreman_proxy_content.conf
            10-pulp.conf
            pulp_iso.conf

        # systemctl start httpd

        # satellite-maintain service restart

Please reach out to the This content is not included.Red Hat Technical Support in case of any further concerns or assistance required.

For more KB articles/solutions related to Red Hat Satellite 6.x Installation/Upgrade/Update Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Installation/Upgrade/Update Issues.

Root Cause

This issue has only been observed in certain scenarios where the satellite is being upgraded since the 6.6 or 6.7 version and even after it was upgraded to 6.9 , some older apache configurations and modules persisted on the Satellite server but they should not be present.
The older config files will only be retained if an entry of apache::purge_configs: false is present in the /etc/foreman-installer/custom-hiera.yaml file of Satellite server.

SBR

SysMgmt

Product(s)

Red Hat Satellite

Components

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.