Repair Stale MongoDB lock file for Satellite 6 pulp_streamer Service

Issue

• After a Satellite server crash, The Pulp streamer service fails during startup. Evidence of the failure appears as output similar to the following in /var/log/messages:

    May 29 15:16:29 sat6 pulp: pulp.server.db.connection:CRITICAL: Database initialization failed: localhost:27017: [Errno 111] 
    Connection refused
    May 29 15:16:29 sat6 pulp: pulp.server.webservices.application:CRITICAL:************************************
    May 29 15:16:29 sat6 pulp: pulp.server.webservices.application:ERROR: (29239-58688) The Pulp server encountered an 
    unexpected failure during initialization
  

Resolution

• Run the below commands:

    # satellite-maintain service stop 
    # sudo -u mongodb mongod --dbpath /var/lib/mongodb --repair 
    # satellite-maintain service restart
  

• After Satellite services have been restarted, verify state of pulp_streamer.service

    # systemctl status pulp_streamer.service
  
    This should not display errors and output should be similar to: 
  
    ● pulp_streamer.service - The Pulp lazy content loading streamer
           Loaded: loaded (/usr/lib/systemd/system/pulp_streamer.service; enabled; vendor preset: disabled)
          Active: active (running) since Fri 2018-06-01 08:34:28 PDT; 6h ago
        Main PID: 1286 (pulp_streamer)
            CGroup: /system.slice/pulp_streamer.service
                    └─1286 /usr/bin/python /usr/bin/pulp_streamer --syslog --prefix=pulp_streamer --pidfile= python...
  

• In some cases incorrect ownership of files and directories inside /var/lib/mongodb/ and the mongo log directory /var/log/mongodb will also cause mongo to fail.

For more KB articles/solutions related to Red Hat Satellite 6.x MongoDB Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x MongoDB-related Issues

Root Cause

• During a normal shutdown of pulp_streamer service the lock file for the mongod.service will be removed. When a server crash occurs, the lockfile is not removed, causing failure during subsequent attempt to start pulp_streamer service.

Diagnostic Steps

• TCP port 27017 is the default listening port for mongod.service. The localhost:27017: [Errno 111] Connection refused error indicates there is likely a problem with startup of the MongoDB component of the pulp_streamer service. Check the status of the mongod.service with the following command:

    # systemctl status mongod.service
  

• In this circumstance, output of the systemctl command indicates the mongod.service failed on startup:

    * mongod.service - High-performance, schema-free document-oriented database
       Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
       Active: failed (Result: exit-code) since Tue 2018-05-29 15:40:14 MDT; 50min ago
       Process: 1917 ExecStart=/usr/bin/mongod $OPTIONS run (code=exited, status=100)
  
       May 29 15:40:12 sat6.example.com systemd[1]: Starting High-performance, schema-free document-oriented database...
       May 29 15:40:14 sat6.example.com mongod[1917]: about to fork child process, waiting until server is ready for connections.
       May 29 15:40:14 sat6.example.com mongod[1917]: forked process: 2456
       May 29 15:40:14 sat6.example.com mongod[1917]: ERROR: child process failed, exited with error number 100
       May 29 15:40:14 sat6.example.com systemd[1]: mongod.service: control process exited, code=exited status=100
       May 29 15:40:14 sat6.example.com systemd[1]: Failed to start >High-performance, schema-free document-oriented database.
       May 29 15:40:14 sat6.example.com systemd[1]: Unit mongod.service entered failed state.
       May 29 15:40:14 sat6.example.com systemd[1]: mongod.service failed.
  

• Additional evidence can be found in the MongoDB log, located at /var/log/mongodb/mongodb.log:

    2018-05-29T15:40:14.673-0600 [initandlisten] exception in initAndListen: 12596 old lock file, terminating**
    2018-05-29T15:40:14.673-0600 [initandlisten] dbexit:
  

• This confirms that the lock file from the previous mongodb session was not removed as it would have been during a normal shutdown of Satellite 6 services.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.