The rh-mongodb34-mongod service fails to start with error 'Cannot write pid file to /var/opt/rh/rh-mongodb34/run/mongodb/mongod.pid: Permission denied' in Red Hat Satellite or Capsule 6.

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.9 and below
  • Red Hat Capsule 6.9 and below

Issue

  • The rh-mongodb34-mongod service and other pulp services are in failed state in the foreman-maintain service status command.

        \ All services displayed                                              [FAIL]    
    Some services are not running (rh-mongodb34-mongod, pulp_celerybeat, pulp_resource_manager, pulp_streamer)
    --------------------------------------------------------------------------------
    Scenario [Status Services] failed.

    The following steps ended up in failing state:
    
      [service-status]

    Resolve the failed steps and rerun the command. In case the failures are false positives,
    use --whitelist="service-status"

  • The hammer ping output on satellite sever shows failures in pulp and pulp_auth components.

Resolution

  1. First, go through the Diagnostic Steps section to understand what kind of error messages are expected.

  2. Ensure that /var/lib/mongodb and /var/opt/rh/rh-mongodb34 directories - as well as their parent directories - have correct permission and ownership set, like:

    # getfacl /var/opt/rh/rh-mongodb34
getfacl: Removing leading '/' from absolute path names
# file: var/opt/rh/rh-mongodb34
# owner: root
# group: root
user::rwx
group::r-x
other::r-x
#

  3. Set the SELinux mode to permissive and restart the rh-mongodb34-mongod service.

    # setenforce 0
# systemctl restart rh-mongodb34-mongod
# systemctl status rh-mongodb34-mongod -l

  4. If the service gets started properly, fix the SELinux context of related directories.

    # setenforce 1
# semanage fcontext -a -t mongod_var_run_t "/var/opt/rh/rh-mongodb34/run/mongodb(/.*)?"
# restorecon -RFv /var/opt/rh/rh-mongodb34 /var/lib/mongodb
# systemctl restart rh-mongodb34-mongod

  5. If the result of the above step leads to the same failure, then relabel the SELinux context for the entire system by following the solution article 3354661.

    # echo "-vv" > /.autorelabel
# foreman-maintain service stop
# shutdown -r now

  6. Once the server comes back up, give it some time to start the services, and then check back on the health.

    # foreman-maintain service status -b

For more KB articles/solutions related to Red Hat Satellite 6.x MongoDB Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x MongoDB-related Issues

Root Cause

  • Due to the wrong SELinux context or insufficient permissions or file ownership, the MongoDB service couldn't start up properly.

  • Another possible cause is that the MongoDB packages were installed in the same yum transaction which also contained selinux-policy updates, and this results in broken labels.

This is why our docs tell to run yum update before installing the packages

We have on ongoing This content is not included.bugzilla to track that particular problem

Diagnostic Steps

  • The mongod service shows the following errors during startup, in /var/log/messages file.

     Nov 25 17:57:24 satellite.example.com systemd[1]: Starting High-performance, schema-free document-oriented database...
 Nov 25 17:57:24 satellite.example.com mongodb-scl-helper[32229]: about to fork child process, waiting until server is ready for connections.
 Nov 25 17:57:24 satellite.example.com mongodb-scl-helper[32229]: forked process: 32234
 Nov 25 17:57:24 satellite.example.com mongod.27017[32234]: [main] ERROR: Cannot write pid file to /var/opt/rh/rh-mongodb34/run/mongodb/mongod.pid: Permission denied
 Nov 25 17:57:24 satellite.example.com mongodb-scl-helper[32229]: ERROR: child process failed, exited with error number 1

  • The /var/log/audit/audit.log file has the following SELinux denial related messages printed.

     type=AVC msg=audit(1606324268.686:1815): avc:  denied  { write } for  pid=19027 comm="mongod" name="mongodb" dev="dm-10" ino=29360245 scontext=system_u:system_r:mongod_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=dir permissive=0
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.