How to repair MongoDB database?

Solution Verified - Updated

Environment

  • Red Hat Satellite 6
  • Red Hat Update Infrastructure

Issue

  • /var/lib/mongodb/ utilization is high and is rapidly increasing.
  • Rapid increase in filesystem utilization due to MongoDB.
  • How to reduce the disk space of /var/lib/mongodb/.

Resolution

  • Take a snapshot of the Red Hat Satellite or Capsule server if you are running on VM, if not please take backup of /var/lib/mongodb/ directory and keep it safe on the remote system.

  • The below process requires free disk space equal to the size of your current data set plus 2 gigabytes. If the volume that holds dbpath lacks sufficient space, you can mount a separate volume and use that for the repair.

  • If you have enough free space on that volume, execute the below commands in the Red Hat Satellite or Capsule server to recreate all the indexes and to repair the database from any corruption.

      # satellite-maintain service stop
  # du -sh /var/lib/mongodb/ 
  # mongod --dbpath  /var/lib/mongodb/  --repair
  # chown -R mongodb:mongodb /var/lib/mongodb/* 
  # ls -l /var/lib/mongodb/ 
  # du -sh /var/lib/mongodb/ 
  # satellite-maintain service start

  • Otherwise follow this procedure and use another volume for the repair. We used LVM in this example, but any other temporary disk should be fine too.

      # satellite-maintain service stop
  # du -sh /var/lib/mongodb/
  ### Attach a temporary volume to that system, LVM used here, but could be any other storage too 
  # lvcreate -L <size of "du" output + 10%> -n LV_temporary <volume_group_with_enough_space>
  # mkfs.ext4 -m0 /dev/mapper/<volume_group_with_enough_space>-LV_temporary
  # mkdir /var/lib/mongodb/temporary/
  # mount /dev/mapper/<volume_group_with_enough_space>-LV_temporary /var/lib/mongodb/temporary/
  # mongod --dbpath /var/lib/mongodb --repairpath /var/lib/mongodb/temporary --repair
  # chown -R mongodb:mongodb /var/lib/mongodb/*
  # umount /var/lib/mongodb/temporary/
  # rmdir /var/lib/mongodb/temporary/
  # lvremove <volume_group_with_enough_space>/LV_temporary
  # satellite-maintain service start

Note:

  • The above process re-creates all indexes in the database.
  • The time requirement for MongoDB repair depends on the size of the data set.
  • This command obtains a global write lock and will block other operations until it has completed.
  • You should only use the repair command and associated wrappers if you have no other options. These operations remove and do not save any corrupt data during the repair process.
  • --repairpath option is meaningful only in case of MMAPv1 engine. WiredTiger performs in-place repair and does not require any additional disk space. If you are unsure what engine is set, run:
mongo pulp_database --eval "db.serverStatus().storageEngine"

For more KB articles/solutions related to Red Hat Satellite 6.x MongoDB Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x MongoDB-related Issues

SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.