Increasing promote/publish speed in Satellite 6 by compacting mongoDB

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.9 or older
    • MongoDB

Issue

Satellite 6 backend system Pulp uses MongoDB database for storing repository metadata. As more and more data files are added by MongoDB, the total database size grows from a default 3 gigabytes up to tens or hundreds of gigabytes. Data is fragmented and promotion or publishing speed decreases.

This can be improved by bringing the Satellite 6 instance down and performing a MongoDB maintenance operation called "database repair". Our numbers show that the size of the database decreases drastically (around 50 %) and promote (or publish) operations are faster by 60-80 %.

The same steps from this Solution can also be applied to the Capsule server, where the impact to performance (which has not been measured) is expected to be smaller, but still visible (in cases of already fragmented data in MongoDB).

Resolution

The Mongo "repair database" operation rebuilds the database and indexes by discarding invalid or corrupt data, which may be present due to an unexpected system restart or crash. This is analogous to a fsck command for file systems.

WARNING: Before doing any action with mongodb command line tool, make a full backup of mongodb database.

IMPORTANT: There must be extra free space on mongodb volume of 120 %. (In the output of the df /var/lib/mongodb command, the Use% should be 45% or less). E.g. if the database is 20 GB in size, volume must have at least 24 GB of free space.

Before performing MongoDB repair, it is worth deleting old Content View versions in Satellite 6 UI. There is not yet an auto-removing feature, but we currently provide an external Content from github.com is not included.helper script which will help in automating such a cleanup.

To clean up mongodb, shutdown all services and make a full backup (with running mongod) first:

foreman-maintain service stop --exclude=rh-mongodb34-mongod
mongodump --host localhost --out /some/path/to/mongo_dump
foreman-maintain service stop --only=rh-mongodb34-mongod

(on older Satellite6 versions, use katello-service stop; service mongod start to let just mongo running, and then stop it via service mongod stop)

Then do the clean itself:

sudo -u mongodb mongod --dbpath /var/lib/mongodb --repair

This option, which checks and repairs errors and inconsistencies in data storage, basically makes a copy of the whole database, getting rid of unused records and space. Before running the command, make sure that /var/lib/mongodb/mongod.lock does not exist and that the service is not running. (katello-service stop should have stopped it.).

NOTE: According to the size of your mongodb instance, the repair could take a lot of time. You're suggested to execute the commands from a tmux or screen session in order to preserve the task from stopping if you'll be disconnected.

If the following error message is reported:

You specified --repair but there are dirty journal files. Please
restart without --repair to allow the journal files to be replayed.
If you wish to repair all databases, please shutdown cleanly and
run with --repair again.

you might need to increase memory on your server.

Use the --repairpath switch to specify the folder in which to store temporary repair files, if default /var/lib/mongodb does not suit. Be aware that above command removes and do not save any corrupt data during the repair process.

Once the repair is completed, start all Red Hat Satellite/Capsule processes:

foreman-maintain service start

(on older Satellite6 versions, use katello-service start)

Other MongoDB tuning

It is recommended to mount MongoDB data volumes with "noatime" option. The recommended file system for these volumes is XFS.

The amount of RAM is also a key value for MongoDB. Although Satellite 6 can be installed on systems with as little as 16 GB of RAM, memory can be a bottleneck on such systems when MongoDB grows to more than 10 GB in (disk) size. The rule of thumb is to add the size of MongoDB to the minimum memory recommended by Red Hat. For example, if MongoDB is 10 GB in disk size, the total RAM should be 16 + 10 = 26 GB RAM minimum. For systems with more than 64 GB RAM, it is recommended to tune dirty ratio to free up more RAM for regular use:

vm.dirty_ratio = 15
vm.dirty_background_ratio = 5

Splitting up services into individual servers (Pulp, Candlepin, PostgreSQL, MongoDB) can help increase performance and scalability. This is currently (fall 2017) not supported, but there are plans to do that.

For more KB articles/solutions related to Red Hat Satellite 6.x MongoDB Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x MongoDB-related Issues

Root Cause

Fragmented MongoDB data files.

Diagnostic Steps

MongoDB data files consume a lot of space which does not fit into RAM. If the MongoDB size is bigger than the amount of RAM, then it should be repaired.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.