Mongod process with wiredTiger engine consumes up to half of RAM in Satellite6

Solution Verified - Updated

Environment

  • Red Hat Satellite 6
    • mongo configured with wiredTiger mongo engine
    • Satellite 6.7.1 or earlier

Issue

  • mongo updated to use wiredTiger storage engine
  • sometimes, mongod consumes randomly high memory (up to 1/2 of all RAM)
  • this aggressive memory usage limits other processes or can cause OOM killer to kill mongod

Is that high memory usage expected? Is there a way to limit memory footprint of mongod?

Resolution

There are four options for how to relieve the memory pressure:

  1. Upgrade to Satellite 6.7.2 or later which enables memory limits by default. See this Bugzilla and Errata for more details:

  1. Usually, there is another process consuming more than the expected amount of memory. Or a group of processes, when one e.g. sets passenger max pool size too high, causing many Passenger RackApp: /usr/share/foreman processes to pop up. We recommend checking this in the first place.

  2. More memory might be needed for the Satellite server. This depends on the currently allocated RAM and Satellite usage.

  3. We recommend limiting the wiredTiger max cache. This has been tested at limiting the cache to 20% of the Satellite's physical RAM with little to zero performance impact. To set this value, do the following:

Update custom-hiera.yaml:

  1. Edit /etc/foreman-installer/custom-hiera.yaml and add the entry below inserting the value that is 20% of the physical RAM, in this case, approximately 6GB for a 32GB server:
mongodb::server::config_data:
  storage.wiredTiger.engineConfig.cacheSizeGB: 6

Note: If there is already mongodb::server::config_data: section there, append the parameter as mentioned in the example to the existing section, like:

mongodb::server::config_data:
  operationProfiling.slowOpThresholdMs: 1000
  storage.wiredTiger.engineConfig.cacheSizeGB: 6
  1. Run installer to apply changes:
# satellite-installer
...

For more KB articles/solutions related to Red Hat Satellite 6.x MongoDB Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x MongoDB-related Issues

Root Cause

Per Content from docs.mongodb.com is not included.wiredTiger Storage Engine upstream documentation, the engine can consume up to half of all memory for its local cache.

Diagnostic Steps

To check if wiredTiger engine is used, grep in /var/log/messages for a log like:

Oct 16 08:12:51 satellite007 mongod.27017[24485]: [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.