There is a degradation in the performance of a GFS2 filesystem(s) after they accessed over a million files on RHEL 6 and RHEL 7

Solution Unverified - Updated

Environment

  • Red Hat Enterprise Linux Server 6 (with the High Availability and Resilient Storage Add Ons)
  • Red Hat Enterprise Linux Server 7 (with the High Availability and Resilient Storage Add Ons)
  • A Global Filesystem 2(GFS2)

Issue

  • There is a degradation in the performance of a GFS2 filesystem(s) after they accessed over a million files on RHEL 6 and RHEL 7.
  • Is there a way to modify the glock hash table size for GFS2 filesystem(s) on RHEL 6 or RHEL 7?

Resolution

The following errata add support for resizable glock hash tables and should only be used when there is a performance issue on GFS2 filesystems and there are millions of glocks that are in the locktable (See the diagnostic steps on how to determine the number of glocks):

RHEL 6

The updated kernel for RHEL 6 includes a new parameter for the `gfs2` module called `gl_hash_size`. This parameter `gl_hash_size` allows the glock hash size to be specified when the gfs2 kernel module is loaded. It also prints the hash size used, unless it is the default value (of 32K).

A bigger glock hash table size means we have more hash buckets to manage, but fewer glocks per bucket. Which will mean that finding the right glock is quicker because there are fewer glocks to search within each unique bucket.

The parameter can be set like the following example after all gfs2 filesystems are unmounted. Output will be logged to dmesg and will indicate the current gl_hash_size value unless it is the default value of of 32K:

# rmmod gfs2
# modprobe gfs2 gl_hash_size=256K
# dmesg |tail -2
GFS2 (built Sep  4 2015 09:08:15) installed
GFS2: Using glock hash table size: 262144

If there are GFS2 filesystems already mounted, those filesystems should be unmounted, then the gfs2 kernel module should be removed from memory with rmmod gfs2, then inserted again with modprobe or insmod, specifying the gl_hash_size parameter. The gl_hash_size parameter may only be set when loading the gfs2 kernel module into memory, before any GFS2 filesystems are mounted.

When the parameter is set it should write a log to dmseg unless it is set to default size of 32k. The size of parameter can be checked by viewing the /sys/module/gfs2/parameters/gl_hash_size.

# dmesg | tail -n 2
GFS2 (built Apr 13 2016 00:52:15) installed
GFS2: Using glock hash table size: 262144
# cat /sys/module/gfs2/parameters/gl_hash_size
256K

In order to make the change permanent then perform the following:

  1. Edit or create the file /etc/modprobe.d/gfs2.conf.
  2. Add the following line to the file: options gfs2 gl_hash_size=256K then save the file.

Whenever the gfs2 is loaded then the gl_hash_size should be set. The module will only set the parameter size when it is loaded and should be set on next reboot.

The initramfs is not required to be rebuilt for the gfs2 module, but can be. Here is example article on doing this when the devices use multipath: How do I rebuild the initramfs with multipath in Red Hat Enterprise Linux 6 and 7?

RHEL 7


The parameter `gl_hash_size` parameter is unnecessary (and will not be available) on RHEL 7 because GFS2 automatically resizes its glock hash table size based on demand since RHEL 7.2, unlike RHEL 6. This updated kernel added some changes to fix bottlenecks that were causing performance to degrade when there was millions of glock.
Workaround:

NOTE: When or if this issue will occur will vary from cluster to cluster and depends on hardware, operations performed on the GFS2 filesystems(ex: creating millions of small files vs. thousands of large files), etc. Some configuration will be able to handle more total glocks that exist in hashtable than other clusters.

Root Cause

There is a known scalability issue that occurs when there are millions of active glocks with GFS2 filesystems.

Diagnostic Steps

Capture a lockdump of the GFS2 filesystems when the issue is occurring with the gfs2_lockcapture script. For more information see the following article: What data should I gather when access to a GFS2 filesystem appears to be hung or unresponsive on RHEL 5, RHEL 6, or RHEL 7?

After lockdumps are captured count the number of glocks for all the GFS2 filesystems. If there are millions of glocks that exists in the locktable on a cluster node then it is possible that the GFS2 filesystems are affected by this issue. For example:

Mount the debug directory on every cluster node:

# mount -t debugfs debugfs /sys/kernel/debug

Then count the number of glocks for all the GFS2 filesystem that exist in the lockdumps for a cluster node. *You may need to check each cluster node. *

# grep "G\:" /sys/kernel/debug/gfs2/*/glocks | wc -l
1370805

If the total is more than a million then a glock scalability issue might be the reason for the performance degradation.

The more glocks in each hash bucket will lead to longer times to finding the correct glock. By default the number of hash buckets is 32K (32,0000). To determine the average number of glocks in each hash bucket the following formula can be used. If there was 1,000,000 total glocks across all the GFS2 filesystems and a default hash table size of 32K then the following can be used to determine the average number of glocks in each bucket:

1000000 / 32768 = 30.5

In this example there would be roughly 30 glocks in each bucket to search through til you find the right one.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.