How do I tune the size of the DLM hash table to increase the performance of my GFS/GFS2 filesystem in a RHEL 5 or 6 cluster?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 5 or 6 with the High Availability or Resilient Storage Add On
    • RHEL 5: cman-2.0.115-118.el5_10.1 or higher
    • RHEL 6: All cman releases

Issue

  • How do I tune the size of the DLM hash table to increase the performance of my GFS/GFS2 filesystem?
  • I want to increase the size of the DLM hash table on my cluster but I cannot go over 2048.
  • How can I increase the GFS2 locktable buffers?

Resolution

To set DLM hash table sizes persistently to be applied each time cman starts, configure values for the following lines in /etc/sysconfig/cman (which can be created manually if it does not yet exist):

DLM_LKBTBL_SIZE=<value>
DLM_RSBTBL_SIZE=<value>
DLM_DIRTBL_SIZE=<value>

For example: 

DLM_LKBTBL_SIZE=16384
DLM_RSBTBL_SIZE=16384
DLM_DIRTBL_SIZE=16384

NOTE: These values will not be applied to DLM until cman is started again, which requires unmounting all GFS2 filesystems and stopping all other cluster daemons.

To check or change these settings dynamically while the cluster is running, the values can be catted or echoed at locations in the configfs, which should already be mounted if cman has been started.

# grep configfs /proc/mounts 
none /sys/kernel/config configfs rw,relatime 0 0
# cat /sys/kernel/config/dlm/cluster/lkbtbl_size 
1024
# cat /sys/kernel/config/dlm/cluster/rsbtbl_size 
1024
# cat /sys/kernel/config/dlm/cluster/dirtbl_size 
1024
# echo 16384 > /sys/kernel/config/dlm/cluster/lkbtbl_size 
# echo 16384 > /sys/kernel/config/dlm/cluster/rsbtbl_size 
# echo 16384 > /sys/kernel/config/dlm/cluster/dirtbl_size

NOTE: Changing the values on-the-fly via the configfs does not change the setting for any existing DLM lockspaces; these values are only taken into account upon lockspace creation, so any existing GFS2 file systems or other users of DLM would need to be restarted in order to take advantage of the change. Also note that these settings are not persistent, so an unload of the dlm module would set these back to their default values. Using /etc/sysconfig/cman and restarting cman as stated above is recommended.

To change these settings to a value greater than the default value of 1024, up to 16384, one of the following kernels or later must be used:

NOTE: There is a method for changing glock hashtables but should be used only in certain situations outline in article: There is a degradation in the performance of a GFS2 filesystem(s) after they accessed over a million files on RHEL 6 and RHEL 7

Root Cause

There was no way to permanently change the DLM hash table list sizes from the default of 1024 in earlier releases of RHEL 5 and 6, and DLM hash table default sizes can be too low in some cases for a GFS/GFS2 filesystem under heavy use. Even with the ability to change them, the maximum allowed value of 2048 was also be too low for some situations in kernels prior to release RHEL 5.7: 2.6.18-274.3.1.el5, RHEL 5.8: 2.6.18-308.el5, or RHEL 6.2: 2.6.32-220.el6.

Increasing the size of DLM's hash tables can reduce overhead in operations carried out frequently such as in GFS/GFS2 locking by ensuring that buckets within those tables are stuffed with less entries, bringing down search time for the objects stored within.

NOTE: RHEL 7 has eliminated 2 out of 3 of these settings, and the third should not need to be tuned in any situation due to a change in the implementation of DLM in this area.

Diagnostic Steps

The following symptoms may be indicative of this issue:

  • Mounting a GFS or GFS2 filesystem takes a long time after a reboot.

  • The processes dlm_recv and dlm_recoverd consume 100% CPU on other cluster nodes.

  • If there are a lot of DLM locks in use on the system. Which can be determined by looking at the first figure after dlm_lkb in the file /proc/slabinfo.

    # grep dlm_lkb slabinfo | awk '{print $2}'
2304563

To determine if DLM lock-table tuning may be beneficial then the length of DLM's internal hash table lists needs to be calculated. If the calculated length of the DLM's internal hash table lists is in the hundreds then increasing DLM hash table sizes may help.

The total number of DLM hash table lists is listed in the following file:

# grep dlm_lkb /proc/slabinfo
# name  <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
dlm_lkb    2304563     6342530      232         17           1         : tunables   120       60            8         : slabdata     373090        373090      959

Each DLM hash table list contains DLM objects. The total number of objects is found by running this command:

# echo "$(grep dlm_lkb /proc/slabinfo | awk '{print $3}')"
6342530

The length of each DLM hash table list can be calculated by <total number of DLM hash objects> / <Total number of DLM lists>.

echo "$(grep dlm_lkb /proc/slabinfo | awk '{print $3}') / $(cat /sys/kernel/config/dlm/cluster/lkbtbl_size)"  | bc
6193

In this example the size of each DLM hash list is in the thousands(6193), so increasing the total number of DLM hash lists could be beneficial.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.