The dentry_cache / dentry slab cache size continually grows on Red Hat Enterprise Linux

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 4
  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 9

Issue

  • application creates a large number (~15,000/hour) of uniquely named files which are deleted after a certain time
  • Only about ~5000 files exist at any given time
  • The memory usage reported by free becomes very high
  • Investigation shows that the kernel memory slab cache of dentry structures (called dentry_cache in RHEL4 or RHEL5, called dentry in RHEL6 or RHEL7) is the cause of high usage
  • What will happen when the dentry_cache exhausts the available (cached/free) memory?
  • Why are dentry objects not freed after deletion of the file? The process owner? Could there be a hidden recycle bin somewhere due to the java delete method?
  • Can you explain the vfs_cache_pressure in more detail?
  • What other means can we do to reduce the dentry_cache size?
  • Can vm.drop_caches=2 be used?  Is there a different/preferred method? How safe is it? Can we run this a cron job?

Resolution

The dentry cache (and any other slab cache for that matter) is free to grow to any size while there is free memory. It is simply making good use of available memory. When free memory is low the cache will be reclaimed on demand and the freed memory will be available for other purposes. Also when a filesystem is unmounted all of its directory entries will be removed from the cache so unmounting is one workaround to help clear the cache. Only if the cache is not being reclaimed is there a problem and that may lead to memory starvation issues.

The dentry_cache can be purged by periodically telling the kernel to drop the cache.

echo 2 > /proc/sys/vm/drop_caches

The drop cache request will free any dentries and inodes that are not currently in use. Care should be taken to avoid dropping kernel cache while under very high io loads. Typically using a cron job to drop the cache late at night after backups/night maintenance activities but before daily production hours start.

NOTE: in testing, increasing vfs_cache_pressure seemed to have no practical effect on this particular issue.

Root Cause

Dentry cache is also called "directory entry cache":

The Linux directory entry ("dentry") cache is a key part of the kernel's filename lookup mechanism. The dentry cache speeds the process of looking up files considerably. Dentry cache is used to improve the performance. A file system will have one root dentry (referenced in the superblock), this being the only dentry without a parent. All other dentries have parents, and some have children. For example, if a file is opened that's made up of /home/user/name, four dentry objects are created: one for the root /, one for the home entry of the root directory, one for the name entry of the user directory, and finally, one dentry for the name entry of the user directory. In this way, dentries map cleanly into the hierarchical file systems in use today.

For more information on dentry cache refer this link: Content from lwn.net is not included.Content from lwn.net is not included.http://lwn.net/Articles/419811/

We can see high dentry_cache usage on the systems those who are running some programs which are opening and closing huge number of files. Sometimes high dentry_cache leads the system to run out of memory in such situations performance gets severly impacted as the system will start using swap space.

Diagnostic Steps

  • Testing attempted using vm.vfs_cache_pressure values >100 has no effect.
  • Testing using the workaround of echo 2 > /proc/sys/vm/drop_caches immediately reclaims almost all dentry_cache memory.
  • use slabtop to monitor how much dentry_cache is in use:
OBJS    ACTIVE  USE  OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                  
9544482 9544482 100%    0.21K 530249       18   2120996K dentry_cache
                                                ^^^^^^^^ w/this continually growing.
  • Used the attached test program (./test.bsh) which seems to reproduce and demonstrate the issue.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.