What does GFS2/DLM use /proc/slabinfo for?

Updated

The file /proc/slabinfo is provided by the kernel to report usage of the slabcache and the number and size of active/cached objects of different types. All the GFS2 structures are defined in the kernel source code under the fs/gfs2/incore.h header file.

The /proc/slabinfo/gfs2_* structures will only increase when the cluster node accesses the GFS2 file-system. The /proc/slabinfo/dlm_* structures can continue to increase even if the cluster node is not accessing the GFS2 file-system because of the way that DLM is managed.

Caches that are used by GFS2

  • gfs2_glock
    This is a cache of GFS2's glocks (struct gfs2_glock). Glocks are central to GFS2 inter-node locking layer. There are two glocks associated with each in-memory inode. One is the LM_TYPE_INODE (type 2) and another is the LM_TYPE_IOPEN (type 5). There is one glock of LM_TYPE_RGRP for each resource group and some miscellaneous glocks for special purposes like quota, journaling, etc. For more information on glocks see the articles in the reference section.

    Each inode's metadata is kept in a separate address space because when glocks are requested on remote nodes, we need to be able to efficiently locate the data and metadata for that glock (inode) in order to sync/sync+invalidate glock (inode). In RHEL 5, an extra inode is used that contains the extra address space and the other inode fields are left unused (which leads to memory usage that is rather inefficient. See gfs2_glock(aspace) for how this is handled in RHEL 6.

  • gfs2_glock(aspace)
    In RHEL 6+, a new type of glock was added which includes an address space (in addition to the other fields) called gfs2_glock(aspace). This lock type is used for inode and rgrp glocks (other remain the same as before). This renders an approximate 25% savings in memory used, a removal of circular dependency between inodes and glocks and less confusion in the code that needed to distinguish between normal and metadata inodes. The enhancement also halves the number of inodes used when compared to RHEL 5 which is described in gfs2_glock. This cache object is only available in RHEL 6+ and there is one such object for each cached inode.

  • gfs2_inode
    This is a cache of the GFS2 in-memory inodes struct gfs2_inode. The vfs determines when to allocate and destroy cached inodes by calling the gfs2_alloc_inode() and gfs2_destroy_inode() functions. The GFS2 incore inode contains the VFS inode and additional bits to manage the inode like the inode glock to synchronize access, reservation structure (see gfs2_blkreserv), resource group to allocate blocks from, etc. There is one object per cached inode in the gfs2_inode cache.

  • gfs2_bufdata
    Cache of the struct gfs2_bufdata. These structs refer to data and metadata blocks that require journaling and hang off of lists in the superblock. Metadata and journaled data (ordered) blocks are kept track of, using these structs and are added to transactions to be committed to the journal as and when required during various filesystem operations.

  • gfs2_rgrpd
    Cache of struct gfs2_rgrpd. Each gfs2_rgrpd describes a unique resource group on disk and holds values for rgrp attributes like, disk address, number of data blocks, number of free blocks, number of bytes used in data bitmaps, etc. These structs are arranged in an rbtree that hangs off of the superblock. The number of objects corresponds to the number of resource groups in memory.

  • gfs2_quotad
    Cache of struct gfs2_quota_data. These objects hang off of a list in the superblock and are used to hold quota information for uids/gids. These objects are created when quotas are enabled for uids/gids when the first quota-affecting operation is run. Normally, they remain in cache in case they are needed again and the entire cache is destroyed during an unmount of the GFS2 file-system. Refer to fs/gfs2/quota.c for more info on how quotas work and GFS2 Administration Guide on GFS2 Quota Management.

  • gfs2_mblk
    Contains struct gfs2_blkreserv. A resource group has several of these multi-block reservation structures in an rbtree with a reservation attached to each GFS2 inode. The idea is to pre-reserve blocks which would reduce file fragmentation and result in an improved disk layout thereby improving performance. If an inode needs to be written to, a struct gfs2_blkreserv is allocated for it. This will be deallocated when the struct file associated is released. This cache object is only available in RHEL 6+.

Caches that are used by DLM

  • dlm_lkb
    A glock will be represented by one dlm_lkb struct on a system. There are also dlm_lkb structs for remote glock/lkb pairs, and the number of these is entirely dependent on the degree and pattern of remote mastering of resources across the cluster. The only way to determine the number or percentage of remotely mastered resources is by dumping all the dlm lock state.

  • dlm_rsb
    There will be one rsb struct for each glock on a system, plus some additional rsb structs for glocks that the node was using in the past. This cache object is only available in RHEL 7+.

  • dlm_conn
    These are open connections to other nodes in the cluster. Typically 1 to 2 connection objects per node (excluding self).

Reference

Product(s)
Category
Components
Tags
Article Type