The fuser -mu displays wrong information for NFS file systems from NetApp qtrees

Solution Unverified - Updated

Environment

  • NFS v3 Server: NetApp FAS8080, Ontap 8.2.4P4 7-Mode
  • NFS v3 Cient:
  • Red Hat Enterprise Linux (RHEL) 6
  • Red Hat Enterprise Linux (RHEL) 7

Issue

The fuser -mu command lists all processes in the whole NetApp volume, not in the mounted qtree only.

Example for /etc/fstab configuration:

<NetApp Filer>:/vol/<dir>/<share 1> /<mount point 1>  nfs  rw,rsize=65535,wsize=65535,vers=3,bg,hard,intr,proto=tcp  0  0
<NetApp Filer>:/vol/<dir>/<share 2> /<mount point 2>  nfs  rw,rsize=65535,wsize=65535,vers=3,bg,hard,intr,proto=tcp  0  0

Example of fuser output:

# fuser -mu -v <mount point 1> <mount point 2>
USER        PID ACCESS COMMAND
<mount point 1>: root       1627 ..c.. (root)<process>  # process from <mount point 2>
                 root       1636 ..c.. (root)<process>  # process from <mount point 1>
<mount point 2>: root       1627 ..c.. (root)<process>
                 root       1636 ..c.. (root)<process>

Resolution

In order to use fuser -mu /mount-point-1 /mount-point-2 command for NFS shares, you will have to set up a unique filesystem ID for each NFS export on the NFS server by using the fsid= parameter in /etc/exports configuration file.

Notes for NetApp filer:

  • On the NetApp platform, only volumes have a unique fsid, it's not possible to configure a unique FSID for each qtree.
  • NetApp filers have volume limits, which depend on models, so NFS export for qtrees on top of volumes are commonly used.

Notes for RHEL NFS server:

  • Set up a unique filesystem id for each NFS export entry on the server level by adding the fsid= parameter in /etc/exports configuration file.
  • As second option, export NFS shares from separated devices which will have a unique filesystem id (like disks, partitions or logical volumes).
  • For newer versions of psmics package, as in RHEL7, fuser has new option -M, which can be used for particular mount points:
 # fuser -M -u -v /mount-point-1 /mount-point-2.

Root Cause

The fuser command goes over entries in /proc, it resolves /proc/PID/fd symlinks (all opened files are represented as symlinks to original files),
file mappings (/proc/PID/maps) and so on in order to find references to files.

By default, all NFS shares from the same server and from the same device (disk, partition or logical volume), have the same major and minor device numbers associated with the mounted file systems. In other words, they have the same filesystem ID. It causes wrong output from fuser -mu command in RHEL 6 and 7.

Diagnostic Steps

  • Test environment configuration

NFS Server: 

# /etc/exports
/nfs-share-1		192.168.0.0/24(rw,no_root_squash)
/nfs-share-2		192.168.0.0/24(rw,no_root_squash)

Notice: default configuration does not contain fsid= parameter. In addition, both shares are located on the same root partition.

NFS Client:

# /etc/fstab
192.168.0.1:/nfs-share-1        /mount-point-1        nfs defaults,_netdev 0 0
192.168.0.1:/nfs-share-2        /mount-point-2        nfs defaults,_netdev 0 0
  • On client side, identify the major and minor device numbers associated with the mounted file systems
# mountpoint -d /mount-point-1
0:21
# mountpoint -d /mount-point-2
0:21
  • Identify the corresponding fsid from this device number in /proc/fs/nfsfs/volumes file
# cat /proc/fs/nfsfs/volumes
NV SERVER   PORT DEV     FSID              FSC
v4 c0a86f14  801 0:21    672381d7f8bc424f  no
  • Identify nfs servers
# cat /proc/fs/nfsfs/servers
NV SERVER   PORT USE HOSTNAME
v4 c0a86f14  801   1 192.168.0.1
  • Check output from fuser -mu -v command
    Notice: fuser incorrectly shows processes from both mount points
# fuser -mu -v  /mount-point-1 /mount-point-2
 USER        PID ACCESS COMMAND
/mount-point-1: root       1627 ..c.. (root)bash
                root       1636 ..c.. (root)bash
/mount-point-2: root       1627 ..c.. (root)bash
                root       1636 ..c.. (root)bash
  • Modify NFS server configuration by adding different value for fsid= options per share
# /etc/exports
/nfs-share-1		192.168.0.0/24(rw,no_root_squash,fsid=1)
/nfs-share-2		192.168.0.0/24(rw,no_root_squash,fsid=2)
  • On client side, identify the major and minor device numbers associated with the mounted file systems once again
    Notice: different minor numbers are displayed as expected
# mountpoint -d /mount-point-1
0:21
# mountpoint -d /mount-point-2
0:22
  • Identify the corresponding fsid from this device numbers from /proc/fs/nfsfs/volumes file
    Notice: different fsid values are displayed as expected
# cat /proc/fs/nfsfs/volumes 
NV SERVER   PORT DEV     FSID              FSC
v4 c0a86f14  801 0:22    2:0               no 
v4 c0a86f14  801 0:21    1:0               no
  • Check output from fuser -mu -v command once again
    Notice: expected results are present, fuser shows processes only for specific file system
# fuser -mu -v  /mount-point-1 /mount-point-2
 USER        PID ACCESS COMMAND
/mount-point-1:      root       1636 ..c.. (root)bash
/mount-point-2:      root       1627 ..c.. (root)bash
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.