During failover of a pacemaker resources, a Filesystem resource kills processes not using the filesystem

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux Server 6, 7, 8 or 9 (with the High Availability Add On)
  • pacemaker

Issue

  • During failover of pacemaker resources, a Filesystem resource is not stopping.
  • A Filesystem resource mounting an nfs filesystem fails to stop successfully.
  • When a Filesystem resource that is mounting an nfs filesystem is stopped it causes other pacemaker resources to fail.
  • During failover of a pacemaker resources, a Filesystem resource kills processes not using the filesystem

Resolution

In some cases the initial umount attempt made on a filesystem or cluster filesystem resource may fail, for instance if something is still using the file system. In such cases, the force_unmount option can be enabled to cause the agent to check for any processes still using the file system and kill them, following it with more unmount attempts.

If the default force_unmount value of true does not allow the Filesystem to be successfully stopped then change the attribute value to safe for the attribute force_unmount.

# pcs resource update <ResourceID>  force_unmount="safe"

The default value of true for force_unmount has been known to cause Filesystem resource for nfs filesystems to not be able to stop successfully because fuser is unable stop all processes that are accessing the filesystem which leads to the filesystem not being able to stop successfully. If the nfs server is unresponsive this could lead to fuser not being able to stop all the processes accessing the filesystem successfully. If the nfs server continues to be unresponsive, then using safe could yield a better chance of the filesystem being successfully stopped.

If there are multiple Filesystem resources mounting an nfs filesystem from the same nfs server, then we recommend that the Filesystem resources mounting nfs filesystem use the option force_unmount="safe". For example, in This content is not included.SAP netweaver cluster they use multipleFilesystem resources and they are configured to use force_unmount="safe" to avoid this issue.

Another solution that can be done is to declare an fsid on the nfs export that is declared on the nfs server. When the cluster node mounts the nfs filesystem then the fuser will be able to distinguish between which processes are using files on the different nfs filesystems that are mounted. If an fsid is declared on the server side for an nfs export then the value of force_unmount can be true or safe as both should work correctly by only killing process that are running on that particular Filesystem resource.

For more information on this issue then see the following article: The fuser -mu displays wrong information for NFS file systems from NetApp qtrees

For RHEL 8 and 9 :

Root Cause

The force_unmount option for the Filesystem resource has 3 possible values:

  • true : Default value, kill processes accessing mount point
  • safe : Kill processes accessing mount point using methods that avoid functions that could potentially block during process detection.
  • false : Do not kill any processes.

The safe option uses shell logic to walk the /procs/ directory for pids using the mount point while the default option true uses the fuser cli tool. The fuser cli tool is known to perform operations that can potentially be block if fuser is searching an unresponsive nfs mounts.

In addition, if force_unmount="true" is used then that can cause fuser to identity other processes not accessing the Filesystem resource to be killed. This can cause other pacemaker resources to fail because their one of their processes was killed when it should not have been. This issue is described in the article: The fuser -mu displays wrong information for NFS file systems from NetApp qtrees

Diagnostic Steps

  • Are there multiple Filesystem resources mount nfs filesystems from the same nfs server?
  • Does stopping a Filesystem cause another resource to fail and/or the Filesystem is killing processes that are running on a different filesystem? [1]

[1] Example of Filesystem resource killing processes running on another filesystem.

The Filesystem resource is stopped and tries to unmount the filesystem with /usr/sap/RH2/ERS29. Then unmount does not occur because the Filesystem resource believes there are processes still using that filesystem.

Feb 12 12:40:26  Filesystem(rh2_fs_ers29)[20177]:    INFO: Trying to unmount /usr/sap/RH2/ERS29
Feb 12 12:40:26  Filesystem(rh2_fs_ers29)[20177]:    ERROR: Couldn't unmount /usr/sap/RH2/ERS29; trying cleanup with TERM
Feb 12 12:40:27  Filesystem(rh2_fs_ers29)[20177]:    INFO: sending signal TERM to: rh2adm    4245     1  0 12:27 ?        Ssl    0:00 /usr/sap/RH2/ERS29/exe/sapstartsrv pf=/sapmnt/RH2/profile/RH2_ERS29_rhers -D -u rh2adm

Some of the process that it believes is running on the filesystem are actually running on a different filesystem as shown below.

Feb 12 12:40:27  Filesystem(rh2_fs_ers29)[20177]:    INFO: sending signal TERM to: rh2adm   18964     1  3 12:40 ?        Ssl    0:00 /usr/sap/RH2/ASCS20/exe/sapstartsrv pf=/sapmnt/RH2/profile/RH2_ASCS20_rhascs -D -u rh2adm           
Feb 12 12:40:27  Filesystem(rh2_fs_ers29)[20177]:    INFO: sending signal TERM to: rh2adm   19167 18964  0 12:40 ?        Ss     0:00 sapstart pf=/sapmnt/RH2/profile/RH2_ASCS20_rhascs
Feb 12 12:40:27  Filesystem(rh2_fs_ers29)[20177]:    INFO: sending signal TERM to: rh2adm   19177     1  0 12:40 ?        Ss     0:00 sapstart pf=/sapmnt/RH2/profile/RH2_ASCS20_rhascs
Feb 12 12:40:27  Filesystem(rh2_fs_ers29)[20177]:    INFO: sending signal TERM to: rh2adm   19214 19177  0 12:40 ?        Ss     0:00 ms.sapRH2_ASCS20 pf=/usr/sap/RH2/SYS/profile/RH2_ASCS20_rhascs
Feb 12 12:40:27  Filesystem(rh2_fs_ers29)[20177]:    INFO: sending signal TERM to: rh2adm   19215 19177  4 12:40 ?        Ssl    0:00 en.sapRH2_ASCS20 pf=/usr/sap/RH2/SYS/profile/RH2_ASCS20_rhascs
Feb 12 12:40:28  Filesystem(rh2_fs_ers29)[20177]:    INFO: unmounted /usr/sap/RH2/ERS29 successfully

When those processes running on a different filesystem are killed the pacemaker resource that spawned those process will fail a monitor operation. This will cause the resource to begin recovery because of a failed monitor that only occurred because fuser believe the process was running on a different filesystem resource.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.