Pacemaker cluster resources are unable to mount EFS with EFS Mount helper

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux Server 7, 8, 9 (with the High Availability Add On)- Pacemaker
  • AWS Virtual Machines
  • nfs
  • efs

Issue

  • Unable to use efs filesystem type with ocf:heartbeat:Filesystem resource agent.
  • Creating a Filesystem resource with efs fstype in Pacmeaker triggers a failure to start the resource.

Example:

When starting as a cloned resource:

$ pcs resource create efs_fs Filesystem device='nfsshare:/' directory='/mnt/efs_fs' fstype='efs' clone interleave=true

$ cat /var/log/messages
------------------------------------->8--------------------------------
Dec  9 12:08:02 nodea Filesystem(efs_fs)[928948]: INFO: Running start for nfsshare:/ on /mnt/efs_fs
Dec  9 12:08:02 nodea Filesystem(efs_fs)[928948]: ERROR: DANGER! efs on nfsshare:/ is NOT cluster-aware!
Dec  9 12:08:02 nodea Filesystem(efs_fs)[928948]: ERROR: DO NOT RUN IT AS A CLONE!
Dec  9 12:08:02 nodea Filesystem(efs_fs)[928948]: ERROR: Politely refusing to proceed to avoid data corruption.
Dec  9 12:08:02 nodea pacemaker-execd[6845]: notice: efs_fs_start_0[928948] error output [ ocf-exit-reason:DANGER! efs on nfsshare:/ is NOT cluster-aware! ]
Dec  9 12:08:02 nodea pacemaker-execd[6845]: info: efs_fs start (call 83, PID 928948) exited with status 6 (execution time 114ms, queue time 0ms)
Dec  9 12:08:02 nodea pacemaker-controld[6848]: notice: Result of start operation for efs_fs on nodea: not configured
Dec  9 12:08:02 nodea pacemaker-controld[6848]: notice: nodea-efs_fs_start_0:83 [ ocf-exit-reason:DANGER! efs on nfsshare:/ is NOT cluster-aware!\n ]

When starting as a standalone resource:

$ pcs resource create efs_fs Filesystem device='nfsshare:/' directory='/mnt/efs_fs' fstype='efs'

Dec  9 12:10:49 nodea Filesystem(efs_fs)[930163]: INFO: Running start for nfsshare:/ on /mnt/efs_fs
Dec  9 12:10:49 nodea Filesystem(efs_fs)[930163]: ERROR: Couldn't find device [nfsshare:/]. Expected /dev/??? to exist
Dec  9 12:10:49 nodea pacemaker-execd[6845]: notice: efs_fs_start_0[930163] error output [ ocf-exit-reason:Couldn't find device [nfsshare:/]. Expected /dev/??? to exist ]
Dec  9 12:10:49 nodea pacemaker-execd[6845]: info: efs_fs start (call 102, PID 930163) exited with status 5 (execution time 274ms, queue time 0ms)
Dec  9 12:10:49 nodea pacemaker-controld[6848]: notice: Result of start operation for efs_fs on nodea: not installed

Resolution

Red Hat Enterprise Linux 7


This feature is not supported on RHEL 7.

Red Hat Enterprise Linux 8

  • The issue (bugzilla bug:2049319) has been resolved with the errata RHBA-2023:6899 with the following package(s): resource-agents-4.9.0-48.el8_9, resource-agents-paf-4.9.0-48.el8_9 or later.

Red Hat Enterprise Linux 9

  • The issue (bugzilla bug: 2142002) has been resolved with the errata RHBA-2023:6312 with the following package(s): resource-agents-paf-4.10.0-44.el9_3, resource-agents-4.10.0-44.el9_3, resource-agents-cloud-4.10.0-44.el9_3 or later.

With the above errata the ocf:heartbeat:Filesystem cluster resource agent now supports the Amazon Elastic File System (EFS). You can now specify fstype=efs when configuring a Filesystem resource. You will need to consult with your vendor on setting up EFS to work with your cluster nodes.

To use the new fstype=efs a Content from docs.aws.amazon.com is not included.3rd party utility will be need to be installed (which provides the command mount.efs) for mounting the EFS filesystem. Consult with your vendor if you have more information related to installing mount.efs command.

For example:

# pcs resource create p_Filesystem ocf:heartbeat:Filesystem device=fs-00aXXXXXXXXXX:/ directory=/mnt/efs fstype=efs options="mounttargetip=192.168.1.125" op monitor OCF_CHECK_LEVEL="0"

Workaround

As a workaround a Filesystem using the "efs" wrapper script can be mounted using the "nfs4" fstype:

$ pcs resource create efs_fs Filesystem device='nfsshare:/' directory='/mnt/efs_fs' fstype='nfs4' clone interleave=true

$ pcs resource create efs_fs Filesystem device='nfsshare:/' directory='/mnt/efs_fs' fstype='nfs4'

This configuration has been confirmed working, but the added functionality available from the mount.efs wrapper script will not be available while using this workaround.

For more information:

Root Cause

Amazon provides a repo from which you can build package called amazon-efs-utils [1]. The amazon-efs-utils provides the EFS mount helper (mount.efs), which gives access to AWS-specific mount options like iam, accesspoint, awsprofile, etc. It also allows the user to configure added security [2].

Upon install of the amazon-efs-utils package, the efs filesystem type becomes available under /proc/filesystems and can be mounted manually as efs filesystem type using mount command or through /etc/fstab. This filesytem type is not currently available with the Filesystem resource agent, and will be made available on completion of the BUG. An efs filesystem can be mounted with fstype nfs, which should work with the Filesystem resource agent but which doesn't provide the added features of the EFS mount helper.

[1] Content from docs.aws.amazon.com is not included.Manually installing the Amazon EFS client
[2] Content from docs.aws.amazon.com is not included.Mounting file systems using the EFS mount helper

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.