How should I configure my RHEL Resilient Storage cluster to manage active/passive shared LVM volumes when I also need to manage active/active LVM volumes for GFS2?

Environment

• Red Hat Enterprise Linux (RHEL) 5, 6, 7, 8 and 9 with the Resilient Storage Add On
• Shared LVM volume groups to be used both for:
  • Active/passive workloads that only need access to shared storage on one node at a time, and
  • Active/active workloads that utilize GFS2 for concurrent access to data across multiple nodes
• pacemaker or rgmanager for resource management

Issue

• What is the recommended deployment strategy for a cluster managing both single-host active/passive logical volumes supporting single-host filesystems such as ext3, ext4, etc and multi-host logical volumes supporting GFS2 distributed filesystems?
• I have a cluster that uses clvmd to share LVM volume groups for GFS2 file systems, and I also need to manage shared volumes for single-host file systems in the same cluster. Red Hat's documentation recommends always using HA-LVM when shared LVM volumes are in use, but I'm not sure if I can use clvmd for GFS2 and HA-LVM in the same cluster. How should I configure my resources?
• LVM resources that use exclusive=true fail to activate in clusters that also use cloned LVM resources with exclusive=false and clvmd in a pacemaker cluster.

Resolution

When the use case for a cluster requires that there be one or more LVM volume groups which will be accessed by multiple nodes at the same time as well as one or more LVM volume groups that will be accessed by only one node at a time, then adhere to the following guidelines applicable to the cluster configuration in question:

• RHEL 8 and 9
• RHEL 7
• RHEL 6 with pacemaker
• RHEL 5 or 6 with rgmanager

RHEL 8 and higher

System configuration

The locking_type parameter does not exist for RHEL 8 and up. Due to changes in these versions, no additional steps are required for "active/active" gfs2 configurations. Follow the standard configuration for setting up a shared volume group for use with gfs2:

• Chapter 7. GFS2 file systems in a cluster | RHEL 8

• Chapter 7. GFS2 file systems in a cluster | RHEL 9

The HA-LVM volume groups can be configured referring to the steps detailed in following article:

• How to configure HA-LVM Cluster using lvmlockd in RHEL 8 and above?

Additional Recommendations and Limitations

• For ease of management, the VG's to be used with active/active gfs2 shares, should be separate from the VG's used for HA-LVM active/passive.

• If an lvmlockd resource is configured either activation_mode modes can be used to manage the LVM-activate resources for active/passive volume groups or logical volumes.

  • activation_mode=exclusive
  • activation_mode=shared

• If activation_mode=exclusive is used with system_id in a setup also managing GFS2 filesystem with lvmlockd resource with use_lvmlockd=1 configured in /etc/lvm/lvm.conf, then lvmlockd resource needs to be running in order to manage LVM-activate resource (even resources that use activation_mode=exclusive). If lvmlockd is not running then LVM-activate resources can fail to start.

• Make sure that all LVM-activate resources have a order constraint configured to start lvmlockd resource before starting the LVM-activate resource if a lvmlockd resource is managed by pacemaker. For more information refer following articles:

  • An LVM-activate resource failed to activate with the error: Cannot access VG shared_vg2 with system ID rhel8-1.examplerh.com with local 'system ID rhel8-2.examplerh.com.
  • Why does the LVM-activate resource fails to activate the VG with ERROR: vg_name: failed to activate ?

RHEL 7

System configuration

Configure locking_type = 3 in /etc/lvm/lvm.conf, and create a clvm clone resource to control starting and stopping of clvmd.

Volume management

• Shared volume groups for GFS2 file systems: Create cloned LVM resource with exclusive=false (the default value). Check that the clustered ('c') attribute is enabled for each of these volume groups in the output of vgs, and enable that bit with vgchange -cy <vg> for any of these shared volume groups that do not yet have it set.

• Shared volume groups to be used on a single node at a time: Create an LVM resource in the pacemaker configuration for each of these volume groups that will only be accessed in an active/passive manner, using resource attribute exclusive=true. Enable the cluster flag with vgchange -cy <vg> .These volume groups should not be cloned LVM resource.

RHEL 6 with pacemaker

System configuration

Configure locking_type = 3 in /etc/lvm/lvm.conf, and enable clvmd to start on boot or ensure that it is started manually with the other cluster daemons when joining a node to the cluster.

Volume management

• Volume groups local to individual nodes that are not shared across the cluster: No special configuration is required to have these volumes accessible on their respective nodes. They do not need to be included in any volume_list in /etc/lvm/lvm.conf, unless such a volume list is being used for other non-cluster-related reasons.

• Shared volume groups for GFS2 file systems: Check that the clustered ('c') attribute is enabled for each of these volume groups in the output of vgs, and enable that bit with vgchange -cy <vg> for any of these shared volume groups that do not yet have it set. The volumes in these groups can optionally have LVM resources created for them in the pacemaker configuration, and/or can be listed in /etc/fstab to be mounted automatically by the gfs2 init service. unless such a volume list is being used for other non-cluster-related reasons.

• Shared volume groups to be used on a single node at a time: Check that the clustered ('c') attribute is enabled for each of these volume groups in the output of vgs, and enable that bit with vgchange -cy <vg> for any of these shared volume groups that do not yet have it set. The volumes in these groups should have LVM resources created for them in the pacemaker configuration using attribute exclusive=false, and also have a Filesystem resources created for each volume that contains a file system. No volumes in any of these volume groups should be listed in /etc/fstab. unless such a volume list is being used for other non-cluster-related reasons.

  • CAUTION: These volumes will be activated automatically on all nodes on boot by the clvmd init script, and thus will be accessible to administrators or software to be manually mounted at-will. It is critically important that the volumes are never mounted manually, and I/O is never performed against any of these volumes except on the primary node where the LVM and Filesystem resource is active. Doing otherwise may result in data corruption.

RHEL 5 or 6 with rgmanager

System configuration

Configure locking_type = 3 in /etc/lvm/lvm.conf, and enable clvmd to start on boot or ensure that it is started manually with the other cluster daemons when joining a node to the cluster.

Volume management

• Volume groups local to individual nodes that are not shared across the cluster: No special configuration is required to have these volumes accessible on their respective nodes. They do not need to be included in any volume_list in /etc/lvm/lvm.conf, unless such a volume list is being used for other non-cluster-related reasons.

• Shared volume groups for GFS2 file systems: Check that the clustered ('c') attribute is enabled for each of these volume groups in the output of vgs, and enable that bit with vgchange -cy <vg> for any of these shared volume groups that do not yet have it set. The volumes in these groups do not require lvm resources be created for them in /etc/cluster/cluster.conf, and can optionally be listed in /etc/fstab to be mounted automatically by the gfs2 init service or can have <clusterfs/> resources created for them in /etc/cluster/cluster.conf to manage them as part of a service. They do not need to be included in any volume_list in /etc/lvm/lvm.conf, unless such a volume list is being used for other non-cluster-related reasons.

• Shared volume groups to be used on a single node at a time: Check that the clustered ('c') attribute is enabled for each of these volume groups in the output of vgs, and enable that bit with vgchange -cy <vg> for any of these shared volume groups that do not yet have it set. Create an lvm resource in /etc/cluster/cluster.conf for each of these volume groups that will only be accessed in an active/passive manner. They do not need to be included in any volume_list in /etc/lvm/lvm.conf, unless such a volume list is being used for other non-cluster-related reasons.

Root Cause

• RHEL 7 with pacemaker: LVM resources should be used to manage multi-host volumes and shared single-host volumes. The multi-host volumes can use exclusive=false and rely on clvmd for shared access control, whereas the single-host volumes can use exclusive=true with the cluster (c)flag enabled.

• RHEL 6 with pacemaker: LVM resources can be used to manage multi-host volumes and should be used to manage shared single-host volumes. Both multi-host and shared single-host volumes should use exclusive=false and rely on clvmd for shared access control.

  • NOTE: This use case differs from the RHEL 7 case in a few ways which make it impossible to manage access control of volumes on-the-fly using a volume_list as the LVM resource agent does with exclusive=true resources while clvmd is also in use. This does not come with the same problems in RHEL 7, where its entirely possible to use both exclusive volumes with tagging and non-exclusive volumes with clvmd simultaneously; but in RHEL 6 there is no way to mix clvmd with HA-LVM + tagging, so the end result is that all shared volumes must be managed by clvmd and be treated as non-exclusive volumes. The downside to this approach is that shared volumes are active on all nodes after boot, which leaves them vulnerable to being accidentally mounted by an administrator or malfunctioning software on multiple nodes at once. HA-LVM + tagging normally provides protection from such a scenario, but as long as there is a need for clvmd to provide non-exclusive access for GFS2-backing volumes, then tagging is unable to be used. So, the administrator must simply take measures to ensure that the exclusive volumes are not accessed inappropriately.

• RHEL 5 or 6 with rgmanager: HA-LVM with clvmd should be used to manage shared single-host volumes, whereas clvmd should be used to manage multi-host volumes for GFS2.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.