What is the recommended LVM configuration when multiple Red Hat Enterprise Linux cluster nodes are accessing the same shared storage?

Solution Verified - Updated

Environment

  • Red Hat Cluster Suite 4 / Red Hat Global File System 6.1
  • RHEL 5, 6, or 7 with the High Availability and/or Resilient Storage Add On
  • Shared storage devices that will be managed by the Logical Volume Manager (LVM)

Issue

  • How can multiple systems be confirgured to access the same backend storage?
  • When should clvmd be used instead of HA-LVM?
  • I have shared LVM devices. Do I need to do anything special to protect the metadata from being corrupted by multiple nodes accessing the volumes at the same time?

Resolution

Why Use LVM in a Highly Available Cluster?

Logical volume management (LVM) creates a layer of abstraction over physical storage, allowing the creation of logical storage volumes. This provides much greater flexibility in a number of ways than using physical storage directly, and can be employed on standalone systems or on clusters accessing shared storage.  However, in a cluster, some precautions must be taken to ensure the integrity of data stored on devices managed by LVM.

When a system first scans its storage devices for LVM physical volumes, it reads the metadata on any it finds and uses the device-mapper infrastructure to assemble a linear, striped, mirrored or snapshotted map of that physical storage.  These maps define the boundaries of each logical volume and thus what storage addresses can be used for data and filesystem information.  After the initial scan, this information may be cached in memory and used later when accessing those storage devices.  When multiple systems are accessing the same devices, such as in a cluster, it is critical that they communicate any changes made to the LVM metadata back to the other systems.  If one system were to change the metadata and the other system was not notified of this, it could lead to corruption of the LVM metadata or the data residing on the physical volumes.

In RHEL High Availability or Resilient Storage clusters, there are two methods for managing shared LVM volumes across nodes:

clvmd for Active/Active Workloads Concurrently Accessing Shared Storage

An extension to the lvm2 package called lvm2-cluster exists in the Resilient Storage (RHEL 6 and 7) or Cluster Storage (RHEL 5) channel or repo.  This package provides the clvmd daemon which is responsible for managing clustered volume groups and communicating metadata changes for them back to other cluster members.  With this option, the same volume group and logical volumes can be activated and mounted on multiple nodes as long as clvmd is running and the cluster is quorate.  This option is better suited for clusters utilizing GFS or GFS2 filesystems, since they are often mounted on multiple nodes at the same time (and thus require that the logical volume be active on each). This is also applicable for other situations where the data on the shared volumes is required on multiple nodes and is made available through some cluster-aware mechanism.

NOTE: Although clvmd allows for the same volume group to be activated on multiple systems concurrently, this does not make it possible to mount a non-clustered filesystem such as ext2, ext3, ext4, XFS, or others that are not cluster-aware on multiple nodes at the same time.  Doing so will likely cause corruption of the filesystem and data.  Proper precautions should be taken to ensure no two nodes mount the same non-clustered filesystem at once.

To configure clvmd, first ensure the lvm2-cluster package is installed.  Next the lvm configuration file /etc/lvm/lvm.conf must be modified to tell it to allow for the usage of clustered volume groups.  Change the locking_type option to 3 with the following command:

# lvmconf --enable-cluster

Or manually in /etc/lvm/lvm.conf:

locking_type=3

In RHEL 7 with pacemaker, the clvmd daemon can be configured to start via highly available resources.

In RHEL 6 and 5, the clvmd daemon can be enabled to start on boot and be started manually if the cluster is quorate:

# chkconfig clvmd on
# service clvmd start

Any volume groups with the clustered flag set will be activated when clvmd starts in RHEL 5 and 6, or when an LVM resource activates the volume group in RHEL 7.  To determine if a volume group has the clustered flag set, use vgdisplay:

# vgdisplay myVG | grep Clustered
  Clustered             yes

To set this flag while creating a volume group:

# vgcreate -cy <volume group name> <device path>

Or to add the clustered flag to an existing volume group:

# vgchange -cy <volume group name>

NOTE: Any volume group with the clustered flag set will not be available until the clvmd daemon is started and the cluster is quorate.

All nodes in the cluster must have the same view of all storage devices contained within clustered volume groups.  For instance if a new device is presented to one node and added to a volume group, errors may be thrown if that same device is not presented to the other nodes.  Or if a device is partitioned on one node and then added to a volume group, errors may be thrown if the other nodes have not re-read the partition table by running

# partprobe
HA LVM for Active/Passive Workloads Only Needing Exclusive Access to Shared Storage

The other option for managing LVM in a cluster is referred to as Highly Available LVM (HA-LVM), which effectively means activating LVM volumes on the appropriate nodes via a resource controlled by the resource manager.  This configuration only allows one cluster node to have a specific volume group or logical volume activated at a time when using rgmanager, whereas multiple nodes can have volumes activated with HA-LVM in pacemaker via the LVM resource.  More information on configuring HA LVM can be found in What is a Highly Available LVM (HA-LVM) configuration and how do I implement it?.  This option is better suited for clusters utilizing non-clustered filesystems like ext2, ext3, ext4, XFS, and similar, because it prevents multiple nodes from mounting the same filesystem at once.

SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.