Do I need an LVM resource managed by the cluster resource manager if I'm using clustered LVM (clvmd) in a RHEL 5, 6, or 7 Resilient Storage cluster?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 5, 6, or 7 with the Resilient Storage Add On
  • lvm2-cluster
    • locking_type = 3 in /etc/lvm/lvm.conf
    • One or more volume groups with the 'c' (clustered) attribute enabled

Issue

  • I'm using clvmd to manage my shared LVM volumes. I don't need an lvm resource in my cluster service, right?
  • I have clustered LVM yet the file system on one of the shared volumes was corrupted when a second node mounted it. How do I prevent this?
  • Does clvmd protect my devices from being mounted on multiple nodes simultaneously?
  • Is there any reason to use an LVM resource if I already have clvm?

Resolution

The decision of whether to configure an LVM resource to be managed by the resource manager in a cluster should be based on two factors:

  1. Will the volume(s) in question be accessed by a single node at a time, or concurrently by multiple nodes?
  2. Will other resources managed by the resource manager access the volume(s) in question?

If the answer to either question is "Yes", then it is recommended that an LVM resource be created to manage the shared volumes, with one exception in the case of GFS/GFS2 usage.

Active/Passive: Accessing LVM Volumes From One Node At a Time


When the shared LVM volumes only need access from one node at a time, such as when using a single-node file system like ext3, ext4, XFS, etc, an LVM resource managed by the resource manager (`pacemaker` or `rgmanager`) is strongly recommended, even if `clvmd` is in use.

Active/Active: Accessing LVM Volumes from Multiple Nodes Simultaneously


When the shared LVM volumes need to be active on multiple nodes, the configuration can vary depending on the style of cluster, as well as the use case.
LVM devices for GFS/GFS2 File Systems in RHEL 5 and 6 cman-based Clusters

In `cman`-based clusters, including those that use `rgmanager` and those that use `pacemaker` for resource management, the use of LVM resources is a matter of preference, with no strong requirements either way.

With rgmanager, it is not possible to have a volume activated on multiple nodes concurrently and use an <lvm> resource, so if the GFS or GFS2 file system must be accessed by multiple nodes at the same time, simply skip creating an <lvm> resource. If using a GFS or GFS2 that is managed by a <clusterfs> resource and is only mounted on a single node at a time, then an <lvm> resource in the relevant <service> can add monitoring for that resource and ensure the volume always gets activated properly on the proper node, so this type of configuration may be preferred in setups such as this.

If using pacemaker, LVM resources can be created as clones for the purpose of having the resource manager manage and monitor that volume throughout the cluster. If the additional complexity of doing so its not desired, then

LVM devices for GFS/GFS2 File Systems in RHEL 7 corosync+pacemaker-based Clusters

When using `pacemaker` in RHEL 7, it is required that `LVM` resources be configured in the CIB to activate and monitor any shared LVM volume groups. The `DLM` and `clvmd` daemons must be managed by the cluster resource manager anyways, and the additional mechanisms like the `gfs` or `gfs2` init service and the `clvmd` auto-activation of volumes are gone in this release, so there is no benefit to leaving the volume groups out of the resource configuration.

For volume groups that will be accessed concurrently on multiple nodes by GFS/GFS2 file systems, simply create them as a clone with exclusive=false. For volume groups that only need to be activated on a single node, create a normal resource with exclusive=true.

Root Cause

The RHEL High Availability and Resilient Storage Add Ons come with resource agents to have shared LVM volumes activated and monitored by the resource manager, which would be rgmanager or pacemaker, depending on how the cluster was set up and what RHEL release is in use. However, it is not always clear whether an LVM resource should be created, especially when clvmd is in use.

A common misconception is that clvmd alone is enough to protect data on shared LVM volumes from corruption by being accessed by multiple nodes simultaneously. This is false. clvmd is in charge of managing access to and communicating changes to shared LVM volume metadata across nodes in a cluster; however, it does not do anything automatically to prevent a shared volume from being activated on two nodes at once and the data on that volume being accessed at the same time. So for example, if a clustered volume group contains a logical volume with an ext4 file system, it would be possible for an administrator to activate that logical volume on node 1 and 2 of the same cluster and mount that same file system on both nodes. Since ext4 is not cluster-aware, both nodes would likely conflict in their access to the file system metadata causing the fs to be quickly corrupted. So, if there is an expectation that only one node be able to access data on a shared volume at once, some further mechanism is needed in addition to clvmd.

This is where the LVM resource agents come in. Even if using clvmd, it may be necessary or beneficial to have something limiting access to the data on shared volumes to a single active node at a time. The ocf:heartbeat:LVM resource can do this for pacemaker clusters, whereas the lvm resource can do this for rgmanager clusters.

SBR
Product(s)
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.