LVM-activate resource fails to start properly after storage expansion

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 8, 9 and 10 with High-Availability Add-On
  • lvm2-2.03.12-10.el8 with use_devicesfile enabled

Issue

This issue can occur after expanding the cluster's LVM storage in both "active/passive" clusters using a LVM based Filesystem resource as well as an "active/active" GFS2 cluster.

On an "active/passive" cluster, the issue would appear after running a pcs resource move or when the resource automatically fails over to the other node. The following error is seen:

Failed Resource Actions:
  * my_lvm start on fastvm-rhel-9-1-65 returned 'error' (Volume group[vg_shared] doesn't exist, or not visible on this node!) at Sat Nov 19 16:45:24 2022 after 234ms

On an "active/active" cluster, the issue would appear immediately as all active cluster nodes activate the shared LVM VG at the same time. The cloned LVM-activate resource on all other nodes would go into a stopped state and show the vg with missing devices, similiar to this :

Failed Resource Actions:
  * my_lv start on fastvm-rhel-9-1-65 returned 'error' (Volume group [data_vg] has devices missing.  Consider majority_pvs=true) at Tue Jul 15 01:47:05 2025 after 236ms

Resolution

On all nodes where it's failing to start, run the following command:

lvmdevices --adddev <PV of the shared VG>

Note: If there are more than one PV, repeat the command for all PVs holding the shared VG.

Root Cause

Support for the lvmdevices command set was made available in RHEL 8.5+ and later. This command set is used to limit logical volume manager access to only LV's attached to devices listed in the /etc/lvm/devices/system.devices file. If you are using an LVM devices file, and use_devicesfile is enabled in lvm.conf, additionally add the shared device to the devices file on the second node of the cluster.

The lvmdevices is enabled by default in RHEL 9 while in RHEL 8.5+ it is disabled by default. When lvmdevices is enabled, ensure that the shared devices are added to lvmdevices on the passive node as well else it will result in issue as described above.

There are a few commands which will implicitly update the LVM devices file. When defining a shared LVM device, these commands will be executed only on one node of the cluster and, therefore, the LVM devices file on the other nodes won't be updated unless manually running the lvmdevices commands from the Resolution section.

Diagnostic Steps

Compare the output of the following command between all cluster nodes. The PV(s) holding the shared VG(s) (/dev/sdb in this example) should be visible in the output of lvmdevices on all nodes. If it's not, the error described in the Issue section is expected.

[root@fastvm-rhel-9-1-64 ~]# lvmdevices
  Device /dev/sda2 IDTYPE=devname IDNAME=/dev/sda2 DEVNAME=/dev/sda2 PVID=Q4rrQQuSfiljYYcQDkDeswrn4QdcgyUe PART=2
  Device /dev/sdb IDTYPE=sys_wwid IDNAME=naa.60014050b82be98b3814747ace266def DEVNAME=/dev/sdb PVID=CPzgFdL3lK7aXOmMJFiPeZEgVzox0FhP

[root@fastvm-rhel-9-1-65 ~]# lvmdevices
  Device /dev/sda2 IDTYPE=devname IDNAME=/dev/sda2 DEVNAME=/dev/sda2 PVID=Q4rrQQuSfiljYYcQDkDeswrn4QdcgyUe PART=2

The pvs command may also indicate the cluster's shared VG has missing PVs 

$ pvs 
WARNING: VG data_vg is missing PV 12345-1234-abcd-wxyz-12ab-34cd-567efg (last written to /dev/sdd).
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.