An LVM-activate resource failed to activate with the error: Cannot access VG shared_vg2 with system ID rhel8-1.examplerh.com with local 'system ID rhel8-2.examplerh.com.

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux Server 7, 8, 9 (with the High Availability Add On)

Issue

  • An LVM-activate resource failed to activate

      Apr  9 14:02:26 rhel8-2 LVM-activate(lvm_vg2)[161887]: INFO: Activating shared_vg2
  Apr  9 14:02:27 rhel8-2 LVM-activate(lvm_vg2)[161907]: ERROR:  WARNING: lvmlockd process is not running. Cannot access VG shared_vg2 with system ID rhel8-1.examplerh.com with local 'system ID rhel8-2.examplerh.com.
  Apr  9 14:02:27 rhel8-2 LVM-activate(lvm_vg2)[161917]: ERROR: shared_vg2: failed to activate.
  Apr  9 14:02:27 rhel8-2 pacemaker-controld[1466]: notice: Result of start operation for lvm_vg2 on rhel8-2.examplerh.com: error (shared_vg2: failed to activate.)
  Apr  9 14:02:27 rhel8-2 pacemaker-controld[1466]: notice: lvm_vg2_start_0@rhel8-2.examplerh.com output [   WARNING: lvmlockd process is not running.\n  Cannot access VG shared_vg2 with system ID rhel8-1.examplerh.com with local system ID rhel8-2.examplerh.com.\n  WARNING: lvmlockd process is not running.\n  WARNING: lvmlockd process is not running.\n  Global lock failed: check that lvmlockd is running.\n  WARNING: lvmlockd process is not running.\n  Cannot access VG shared_vg2 with system ID rhel8-1.examplerh.com with local system ID rhel8-2.examplerh.com.

Resolution

The pacemaker resource lvmlockd should only be used if there are shared (activation_mode=shared) LVM-activate resources managed by pacemaker. If there are no shared LVM-activate resources configured then our recommendation is not to use lvmlockd to manage activation of LVM volume groups and use the halvm method: How to configure HA-LVM Cluster using system_id in RHEL 8 and above?

If an lvmlockd resource is managed by pacemaker then all LVM-activate resources must include a constraint to start lvmlockd before starting an LVM-activate resource.

For example if we had a controld and lvmlockd resource in a cloned group called locking-clone and a resource group composed of a LVM-activate resource and a Filesystem resource called halvm_data. 

   # pcs resource create dlm ocf:pacemaker:controld op monitor interval=30s on-fail=fence --group locking 
   # pcs resource create lvmlockd ocf:heartbeat:lvmlockd op monitor interval=30s on-fail=fence --group locking 
   # pcs resource clone locking interleave=true
   # pcs resource create lvm_vg2 ocf:heartbeat:LVM-activate vgname=shared_vg2 vg_access_mode=system_id activation_mode=exclusive  --group halvm_data
   # pcs resource create data_fs ocf:heartbeat:Filesystem device=/dev/mapper/shared_vg2-lv1 directory=/mnt/data fstype=xfs --group halvm_data'
   # pcs constraint order start locking-clone then halvm_data

Root Cause

The cluster had an lvmlockd resource configured and use_lvmlockd=1 is set in /etc/lvm/lvm.conf. When they enabled the LVM-activate resource that was configured with activation_mode=exclusive for active/passive LVM volume group the LVM-activate resource failed to start.

The resource failed to start because the "system id" of the volume group could not be set because lvmlockd was not running. If use_lvmlockd=1 is set in /etc/lvm/lvm.conf then lvmlockd has to be running to modify the LVM volume group.

There was no constraint configured for the lvmlockd resource to start before starting the LVM-activate resource.

Diagnostic Steps

  • If lvmlockd is managed by pacemaker but is not running:

      # pcs status --full | grep lvmlockd

  • If use_lvmlockd=1 is set in /etc/lvm/lvm.conf and lvmlockd is not running:

      # grep use_lvmlockd /etc/lvm/lvm.conf
  # ps aux | grep lvmlockd
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.