RHEL8: Cannot activate LVs in VG <vg_name> while PVs appear on duplicate devices

Environment

• Red Hat Enterprise Linux 8
• LVM2
• Multiple paths to storage device(s)
• Disk level snapshots VMware

Issue

• Cannot activate the volume group.

[root@testbox lvm]# vgchange -ay testvg
  WARNING: Not using device /dev/sde1 for PV pv_uuid.
  WARNING: Not using device /dev/sdf1 for PV pv_uuid.
  WARNING: Not using device /dev/sdg1 for PV pv_uuid.
  Cannot activate LVs in VG testvg while PVs appear on duplicate devices.  <<<<
  0 logical volume(s) in volume group "testvg" now active  <<<<

• Why do I get duplicate PV warnings when I run LVM commands?

• Why we do see seqno conflicts Warnings in pvs ouptut

WARNING: ignoring metadata seqno X on /dev/sdY for seqno Y on /dev/sdZ for VG VG_NAME.
WARNING: Inconsistent metadata found for VG VG_NAME.
See vgck --updatemetadata to correct inconsistency.
WARNING: VG VG_NAME was previously updated while PV /dev/sdY was missing.

Resolution

• Configure the multipath and set the LVM filter to fix the issue.
  Note on Transient Backup/Snapshot Devices:

• If these warnings occur during or immediately after virtual machine backup operations (e.g., using Veritas NetBackup or Veeam), this is likely caused by temporary backup snapshot disks containing duplicate copies of your volume group physical volumes.
• Once the backup finishes and the snapshot is successfully detached, the duplicate devices will disappear. If the backup software leaves stale snapshot devices attached, rebooting the VM or manually detaching the disks will clean up the duplicates and restore consistent metadata automatically without requiring vgck --updatemetadata.

Root Cause

The allow_changes_with_duplicate_pvs config option within lvm.conf is set to boolean value 0 in RHEL 8. This change disallows a VG to activate/change while duplicate PV's are detected to protect from metadata corruption.

	# Configuration option devices/allow_changes_with_duplicate_pvs.
	# Allow VG modification while a PV appears on multiple devices.
	# When a PV appears on multiple devices, LVM attempts to choose the
	# best device to use for the PV. If the devices represent the same
	# underlying storage, the choice has minimal consequence. If the
	# devices represent different underlying storage, the wrong choice
	# can result in data loss if the VG is modified. Disabling this
	# setting is the safest option because it prevents modifying a VG
	# or activating LVs in it while a PV appears on multiple devices.
	# Enabling this setting allows the VG to be used as usual even with
	# uncertain devices.
	allow_changes_with_duplicate_pvs = 0

Diagnostic Steps

• Messages logs are pilled up with duplicate PV messages:

Nov 25 18:52:33 testbox lvm[1371]:  WARNING: PV pv_uuid prefers device /dev/sdc because device was seen first.
Nov 25 18:52:33 testbox lvm[1717]:  Cannot activate LVs in VG <vg_name> while PVs appear on duplicate devices.
Nov 25 18:52:33 testbox lvm[1717]:  Cannot activate LVs in VG  <vg_name> while PVs appear on duplicate devices.

• Verify allow_changes_with_duplicate_pvs value in lvm.conf file:

 # cat etc/lvm/lvm.conf| grep allow_changes_with_duplicate_pvs| egrep  -v "#|^$"
   allow_changes_with_duplicate_pvs = 0

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.