LVM2 volume group in partial mode with physical volumes marked missing even though they are available in RHEL

Environment

• Red Hat Enterprise Linux (RHEL) 5 and above versions
• lvm2

Issue

• Couldn't find device with uuid aaaaaa-aaaa-aaaa-aaaa-aaaa-aaaa-aaaaaa and There are 1 physical volumes missing messages in LVM commands even when the PV for the mentioned UUID is seen available.

• Attempting to modify a volume group or logical volume fails due to missing devices that are not actually missing:

    # lvextend -l+100%PVS /dev/myvg/lv02 /dev/mapper/mpath80
      WARNING: Inconsistent metadata found for VG myvg - updating to use version 89
      Missing device /dev/mapper/mpath73 reappeared, updating metadata for VG myvg to version 89.
      Device still marked missing because of alocated data on it, remove volumes and consider vgreduce --removemissing.
  

• Any attempt to change a VG or LV claims PVs are missing:

      Cannot change VG myvg while PVs are missing.
      Consider vgreduce --removemissing.
  

• LVM physical volumes are marked with the missing (m) flag in pvs output even though they are healthy and available:

    # pvs
      PV          VG            Fmt  Attr PSize    PFree   DevSize
      /dev/dev_a  vg_name       lvm2 a--  1019.75g  30.88g 1019.75g
      /dev/dev_b  vg_name       lvm2 a-m   299.92g 189.92g  299.92g <===
  

• Volume group is marked as 'partial' and causes lvm commands to fail:

      VG      #PV #LV #SN Attr   VSize VFree  
      myvg     42  10   0 wz-pn- 2.31T 777.11G
  

Resolution

NOTE This resolution only applies to situations where a device is marked as missing, but is actually available, contains the expected metadata, and shows up in the output of lvm scans. If a device is legitimately missing, additional steps may be required.

Restore each missing physical volume with:

# vgextend --restoremissing <volume group> <physical volume>

# vgextend --restoremissing myvg /dev/mapper/mpath24
  Volume group "myvg" successfully extended

Root Cause

Once a device is marked as missing (such as if it cannot be found during a scan and the volume group is activated in "partial" mode), it may need to be manually restored. The vgextend --restoremissing command can do this.

Diagnostic Steps

Check the existing metadata in /etc/lvm/backup/vgname for the MISSING flag

pv2 {
                        id = "aaaaaa-aaaa-aaaa-aaaa-aaaa-aaaa-aaaaaa"
                        device = "/dev/mapper/mpathkp1" # Hint only

                        status = ["ALLOCATABLE"]
                        flags = ["MISSING"]
                        dev_size = 628976817    # 299.92 Gigabytes
                        pe_start = 2048
                        pe_count = 76779        # 299.918 Gigabytes
                }

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.