lvm commands are not finding or displaying the expected volumes in RHEL

Solution Verified - Updated

Environment

  • Red Hat Enteprise Linux (RHEL)
  • lvm2

Issue

  • lvm commands do not show my physical volume, volume group, or logical volume anymore.
  • When running pvscan "unknown device" is seen for a device that is being used as a physical volume in a running configuration.
      PV unknown device   VG VolGroup13      lvm2 [656.00 GB / 6.00 GB free]

  • lvm commands are reporting "volume group does not exist" even though it should exist.

  • After multipath config changes, OS cannot see the lvm volumes anymore.

  • Data was written to the first megabyte or head of all physical volume disks.

  • LVM metadata appears to have been lost from one of the disks of a 2 disk LVM.

  • After unpresenting storage devices from the system, certain volume groups are throwing "Couldn't find device" errors and will not activate:

    Couldn't find device with uuid paH2Cp-w92t-lIz2-WznV-ihDJ-dg8M-oFUpMa.
   There are 1 physical volumes missing.
 PV             VG     Fmt  Attr PSize  PFree
 /dev/sda2      rootvg lvm2 a-    9.69G    0 
 /dev/sdb1      appsvg lvm2 a-   79.97G    0 
 /dev/sdd1      rootvg lvm2 a-    4.97G 1.97G
 unknown device appsvg lvm2 a-   79.97G    0

  • Removed redundant multipath device, now the system is not showing any LVs.

Resolution

  • An "unknown device" or "Couldn't find device with UUID" error can be caused by a number of different factors, and the solution depends on the cause. See the diagnostic steps below for information on identifying the cause of this issue.
  • Problems of this nature are most often caused by an incorrect filter in /etc/lvm/lvm.conf. The default filter scans any and all devices in /dev, so if a problem is encountered with devices not being found, it is often helpful to revert to this default filter:
        filter = [ "a/.*/" ]

Now run the same commands again and see if the volume is found. If so, then a modification to the filter may be needed to properly scan the appropriate physical volumes. More details at: What is LVM's filter setting and how do I configure it easily in RHEL?

  • If the filter is not causing the issue, and the storage device is present on the system but metadata cannot be found on it, then it is possible the device does not contain the expected lvm label and metadata. This issue can be diagnosed further by inspecting the lvm metadata backup copies for that volume group in /etc/lvm/backup and /etc/lvm/archives. This can give clues as to where the device used to be, which can facilitate manual inspection of that device for metadata.

  • In some situations it may be necessary to restore older versions of the VG metadata with vgcfgrestore, or restore PV labels with pvcreate. However, these actions are often risky and require a deep understanding of LVM metadata layout and the current organization of physical volumes and volume groups. It is recommended that you contact Red Hat Global Support Services for assistance before undertaking any metadata changes or restores.

  • If the LVM volume goes missing after modifying a partition table in the device, LVM might be looking in the incorrect location for the metadata. In this case please review Device missing, status is "unknown", after Physical Volume partition table was modified. In case this happens after creating a partition table, please check Why are my LVM volumes missing, status shown as "unknown", after creating a partition table?.

  • Another potential cause could be a device belonging to the volume group becoming disconnected/unpresented from the system. It's necessary to locate the missing device and re-establish its connection to the system. More details at: logical volumes are in partial state due to missing physical volumes after disk migration.

  • If the missing/unknown device is a result of unpresenting a storage device from the system before reducing it out of the volume group, vgreduce can be used to remove it:

    • WARNING: This should only be used if there are no logical volumes mapped to the missing PV in the metadata. This command will update the Volume Group metadata, and will prevent any Logical Volume that maps to the missing Physical Volume(s) from being used.
    ###### Run as a test first, to check for errors
# vgreduce --test --removemissing <volume group>
# vgreduce --removemissing <volume group>

Diagnostic Steps

  • In RHEL 4 and RHEL 5, the default metadata area size is 192Kb, and in RHEL 6 it is 1Mb (this is configurable in all releases). It is often useful to copy off this portion of the device in question and inspect it with tools like strings and less. For example:

    • RHEL 6

      # dd if=/dev/mapper/mpathap1 of=/tmp/mpathap1.img bs=1M count=1

    • RHEL 5

      # dd if=/dev/mapper/mpath1p1 of=/tmp/mpath1p1.img bs=1K count=192

  • Now this file can be provided to Red Hat Global Support Services, or manually inspected:

    # strings /tmp/mpathap1.img 
LABELONE
LVM2 001USuyPIJW99Oizpbb6Q66ZVbsfrCgJyzP
LVM2 x[5A%r0N*>
test {
id = "sc9uko-G0Q8-AiXm-o5ei-m1WV-QKvI-6e1mB3"
seqno = 1

  • The string LABELONE is the marker lvm2 uses to denote a device label, so if this string is not found, then lvm2 can not use the device as a physical volume. This may indicate that something or someone overwrote the label on that device, or this device is not the correct physical volume.

  • It is often useful to review the data in /etc/lvm/backup and /etc/lvm/archives for clues as to what changes may have been made to a volume group leading up to, or following, the onset of an issue. The state of the volume group should be recorded in a backup file in these directories before and after each command that changes the layout of the VG.

  • There are advanced methods for restoring volume group metadata to a device using a saved backup with the commands pvcreate and vgcfgrestore. However, there are a number of considerations when using these, as they have the potential to make the situation worse if not used properly. Please contact Red Hat Global Support Services for assistance in investigating the missing volumes and potentially restoring metadata if needed.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.