Why are my LVM volumes missing, status shown as "unknown", after creating a partition table?

Environment

• Red Hat Enterprise Linux (RHEL)
• LVM2 Logical Volume Manager (PV is located on whole physical device)

Issue

• Why are my LVM volumes missing after creating a partition table?

• My LVM Physical Volumes (PVs), Volume Groups (VGs), and Logical Volumes (LVs) cannot be found.

• One physical volume /dev/sdX is listed as "unknown device" even though it appears in /proc/partions:

  # lvs -a -o +devices
    Couldn't find device with uuid aaaaaa-bbbb-cccc-dddd-eeee-ffff-gggggg.
    LV                 VG        Attr   LSize  Origin Snap%  Move Log Copy%  Convert Devices             
    lv_root            localvg00 -wi-ao  4.00G                                       /dev/sda2(512)      
    lv_data            localvg00 -wi-ao 40.00G                                       unknown device(0)    <----
  
  # grep sdX /proc/partitions 
     8    16   49283072 sdX
     8    17   49283040 sdX1
  

• I get the following error when trying to restore LVM metadata on a Physical Volume:

  Can't open /dev/sdX exclusively. Mounted filesystem?
  

Resolution

Disclaimer: The steps described below are a data recovery attempt on a device that has been, at least partially, overwritten. Such an operation is outside the scope of Red Hat Technical Support. These steps are provided as a courtesy, but cannot be guaranteed to be successful. Moreover, the procedure described below includes steps that will rewrite information on parts of the disks. This can make any further data recovery attempt more difficult or even impossible. If the incorrectly-made partition, that blocks LVM from discovering the volumes, has been used to create filesystems and store data, it is most likely that the data previously in the LVM volume will have become corrupted and unrecoverable.

Red Hat strongly recommends making a working block-level backup of the disk before proceeding. The backup can be used for data recovery by specialized companies in case the procedure is unsuccessful. Red Hat doesn't provide professional data recovery services.

1. Remove any partitions added by mistake to the devices which used to contain the Physical Volumes that are currently missing:

• Make sure that the new partition is not mounted, used as swap, or mirrored from a currently in-use device.

• If device is a multipath device, then please refer to solution How to delete partition created on device-mapper multipath device?.

  # parted /dev/sdX rm 1
  # parted /dev/sdX rm 2
  # parted /dev/sdX rm 3
  # parted /dev/sdX rm 4
  

2. Wipe the partition LABEL from disk.

   • For dos labeled disk use below command.

       # wipefs -o  0x1fe /dev/sdX
     

   • For gpt labeled disk refer to "How to remove a GPT label"

3. Re-scan the LVM devices, and try activating the restored volumes:

   # pvscan
   # vgscan
   # lvscan
   # vgchange -ay localvg00
   

If step #3 succeeds in activating the VG step 4 is not required and can be skipped.

4. If step 3 fails to activate the VG due to missing PVs, then this is an indication that the partition was written into or the creation of the partition table itself overwrote the PV metadata. You may need to restore the LVM metadata from the backup as described in the following documentation

   • RHEL 9 - LVM Admin: Restoring metadata on an LVM physical volume
   • RHEL 8 - LVM Admin: Restoring metadata on an LVM physical volume
   • RHEL 7 - LVM Admin: Recovering Physical Volume Metadata
   • RHEL 6 - LVM Admin: Recovering Physical Volume Metadata
   • RHEL 5 - LVM Admin: Recovering Physical Volume Metadata

5. Run a filesystem check on each Logical Volume to verify filesystem integrity, and mount the filesystems.

Root Cause

• The LVM metadata is placed at a specific offset from the start of the disk. Adding a partition table when LVM was already using the whole disk can break the logic that finds the LVM metadata. In addition, creating a partition table can end up overwriting the LVM metadata in certain cases. Creating a filesystem on a partition that occupies the first part of the disk pretty much guarantees that all data has been lost and is irrecoverable.

  • The presence of an MBR (dos) or GPT partition table will stop LVM from scanning the block device (e.g. /dev/sda) for LVM volumes.
  • If there are no partitions yet defined, then no partitions are scanned either.
  • If partitions were created, the partitions will be searched. If the LVM metadata survives on disk and has not yet been overwritten then it is now in the wrong place/offset within a partition from where it is expected preventing PV discovery.
  • If the incorrectly-made partition has been used to create filesystems and store data, it is most likely that the data previously in the LVM volume will have become corrupted and unrecoverable.

• Before:

  /dev/sdN
  0   N              N+x                                              End-of-Disk
  +---+--------------+----------------------------------- ... -------------+
  |   : LVM Metadata | PV data container                                   |
  +---+--------------+-----------------------------------------------------+
               
  # hexdump -n 65356 -C /dev/mapper/lvmtest01 | egrep "LABELONE|LVM2" -A 4
  00000200  4c 41 42 45 4c 4f 4e 45  01 00 00 00 00 00 00 00  |LABELONE........|
  00000210  c8 a8 c5 a7 20 00 00 00  4c 56 4d 32 20 30 30 31  |.... ...LVM2 001|
  00000220  53 53 6c 4f 58 52 48 75  7a 6b 44 59 75 41 64 31  |SSlOXRHuzkDYuAd1|
  00000230  68 48 32 59 43 43 6e 4b  4d 68 43 73 4d 38 61 49  |hH2YCCnKMhCsM8aI|
  00000240  00 00 00 00 05 00 00 00  00 00 10 00 00 00 00 00  |................|
  00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
  --
  00001000  16 d6 8e db 20 4c 56 4d  32 20 78 5b 35 41 25 72  |.... LVM2 x[5A%r|
  00001010  30 4e 2a 3e 01 00 00 00  00 10 00 00 00 00 00 00  |0N*>............|
  00001020  00 f0 0f 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
  00001030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
  *
  #
  

• After creating GPT (or MBR) partition table: loss of LVM metadata.

  /dev/sdN
  0   N              N+x                                              End-of-Disk
  +---+--------------+----------------------------------- ... ---+---------+
  |MBR|GPT data      | PV data container                         |GPT data |
  +---+--------------+-------------------------------------------+---------+
  :<- lost data ---->:                                           :<-lost-->:
               
  # hexdump -n 65356 -C /dev/mapper/lvmtest01 | egrep "LABELONE|LVM2" -A 4
  #
  

• After creating partitions and using partitions (typ. irrecoverable loss of PV container data).

  /dev/sdN
  0   N              N+x                                              End-of-Disk
  +---+--------------+----------------------------------- ... ---+---------+
  |MBR|GPT data      : new filesystem data                       |GPT data |
  +---+--------------+-------------------------------------------+---------+
  :<- lost data ---------------------------------------------------------->:
               
  # hexdump -n 65356 -C /dev/mapper/lvmtest01 | egrep "LABELONE|LVM2" -A 4
  #
  

Diagnostic Steps

• Inspect first megabytes of the block device:

# hexdump -C /dev/sdX |less

• Use hexdump or od to view the partition table and LVM metadata data on the disk

• The below example shows where a partition has been added and the LVM metadata is still present:

# hexdump -C /dev/sdX |less
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|  \
*                                                                                \
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 01  |................|     Partition table
000001c0  01 00 83 fe ff ff 3f 00  00 00 9a e5 3f 01 00 00  |......?.....?...|     contained within
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|     first 512 bytes (0x200)
*                                                                                /
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|  /
00000200  4c 41 42 45 4c 4f 4e 45  01 00 00 00 00 00 00 00  |LABELONE........| <--- LVM Physical Volume label
00000210  aa 70 e9 75 20 00 00 00  4c 56 4d 32 20 30 30 31  |.p.u ...LVM2 001|
00000220  61 61 61 61 61 61 62 62  62 62 63 63 63 63 64 64  |aaaaaabbbbccccdd|  \
00000230  64 64 65 65 65 65 66 66  66 66 67 67 67 67 67 67  |ddeeeeffffgggggg|  /  LVM Physical Volume UUID
00000240  00 00 40 06 00 00 00 00  00 00 10 00 00 00 00 00  |..@.............|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 00 00  00 10 00 00 00 00 00 00  |................|
00000270  00 f0 02 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000280  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*

• If you see that PV is expected to be on partition (like /dev/sdX1, /dev/sdX2, ...), then please follow article Device missing after Physical Volume partition table was modified instead.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.