Device missing, status is "unknown", after Physical Volume partition table was modified

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL)
    • Partition(s) created on Physical device.
      • Logical Volume Manager (PV is on physical device partition)
      • Device-mapper-multipath with partitions created on multipath devices.

Issue

  • LVM volumes disappear after reboot. Syslog messages show unknown partition table.

    hostname kernel: sdX: unknown partition table

  • LVM reports a device is missing ("unknown" status)

    # pvs
  Couldn't find device with uuid aaaaaa-bbbb-cccc-dddd-eeee-ffff-gggggg. <----+
  PV             VG         Fmt  Attr PSize PFree                             |
  /dev/sda1      VolGroup01 lvm2 a--  4.99G 4.99G                             |
  unknown device VolGroup01 lvm2 a-m  5.00G 5.00G                             |
  ---+---                                                                     |
     |                                                                        |
     +----------- device with specfied UUID not found on any device scanned --+

  • The previous disk with this UUID appears to be present and available on the system

    # cat /proc/partitions | grep sdX
 253    16   10485760 sdX
# ls -l /dev/sdX
brw-r----- 1 root disk 253, 16 Mar 11 13:54 /dev/sdX

  • The device may have recently had a partition added or removed within the existing partition table

  • Few disks have lost partitions within the partition table after reboot

Resolution

Disclaimer: The steps described below are a data recovery attempt on a device that has been, at least partially, overwritten. Such an operation is outside the scope of Red Hat Technical Support. These steps are provided as a courtesy, but cannot be guaranteed to be successful. Moreover, the procedure described below includes steps that will rewrite information on parts of the disks. This can make any further data recovery attempt more difficult or even impossible. If the incorrectly-made partition, that blocks LVM from discovering the volumes, has been used to create filesystems and store data, it is most likely that the data previously in the LVM volume will have become corrupted and unrecoverable.

Red Hat strongly recommends making a working block-level backup of the disk before proceeding. The backup can be used for data recovery by specialized companies in case the procedure is unsuccessful. Red Hat doesn't provide professional data recovery services.

Situation A: LVM metadata found on the beginning of the drive

  1. Check if the metadata is still available on the RAW device by looking for the LABELONE LVM marker:
# hexdump -C /dev/sdX | less
*
00000200  4c 41 42 45 4c 4f 4e 45  01 00 00 00 00 00 00 00  |LABELONE........| <-----here
00000210  aa 70 e9 75 20 00 00 00  4c 56 4d 32 20 30 30 31  |.p.u ...LVM2 001|
00000220  61 61 61 61 61 61 62 62  62 62 63 63 63 63 64 64  |aaaaaabbbbccccdd|
00000230  64 64 65 65 65 65 66 66  66 66 67 67 67 67 67 67  |ddeeeeffffgggggg|
00000240  00 00 40 06 00 00 00 00  00 00 10 00 00 00 00 00  |..@.............|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 00 00  00 10 00 00 00 00 00 00  |................|
00000270  00 f0 02 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000280  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
  1. Refer to article Why are my LVM volumes missing after creating a partition table? for further troubleshooting.

Situation B: LVM metadata found further into the drive

  1. Check if the metadata is still available on the device by looking for the LABELONE:
# hexdump -C /dev/sdX | less
*
00008000  4c 41 42 45 4c 4f 4e 45  01 00 00 00 00 00 00 00  |LABELONE........| <-----here
00008010  aa 70 e9 75 20 00 00 00  4c 56 4d 32 20 30 30 31  |.p.u ...LVM2 001|
00008020  61 61 61 61 61 61 62 62  62 62 63 63 63 63 64 64  |aaaaaabbbbccccdd|
00008030  64 64 65 65 65 65 66 66  66 66 67 67 67 67 67 67  |ddeeeeffffgggggg|
00008040  00 00 40 06 00 00 00 00  00 00 10 00 00 00 00 00  |..@.............|
00008050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00008060  00 00 00 00 00 00 00 00  00 10 00 00 00 00 00 00  |................|
00008070  00 f0 02 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00008080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
  1. Note the HEX offset of the marker by adding 0x to the beginning of the number in the first column:
0x00008000
  1. Convert the HEX value to DECIMAL:
# echo $((0x00008000))
32768
  1. Note the number directly after the LABELONE marker. This value should be between 00 and 03 (default 01):
00008000  4c 41 42 45 4c 4f 4e 45  01 00 00 00 00 00 00 00  |LABELONE........|
                                     ^^---here
  1. Determine the starting sector on disk by dividing the number in step 3 by 512. Take this value and subtract the number from step 4. This gives you the sector number of the start of the device containing the LVM metadata:
# echo $((( 32768 / 512 ) - 01 ))
63

- If the sector number you calculated is `0`, the Physical Volume was created on a raw device with no partition table and any partitions on the disk should be removed:


  ```
  # parted /dev/sdX rm 1
  # parted /dev/sdX rm 2
  # parted /dev/sdX rm 3
  # parted /dev/sdX rm 4
  ```


- If the sector number you calculated is `non-zero`, the partition needs to start at this sector. If it does not exist, create it using the starting sector from *step 4*:


  ```
  # parted /dev/sdX mkpart primary 63s 100%
  ```
  1. After the partition modifications, ensure LVM can now see the metadata:
# pvs
  PV         VG         Fmt  Attr PSize PFree
  /dev/sda1  VolGroup01 lvm2 a--  4.99G 4.99G 
  /dev/sdX1  VolGroup01 lvm2 a--  5.00G 5.00G

Root Cause

  • LVM does not check partitioned disk for metadata at the raw device level:

    # pvscan -vvv
      /dev/sdX: size is 20971520 sectors
        /dev/sdX: block size is 4096 bytes
        /dev/sdX: Skipping: Partition table signature found
        Closed /dev/sdX

  • If partition starting sectors are modified, LVM may look in the incorrect location for the metadata.

  • Modifying the partition table only adjusts a small section of the disk reserved for the table. Partitions can be deleted and recreated without data loss.

  • If the incorrectly-made partition has been used to create filesystems and store data, it is very likely that the data previously in the LVM volume will have become corrupted and unrecoverable.

Diagnostic Steps

Check the backup LVM metadata under /etc/lvm/backup to verify the correct PVs:

# grep device /etc/lvm/backup/VolGroup01 
			device = "/dev/sda1"	# Hint only
			device = "/dev/sdX1"	# Hint only

Ensure the device is available on the system:

# ls -l /dev/sdX1
ls: /dev/sdX1: No such file or directory
# ls -l /dev/sdX
brw-r----- 1 root disk 253, 16 Mar 11 14:37 /dev/sdX

Check the partition table output for the missing partition

# parted /dev/sdX u s p
Model: Unknown (unknown)
Disk /dev/sdX: 20971519s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start  End  Size  Type  File system  Flags
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.