How to repair metadata on physical volume online?

Environment

• Red Hat Enterprise Linux 8.2
  • lvm2-2.03.09-5.el8 or higher.

Issue

• Need to repair metadata on physical volume online which is corrupted or zeroed.
• Unable to recreate a PV label on an exiting device.

# pvcreate --uuid NgIQIH-irp3-ncrC-yNYg-hTsH-LF1c-6iUd7E --restorefile /etc/lvm/backup/testvg /dev/vdb -ff
  WARNING: Couldn't find device with uuid NgIQIH-irp3-ncrC-yNYg-hTsH-LF1c-6iUd7E.
  Can't open /dev/vdb exclusively.  Mounted filesystem?
  Can't open /dev/vdb exclusively.  Mounted filesystem?

Resolution

Warning These instructions should be used with extreme caution, and only if you are familiar with the implications of each command, the current layout of the volumes, the layout that you need to achieve, and the contents of the backup metadata file. These commands have the potential to corrupt data, and as such, it is recommended that you contact Red Hat Global Support Services for assistance in troubleshooting.

This procedure will NOT restore any data lost from a physical volume that has failed and been replaced. If a physical volume has been partially overwritten (for example, the label or metadata regions have been damaged or destroyed) then user data may still exist in the data area of the volume and this may be recovered using standard tools after restoring access to the volume group using these steps.

• Repair metadata on Physical volume using pvck --repair.

# pvck --repair -f /<path_to_valid_metadata> /path_to_PV  --settings "pv_uuid=<PV_UUID>"

• Then update Volume Group metadata.

 #vgck --updatemetadata VGNAME 

Diagnostic Steps

• pvcreate is failing on in-use device.

# pvcreate --uuid NgIQIH-irp3-ncrC-yNYg-hTsH-LF1c-6iUd7E --restorefile /etc/lvm/backup/testvg /dev/vdb -ff
  WARNING: Couldn't find device with uuid NgIQIH-irp3-ncrC-yNYg-hTsH-LF1c-6iUd7E.
  Can't open /dev/vdb exclusively.  Mounted filesystem?
  Can't open /dev/vdb exclusively.  Mounted filesystem?

• Find the the lvm metadata file and restore it using pvck. pvck can be used to repair a busy device as well.

# pvck --repair -f /etc/lvm/backup/testvg /dev/vdb 
  CHECK: label_header.id expected LABELONE
  CHECK: label_header.sector expected 1
  CHECK: label_header.crc expected 0xe7fdfc98
  CHECK: label_header.offset expected 32
  CHECK: label_header.type expected LVM2 001
  WARNING: No LVM label found on /dev/vdb.  It may not be an LVM device.
  Writing label_header.crc 0xcd9d4ffb pv_header uuid NgIQIHirp3ncrCyNYghTsHLF1c6iUd7E device_size 2147483648
  Writing data_offset 1048576 mda1_offset 4096 mda1_size 1044480 mda2_offset 0 mda2_size 0
Write new LVM header to /dev/vdb? y
  Writing metadata at 4608 length 1113 crc 0xd292052c mda1
  Writing mda_header at 4096 mda1
Write new LVM metadata to /dev/vdb? y

• Refer to Extracting LVM metadata backups from a physical volume in case the metadata backup file is not available.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.