The tool fsck.gfs2 was run against the wrong device and destroyed the LVM metadata.

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 6 (with the Resilient Storage Add-on)
  • Red Hat Enterprise Linux 7 (with the Resilient Storage Add-on)
  • A Global Filesystem 2 (gfs2) filesystem

Issue

  • fsck.gfs2 was run against the LVM physical volume instead of against the logical volume, and this destroyed the LVM metadata.

Resolution

Try to restore the LVM metadata from the backup LVM files.

For example, assuming the volume group is called vgtestdevA with uuid rImA4M-670f-TFt5-03E6-rm45-xJMD-bZRIfK and the physical device is /dev/mapper/testdevA1. 

# pvcreate  --restorefile /etc/lvm/backup/vgtestdevA --uuid rImA4M-670f-TFt5-03E6-rm45-xJMD-bZRIfK /dev/mapper/testdevA1
# vgcfgrestore --file  /etc/lvm/backup/vgtestdevA vgtestdevA
# pvscan
# vgscan
# lvscan
# vgchange -ay vgtestdevA

If restoring the LVM backup fails then you will need to restore from backup or contact Red Hat support.

For more information on LVM debugging then review the following article: How do I debug an LVM problem?

Root Cause

The fsck.gfs2 was run against the physical LVM volume and not the logical LVM volume. This corrupted the filesystem by overwritten the LVM metadata. Once the LVM metadata was overwritten, the physical volume, volume group, and logical volume were no longer seen by LVM.

The tool fsck.gfs2 should only be run against the full path to the LVM logical volume.

Diagnostic Steps

If it is suspected that fsck.gfs2 has overwritten the LVM metadata then try performing the following to confirm this.

Verify that a physical volume, volume group, and logical volume are no longer seen by LVM by running:

  • pvscan
  • vgscan
  • lvscan
  • pvs; vgs; lvs

If physical volume, volume group, and logical volume are no longer seen by LVM then check the head of the disk for the strings for the LVM label and fsck_dlm. If the LVM label LABELONE is not found, but the fsck_dlm string is found the you hit this issue.

For example, if the physical device used by LVM is /dev/mapper/testdevA1 then grab the first 100 megabytes of the storage device and then search for the strings.

# dd if=/dev/mapper/testdevA1 of=/tmp/testdevA1.out bs=1M count=100 iflag=direct; hexdump -C /tmp/testdevA1.out  | grep -ie LABELONE -ie fsck 
00110060  66 73 63 6b 5f 64 6c 6d  00 00 00 00 00 00 00 00  |fsck_dlm........|
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.