How do I restore a volume group in Red Hat Enterprise Linux if one of the physical volumes that constitute the volume group has failed?

Solution Verified - Updated 17 Nov 2025

Environment

Red Hat Enterprise Linux (RHEL) 4, 5, 6, 7, 8, 9, 10
LVM2

Issue

How to restore a volume group and its logical volumes after a disk (physical volume) that the logical volume was using has failed or the volume has become corrupted or was overwritten?
We have strange behavior on a RHEL 6.5 server. The SAN based filesystems (/dev/mapper/VolGroupXX*) are mounted, and they are readable and writable. But when I execute pvs ,vgs ,lvs ,pvscan ,vgscan and lvscan. I don't see any PVs, VGs or LVs related to VolGroupXX*. Only with multipath -ll and dmsetup ls --tree commands I see references to the VolGroupXX*.
unknown device error in pvs output but disks are showing in multipath

Resolution

Warning These instructions should be used with extreme caution, and only if you are familiar with the implications of each command, the current layout of the volumes, the layout that you need to achieve, and the contents of the backup metadata file. These commands have the potential to corrupt data, and as such, it is recommended that you contact Red Hat Global Support Services for assistance in troubleshooting.

This procedure will NOT restore any data lost from a physical volume that has failed and been replaced. If a physical volume has been partially overwritten (for example, the label or metadata regions have been damaged or destroyed) then user data may still exist in the data area of the volume and this may be recovered using standard tools after restoring access to the volume group using these steps.

This procedure requires a recent backup of the LVM configuration. This can be generated with the command vgcfgbackup and is stored in the file /etc/lvm/backup/<volume group name>. The /etc/lvm/archive directory also contains recent configurations that are created when modifications to the volume group metadata are made. It is recommended that these files be regularly backed up to a safe location so that they will be available if required for recovery purposes.

Assuming a physical volume has been lost that was a part of a volume group the following procedure may be followed. The procedure will replace the failed physical volume with a new disk rendering any remaining logical volumes accessible for recovery purposes.

The procedure for recovery is as follows:

Execute the following command to display information about the volume group in question:

vgdisplay --partial --verbose

The output will be similar to the following (note that the --partial flag is required to activate or manipulate a volume group having one or more physical volumes missing and that use of this flag with LVM2 activation commands (vgchange -a) will force volumes to be activated in a read-only state):

 Partial mode. Incomplete volume groups will be activated read-only.
 Finding all volume groups
 Finding volume group "volGroup00"
Couldn't find device with uuid 9eWicl-1HSB-Fkcz-wrMf-DzMd-Dgx2-Kyc11j'.
Couldn't find device with uuid 9eWicl-1HSB-Fkcz-wrMf-DzMd-Dgx2-Kyc11j'.
Couldn't find device with uuid 9eWicl-1HSB-Fkcz-wrMf-DzMd-Dgx2-Kyc11j'.
Couldn't find device with uuid 9eWicl-1HSB-Fkcz-wrMf-DzMd-Dgx2-Kyc11j'.
--- Volume group ---
VG Name               volGroup00
System ID
Format                lvm2
Metadata Areas        4
Metadata Sequence No  33
VG Access             read
VG Status             resizable
MAX LV                0
Cur LV                1
Open LV               0
Max PV                0
Cur PV                5
Act PV                5
VG Size               776.00 MB
PE Size               4.00 MB
Total PE              194
Alloc PE / Size       194 / 776.00 MB
Free  PE / Size       0 / 0
VG UUID               PjnqwZ-AYXR-BUyo-9VMN-uSRZ-AFlj-WOaA6z
--- Logical volume ---
LV Name                /dev/volGroup00/myLVM
VG Name                volGroup00
LV UUID                az6REi-mkt5-sDpS-4TyH-GBj2-cisD-olf6SW
LV Write Access        read/write
LV Status              available
# open                 0
LV Size                776.00 MB
Current LE             194
Segments               5
Allocation             inherit
Read ahead sectors     0
Block device           253:0
--- Physical volumes ---
PV Name               /dev/hda8
PV UUID               azYDV8-e2DT-oxGi-5S9Q-yVsM-dxoB-DgC4qN
PV Status             allocatable
Total PE / Free PE    48 / 0
PV Name               /dev/hda10
PV UUID               SWICqb-YIbb-g1MW-CY60-AkNQ-gNBu-GCMWOi
PV Status             allocatable
Total PE / Free PE    48 / 0
PV Name               /dev/hda11
PV UUID               pts536-Ycd5-kNHR-VMZY-jZRv-nTx1-XZFrYy
PV Status             allocatable
Total PE / Free PE    48 / 0
PV Name               /dev/hda14
PV UUID               OtIMPe-SZK4-arxr-jGlp-eiHY-2OA6-kyntME
PV Status             allocatable
Total PE / Free PE    25 / 0
PV Name               unknown device
PV UUID               9eWicl-1HSB-Fkcz-wrMf-DzMd-Dgx2-Kyc11j
PV Status             allocatable
Total PE / Free PE    25 / 0

Note the PV UUID line. This line contains the universally unique identifier (UUID) of the physical volume that failed and will be needed in the next step:

PV UUID               9eWicl-1HSB-Fkcz-wrMf-DzMd-Dgx2-Kyc11j

If the physical volume failed, it must be replaced with a disk or partition that is equal in size or larger than the failed volume. If the disk did not fail but was overwritten or corrupted, the same volume can be re-used. Run the following command to re-initialize the physical volume:
```
pvcreate --restorefile /etc/lvm/backup/<volume group name> --uuid <UUID> <device>
```
In the above command, the UUID is the value taken from the output in step 1. Command example:
```
# pvcreate --restorefile /etc/lvm/backup/volGroup00 --uuid 9eWicl-1HSB-Fkcz-wrMf-DzMd-Dgx2-Kyc11j /dev/hda15
Couldn't find device with uuid 9eWicl-1HSB-Fkcz-wrMf-DzMd-Dgx2-Kyc11j.
   Physical volume "/dev/hda15" successfully created
```
Note that when overwriting a previously-used LVM2 physical volume (for example when recovering from a situation where the volume had been inadvertently overwritten) the -ff option must be given to the pvcreate command.
Now the new physical volume has been initialized with the UUID of the old physical volume. The volume group metadata may be restored with the following command:
```
vgcfgrestore --file  /etc/lvm/backup/<volume group name> <volume group name>
```
Continuing the earlier example the exact command would be:
```
# vgcfgrestore --file /etc/lvm/backup/volGroup00 volGroup00
Restored volume group volGroup00
```
To check that the new physical volume is intact and the volume group is functioning correctly execute:
```
# vgdisplay -v
```

Root Cause

There are multiple causes for an unknown device PV. They often fall into the following categories:

Disk removed from the Volume Group
Corrupted LVM metadata
One or multiple disk(s) permanently removed from the system
Disk missing or damaged
Disk information was changed with some tool - i.e dd, fdisk, parted
At some point in time, but the change took effect after reboot.

SBR

Product(s)

Red Hat Enterprise Linux

Components

lvm2

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.