How do I manually activate a volume group managed by an LVM-activate resource with system_id on another node?
Environment
- Red Hat Enterprise Linux (RHEL) 8 and 9 with the High Availability Add-on
LVM-activateresource withvg_access_mode=system_id
Issue
-
The active node of the cluster is down, and I cannot activate the cluster controlled volume group on the passive node.
Cannot access VG testvg with system ID rh8-nd2 with local system ID rh8-nd1.
Resolution
Please Note: This should be performed during a maintenance window as the cluster with all resources will have to be stopped.
Prior to the following steps to correct the system ID configuration, confirm /etc/lvm/lvm.conf has auto_activation_volume_list value configured with only local volume groups. If the shared volume group is listed, it will cause issues and prevent clean system ID switches.
Assuming following is the cluster architecture
- Cluster node hostnames are
rh8-nd1andrh8-nd2. - Local VG to both the cluster servers are named
rhel. - Cluster controlled volume group is named
testvg. - The
system_id_sourceparameter under/etc/lvm/lvm.confis set asuname. - Node
rh8-nd2has thetestvgactive on it currently.
Method 1: Update systemid From Any Node in the Cluster
-
Stop the cluster to perform troubleshooting.
pcs cluster stop --all -
Use
vgs -ao+systemidcommand to confirmtestvgis currently only showing active on noderh8-nd2with node hostname listed as current system ID. Noderh8-nd1should not listtestvgor not list a system ID for the volume group.[root@rh8-nd2 ~]# vgs -ao+systemid VG #PV #LV #SN Attr VSize VFree System ID rhel 1 2 0 wz--n- <11.00g 0 testvg 2 1 0 wz--n- <1.20g 0 rh8-nd2 [root@rh8-nd1 ~]# vgs -ao+systemid VG #PV #LV #SN Attr VSize VFree System ID rhel 1 2 0 wz--n- <11.00g 0 -
Activate the VG on passive node
rh8-nd1by temporarily setting the system_id asrh8-nd2withvgchange -y --config local/extra_system_ids=[\"rh8-nd2\"] --systemid rh8-nd1 testvgcommand so it can be interacted with onrh8-nd1. This will allowrh8-nd1to update the system ID independently torh8-nd1.[root@rh8-nd1 ~]# vgchange -y --config local/extra_system_ids=[\"rh8-nd2\"] --systemid rh8-nd1 testvg Volume group "testvg" successfully changed -
Confirm the system ID has successfully updated to
rh8-nd1.[root@rh8-nd1 ~]# vgs -ao+systemid VG #PV #LV #SN Attr VSize VFree System ID rhel 1 2 0 wz--n- <11.00g 0 testvg 2 1 0 wz--n- <1.20g 0 rh8-nd1 <<===---- System ID is now updated to passive node -
Activate the logical volume for
testvgonrh8-nd1.[root@rh8-nd1 ~]# vgchange -ay testvg 1 logical volume(s) in volume group "testvg" now active -
Confirm activation of logical volume with
lvscommand.[root@rh8-nd1 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root rhel -wi-ao---- 9.79g swap rhel -wi-ao---- 1.20g testlv testvg -wi-a----- <1.20g <<===---- LV is now active on passive node and available to get mounted locally on node. -
Confirm the filesystem related to
testvgis not mounted locally on any cluster nodes withmount. -
Confirm the
testvgVG/testlvLV is deactivated on both nodes withvgchange -a n testvgthe confirm withlvson all cluster nodes.[root@rh8-nd1 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root rhel -wi-ao---- 9.79g swap rhel -wi-ao---- 1.20g testlv testvg -wi------- <1.20g <<===---- LV is now deactivated on passive node. -
Start cluster to bring all resources up.
pcs cluster start --all
Method 2: Update systemid From Node Matching Assigned systemid
-
Stop the cluster to perform troubleshooting.
pcs cluster stop --all -
Remove the current systemID from volume group. This must be ran from node with matching "systemid".
vgchange --systemid "" <vgname> -
Add the systemID of preferred node to the volume group.
vgchange --systemid "<hostname>" <vgname> -
Start cluster to bring all resources up.
pcs cluster start --all
Diagnostic Steps
-
Command
vgs -ao+systemidshows different system ID information for the shared cluster volume group.[root@rh8-nd2 ~]# vgs -ao+systemid VG #PV #LV #SN Attr VSize VFree System ID rhel 1 2 0 wz--n- <11.00g 0 testvg 2 1 0 wz--n- <1.20g 0 rh8-nd2 [root@rh8-nd1 ~]# vgs -ao+systemid VG #PV #LV #SN Attr VSize VFree System ID rhel 1 2 0 wz--n- <11.00g 0
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.