How do I manually activate a volume group managed by an LVM-activate resource with system_id on another node?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 8 and 9 with the High Availability Add-on
  • LVM-activate resource with vg_access_mode=system_id

Issue

  • The active node of the cluster is down, and I cannot activate the cluster controlled volume group on the passive node.

    Cannot access VG testvg with system ID rh8-nd2 with local system ID rh8-nd1.
    

Resolution

Please Note: This should be performed during a maintenance window as the cluster with all resources will have to be stopped.

Prior to the following steps to correct the system ID configuration, confirm /etc/lvm/lvm.conf has auto_activation_volume_list value configured with only local volume groups. If the shared volume group is listed, it will cause issues and prevent clean system ID switches.

Assuming following is the cluster architecture

  • Cluster node hostnames are rh8-nd1 and rh8-nd2.
  • Local VG to both the cluster servers are named rhel.
  • Cluster controlled volume group is named testvg.
  • The system_id_source parameter under /etc/lvm/lvm.conf is set as uname.
  • Node rh8-nd2 has the testvg active on it currently.

Method 1: Update systemid From Any Node in the Cluster

  1. Stop the cluster to perform troubleshooting.

    pcs cluster stop --all
    
  2. Use vgs -ao+systemid command to confirm testvg is currently only showing active on node rh8-nd2 with node hostname listed as current system ID. Node rh8-nd1 should not list testvg or not list a system ID for the volume group.

      [root@rh8-nd2 ~]# vgs -ao+systemid
        VG     #PV #LV #SN Attr   VSize   VFree System ID
        rhel     1   2   0 wz--n- <11.00g    0           
        testvg   2   1   0 wz--n-  <1.20g    0  rh8-nd2  
    
    
      [root@rh8-nd1 ~]# vgs -ao+systemid
        VG   #PV #LV #SN Attr   VSize   VFree System ID
        rhel   1   2   0 wz--n- <11.00g    0           
    
  3. Activate the VG on passive node rh8-nd1 by temporarily setting the system_id as rh8-nd2 with vgchange -y --config local/extra_system_ids=[\"rh8-nd2\"] --systemid rh8-nd1 testvg command so it can be interacted with on rh8-nd1. This will allow rh8-nd1 to update the system ID independently to rh8-nd1.

    [root@rh8-nd1 ~]# vgchange -y --config local/extra_system_ids=[\"rh8-nd2\"] --systemid rh8-nd1 testvg
      Volume group "testvg" successfully changed
    
  4. Confirm the system ID has successfully updated to rh8-nd1.

    [root@rh8-nd1 ~]# vgs -ao+systemid
      VG     #PV #LV #SN Attr   VSize   VFree System ID
      rhel     1   2   0 wz--n- <11.00g    0           
      testvg   2   1   0 wz--n-  <1.20g    0  rh8-nd1     <<===---- System ID is now updated to passive node
    
  5. Activate the logical volume for testvg on rh8-nd1.

    [root@rh8-nd1 ~]# vgchange -ay testvg
      1 logical volume(s) in volume group "testvg" now active
    
  6. Confirm activation of logical volume with lvs command.

    [root@rh8-nd1 ~]# lvs 
      LV     VG     Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
      root   rhel   -wi-ao----  9.79g                                                    
      swap   rhel   -wi-ao----  1.20g                                                    
      testlv testvg -wi-a----- <1.20g           <<===---- LV is now active on passive node and available to get mounted locally on node.                                 
    
  7. Confirm the filesystem related to testvg is not mounted locally on any cluster nodes with mount.

  8. Confirm the testvg VG/ testlv LV is deactivated on both nodes with vgchange -a n testvg the confirm with lvs on all cluster nodes.

    [root@rh8-nd1 ~]# lvs 
      LV     VG     Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
      root   rhel   -wi-ao----  9.79g                                                    
      swap   rhel   -wi-ao----  1.20g                                                    
      testlv testvg -wi------- <1.20g           <<===---- LV is now deactivated on passive node.
    
  9. Start cluster to bring all resources up.

    pcs cluster start --all
    

Method 2: Update systemid From Node Matching Assigned systemid

  1. Stop the cluster to perform troubleshooting.

    pcs cluster stop --all
    
  2. Remove the current systemID from volume group. This must be ran from node with matching "systemid".

    vgchange --systemid "" <vgname>
    
  3. Add the systemID of preferred node to the volume group.

    vgchange --systemid "<hostname>" <vgname>
    
  4. Start cluster to bring all resources up.

    pcs cluster start --all
    

Diagnostic Steps

  • Command vgs -ao+systemid shows different system ID information for the shared cluster volume group.

       [root@rh8-nd2 ~]# vgs -ao+systemid
         VG     #PV #LV #SN Attr   VSize   VFree System ID
         rhel     1   2   0 wz--n- <11.00g    0           
         testvg   2   1   0 wz--n-  <1.20g    0  rh8-nd2  
    
    
       [root@rh8-nd1 ~]# vgs -ao+systemid
         VG   #PV #LV #SN Attr   VSize   VFree System ID
         rhel   1   2   0 wz--n- <11.00g    0           
    
SBR
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.