What are the recommendations for number of logical volumes in a HA-LVM volume group on Red Hat High Availability Clusters?

Solution Verified - Updated

Environment

  • Red Hat Cluster Suite (RHCS) 4+
  • Red Hat Enterprise Linux (RHEL) 5 with the High Availability or Resilient Storage Add Ons
  • Red Hat High Availability cluster with 2 or more nodes.
  • rgmanager or Pacemaker
  • HA-LVM
    • One or more lvm (rgmanager) or LVM (Pacemaker) resources configured in the cluster

Issue

  • When trying to move a service from CONGA interface, error clurgmgrd: [5522]: HA LVM requires Only one logical volume per volume group is seen in /var/log/messages file.
  • Why am I not able to configure multiple Logical Volumes from single VG as cluster resources ? We are created 3 LV on 1 VG.
  • Does Cluster support more than one LV on each VG ? How can I configure it?
clurgmgrd[5522]: <notice> Starting stopped service service:SVC_ORASOT 
clurgmgrd: [5522]: <err> HA LVM requires Only one logical volume per volume group. 
clurgmgrd: [5522]: <err> There are currently 3 logical volumes in vgorasot 
clurgmgrd: [5522]: <err> Failing HA LVM start of vgorasot/lvoradata_sot 
clurgmgrd[5522]: <notice> start on lvm "LVM_lvoradata_sot" returned 1 (generic error) 
clurgmgrd[5522]: <warning> #68: Failed to start service:SVC_ORASOT; return value: 1 
clurgmgrd[5522]: <notice> Stopping service service:SVC_ORASOT

  • Can multiple LVM logical volumes from single VG be configured as cluster resources ?

  • Can we configure multiple Volume Groups to be managed by the HA-LVM Cluster?

  • How many logical-volumes can I have in one volume-group under HA-LVM?

  • I want to use HA-LVM in my RHEL 5 cluster, but as I understand it, early versions of HA-LVM only permitted 1 logical-volume in the volume-group under HA-LVM control. My volume-groups have between 7 and 15 logical-volumes; would this work?

  • Does RedHat have a recommendation regarding HA-LVM and the number of logical volumes per volume group?

    • Should we just have a 1 to 1 mapping for LUN-VG-LV? i.e.:
    LUN 1 ---> VolumeGroup01  ---> LogicalVolume01
LUN 2 ---> VolumeGroup02  ---> LogicalVolume02
LUN 3 ---> VolumeGroup03  ---> LogicalVolume03
LUN 4 ---> VolumeGroup04  ---> LogicalVolume04
LUN 5 ---> VolumeGroup05  ---> LogicalVolume05
    • Or should we have multiple LVs per VG? i.e.:
    LUN 1 ---> VolumeGroup01  ---> LogicalVolume01
LUN 1 ---> VolumeGroup01  ---> LogicalVolume02
LUN 1 ---> VolumeGroup01  ---> LogicalVolume03
LUN 1 ---> VolumeGroup01  ---> LogicalVolume04
LUN 2 ---> VolumeGroup02  ---> LogicalVolume05

Resolution

The behavior has changed over various versions of Red Hat Enterprise Linux, and differs for the clvmd variant vs tagging variant of HA-LVM. The cluster version and HA-LVM variant will determine the behavior.

When using rgmanager

For all versions and variants:

  • Specifying a vg_name and leaving lv_name blank causes the resource to manage the entire volume group and all of its logical volumes as a whole. However only certain versions/variants allow more than one logical volume in a volume group (see below).
  • Specifying a vg_name and lv_name will cause only that logical volume to be activated by the resource
  • There is no theoretical limit on the number of volume groups, and when more than 1 logical volume in a volume group is allowed (see below), there is no theoretical limit on the number of logical volumes.
Prior to RHEL4.7 and RHEL5.2, using the Tagging Variant:
  • Each volume group can only contain a single logical volume.
  • If multiple LVM resources are required in the cluster service, multiple LVM volume groups must be created, each with only 1 logical volume.
  • There is no theoretical limit on the number of volume groups (and therefore, LVM resources) that can exist.
  • LVM logical volume resources must not be a member of more than 1 service.
For RHEL4.7+, RHEL5.2+ or RHEL6, using the Tagging Variant:

  • Volume groups can contain more than 1 logical volume, but only if lv_name parameter in the <lvm> resource is left blank (resource manages the entire volume group).

    • If the lv_name parameter is specified and there are 2 or more logical volumes in the volume group, the logical volume will fail to activate.
    clurgmgrd: [5522]: <err> HA LVM requires Only one logical volume per volume group. 
clurgmgrd: [5522]: <err> There are currently 3 logical volumes in vg
clurgmgrd: [5522]: <err> Failing HA LVM start of vg/lv

  • Any individual volume group may only be accessed by a single service.

RHEL5.6+ or RHEL6, using the clvmd Variant:

  • Volume groups may contain multiple logical volumes if needed.

  • Multiple <lvm> resources referencing different logical volumes (lv_name) in the same volume group can be used, across different services if needed.

When using Pacemaker

NOTE: Pacemaker is only supported in certain releases of RHEL 6, and only supports the tagging variant with HA-LVM

  • Volume groups may contain multiple logical volumes if needed
  • The LVM resource available to Pacemaker can only activate entire volume groups. This prevents management of individual logical volumes by different resources or resource groups.

Root Cause

  • The LVM resource script has evolved during the life of RHEL4, RHEL5 and RHEL6, and behaves differently depending on the cluster version and HA-LVM variant employed.

    • Prior to RHEL4.7 and RHEL5.2, the LVM resource script could only manage 1 logical volume per volume group and only offered the tagging variant.
    • For RHEL4.7+ and RHEL5.2+, the LVM resource script was enhanced with the ability to manage more than 1 logical volume per volume group by managing an entire volume group (by leaving lv_name blank), but still only offered the tagging variant.
    • For RHEL5.6+ and RHEL6, the LVM resource group can manage clustered LVM (CLVM) logical volumes, which don't require tagging, allows multiple logical volumes per vg, and can even have individual logical volumes from the same vg managed in different services.

Diagnostic Steps

If the following symptoms are present on your system, then this solution may apply:

  • You have a volume group with more than 1 logical volume in it.
  • You are using rgmanager and create an lvm resource in /etc/cluster/cluster.conf, and the resource has a logical volume name specified in the lv_name field.
  • When you try to start an rgmanager service that contains an LVM resource, and lv_name is specified, but the volume group containing $lv_name has more than 1 volume group, you may see the following error in /var/log/messages and the service may fail to start:
clurgmgrd[5522]: <notice> Starting stopped service service:SVC_MYSERVICE 
clurgmgrd: [5522]: <err> HA LVM requires Only one logical volume per volume group. 
clurgmgrd: [5522]: <err> There are currently 3 logical volumes in vgdata 
clurgmgrd: [5522]: <err> Failing HA LVM start of vgdata/lvdata
clurgmgrd[5522]: <notice> start on lvm "LVM_lvdata" returned 1 (generic error) 
clurgmgrd[5522]: <warning> #68: Failed to start service:SVC_MYSERVICE; return value: 1 
clurgmgrd[5522]: <notice> Stopping service service:SVC_MYSERVICE
  • You are using Pacemaker and and LVM resource
SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.