fence_mpath fails to create persistent reservations on HP 3PAR VV devices in a Pacemaker cluster

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 7 (with the High Availability Add-on)
  • Red Hat Enterprise Linux 8 (with the High Availability Add-on)

Issue

  • The fence_mpath fence agent fails to create persistent reservations on a HP 3PAR Virtual Volume (VV) device, even though there are currently no keys or reservations on the device:

      #  mpathpersist -ird /dev/mapper/mpatha
     PR generation=0x6d8, there is NO reservation held

  #  mpathpersist --out --reserve --param-rk=0x1 --prout-type=5 /dev/mapper/mpatha
     persistent reserve out: scsi status: Reservation Conflict
     PR out: command failed

  • Unfencing a node fails when using fence_mpath with HP 3PAR VV devices, reporting the following error message on the cluster logs: "ERROR: Failed to verify x device(s)"

Resolution

  1. Please update the /etc/multipath.conf file and add on the defaults section, the all_tg_pt option:

     # vi /etc/multipath.conf
    
    defaults {
            find_multipaths yes 
            user_friendly_names yes
            reservation_key 0x1  --> example of reservation
            all_tg_pt yes        --> add the option on the defaults section
    }

For more details, please check: DM Multipath: Chapter 4.3. Configuration File Defaults

  1. Reload the multipathd service and include the new option on the initram image:

     # systemctl restart multipathd
    OR # multipath -r
    
 # dracut --force -v --add multipath --include /etc/multipath /etc/multipath

If the issue persists, please open a Red Hat support case.

Root Cause

During the cluster initialization, the fence_mpath unfencing operation (fence on) fails to register the keys on the HP 3PAR VV devices, which would prevent all resources from starting.

The issue seems to be present on HP 3PAR Virtual Volumes (VV), where this seems to be a reservation race condition on HP 3PAR environments, not seen on other EMC or Hitachi storage. The parameter all_tg_pt set to yes on /etc/multipath.conf would guarantee the access to the devices be serialized to reach the Luns.

For multipath served devices using fence_mpath, we would report all_tg_pt=1 (True) is the correct way to use.
Meaning of all_tg_pt: The all_tg_pt parameter indicates whether all target ports of a storage device should participate in a persistent reservation.

  • all_tg_pt=1 (True)
    When set, the persistent reservation applies to all target ports of the device.
    This means every port on the storage device is involved in the reservation, ensuring that any I/O through any target port adheres to the persistent reservation rules.
    This is commonly used in setups where multiple paths to a device exist, ensuring consistency and preventing conflicts across paths.

  • all_tg_pt=0 (False)
    The persistent reservation applies only to the specific target port used during the reservation setup.
    Other target ports on the device are not affected, allowing separate reservations or unrestricted I/O on those ports.

Diagnostic Steps

  1. Observe a reservation conflict when trying to create a SCSI persistent reservation.

     #  mpathpersist --out --reserve --param-rk=0x1 --prout-type=5 /dev/mapper/mpatha
 persistent reserve out: scsi status: Reservation Conflict
 PR out: command failed

  2. Ensure that there is no existing persistent reservation visible at the OS level.

     #  mpathpersist -ird /dev/mapper/mpatha
   PR generation=0x6d8, there is NO reservation held

  3. Ensure that you're using the correct reservation key.

     # grep key /etc/multipath.conf 
 	reservation_key 0x1

  4. Verify that the storage device is compliant with SPC-3 or greater.

  5. Check that you're using a HP 3PAR VV storage device (visible in the sg_inq output).

     # sg_inq /dev/mapper/mpatha
 standard INQUIRY:
   PQual=0  Device_type=0  RMB=0  LU_CONG=0  version=0x06  [SPC-4]
   ...
  Vendor identification: 3PARdata
  Product identification: VV
  Product revision level: 4210
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.