How to setup/create/configure fence_mpath

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 6 and later (with the High Availability Add-on)

Issue

  • How to configure fence_mpath as stonith device in pacemaker cluster?
  • fence_mpath does not successfully fence a node

Resolution

  1. Edit /etc/multipath.conf on each node and in the defaults section, add a reservation_key line that contains a key (in hexadecimal form) that is specific to that node.

    For example, on node1:

     defaults {
       user_friendly_names yes
       find_multipaths yes
       reservation_key 0x1
 }

    And on node 2:

     defaults {
       user_friendly_names yes
       find_multipaths yes
       reservation_key 0x2
 }

  2. Reload multipath configuration on both nodes.

     ## RHEL 6
 # service multipathd reload

 ## RHEL 7 and later
 # systemctl reload multipathd

  3. Configure a shared stonith device, specifying the list of multipath devices to manage, and the mapping of node names to key. Unlike fence_scsi the fence_mpath accepts the multipath devices in the devices attribute. In order to create a map of node names to keys, we have to set an attribute to inform pacemaker the name of the value in key/value pairs within pcmk_host_map with the attribute pcmk_host_argument. The pcmk_host_map will contain a mapping of the <cluster node name>:<key>. We will need to use the option --force as we are omitting the required attribute key from being configured (instead we are setting key as the value in pcmk_host_map). Do note that configuring fence_mpath as shared stonith device will prevent pacemaker from logging messages about a missing port attribute. 

     # pcs stonith create mpath fence_mpath devices="/dev/mapper/mpath1,/dev/mapper/mpath2" pcmk_host_map="node1:1;node2:2" pcmk_monitor_action="metadata" pcmk_reboot_action="off" pcmk_host_argument="key" meta provides=unfencing --force

    With the release of This content is not included.fence-agents-4.2.1-30.el7 you no longer have to include the option pcmk_host_argument with fence_mpath when using pcmk_host_map. The fence_mpath will assume the value in the pair is the key for the node.

     # pcs stonith create mpath fence_mpath devices="/dev/mapper/mpath1,/dev/mapper/mpath2" pcmk_host_map="node1:1;node2:2" pcmk_monitor_action="metadata" pcmk_reboot_action="off" meta provides=unfencing

    With the release of fence-agents-mpath-4.10.0-86.el9_6 the reservation_key value in /etc/multipath.conf can be used in pcmk_host_map instead of requiring only integers.

     # pcs stonith create mpath fence_mpath devices="/dev/mapper/mpath1,/dev/mapper/mpath2" pcmk_host_map="node1:0x1;node2:0x2" pcmk_monitor_action="metadata" pcmk_reboot_action="off" meta provides=unfencing

  4. Check that multipath devices contain registration keys used in stonith devices.

     # mpathpersist -i -r -d /dev/mapper/mpath1
 # mpathpersist -i -k -d /dev/mapper/mpath1

    If more debug information is needed for clearer diagnostics, feel free to use -v 3 or -v 4:

     # mpathpersist -v 3 -i -r -d /dev/mapper/mpath1
 # mpathpersist -v 3 -i -k -d /dev/mapper/mpath1

    The first command should just show a single reservation with one of the node's keys (it doesn't matter which one), and the second command should show you several keys for each node (most likely corresponding to the number of paths each node has to the device). If seen, then unfencing is successful and can move on to testing fencing.

  5. Test fencing by panicking a node with echo c > /proc/sysrq-trigger then running the mpathpersist commands from previous step again. The registration keys for fenced node should be gone, and if it was the reservation holder, the key on the reservation should correspond to a different node now. If that's the case and you don't see any errors, then fencing is set up successfully.

Note that fence_mpath is different from fence_scsi in that you have to supply a key for each node; it does not auto-generate a key for the node where it is being run. So the stonith device configuration will be different from what you may be used to with fence_scsi. You will have to specify keys explicitly in the stonith devices.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.