How can I configure fencing for redundant power supplies in a RHEL 6 or 7 High Availability cluster with pacemaker?

Solution Unverified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 6 or 7 with the High Availability Add On
  • pacemaker
  • Redundant fence (stonith) devices that must all be turned off before any are turned back on
    • This is typical with multiple power supplies

Issue

  • I have redundant power supplies on my cluster nodes, so when a node must be fenced, both ports must be shut off. How do I configure this in pacemaker?
  • In cman clusters, we used to configure redundant devices in cluster.conf by referencing them in an off/off/on/on pattern. Is there a way to configure this with pcs and stonith?
  • When I try to configure redundant fence devices, I am not able to reference them in a level and also specify the action and port. Do I need to create a separate device for each action that needs to be performed?
  • I have two power supplies. How do I configure stonith to handle this correctly?

Resolution

Overview

In environments where High Availability cluster nodes are managed by multiple redundant devices, such as those having two independent power supplies, require a special stonith configuration to ensure that all devices are turned off and back on in the correct order. For example, if a node has two different power supplies, using the default action of "reboot" for both would result in the first power supply being turned off then on, and then the second power supply off then on; the end result is that the node would never completely lose power, and thus could not be guaranteed to have released its resources, opening up the possibility of nodes accessing these resources simultaneously and corrupting them.

The proper way to address this problem is to create one stonith device for each combination of device settings and action that must be performed. For example, a stonith device for power supply 1 and action "off", a device for power supply 2 and action "off", a device for power supply 1 and action "on", and a device for power supply 2 and action "on". The a stonith level can be created, referencing these devices in the proper order.

Stonith Device Creation


For each fence device managing each node, use `pcs` to create a set of devices, one for the "off" action and one for "on". 
# pcs stonith create <stonith id> <stonith device type> [stonith device options] pcmk_reboot_action="<action>" 
[...]

For example, with 2 separate APC power supplies for each of two nodes:

# pcs stonith create node1_apc1_port1_off fence_apc_snmp ipaddr=apc1.example.com port=1 login=user passwd='7a4D#1j!pz864' pcmk_reboot_action="off"
# pcs stonith create node1_apc2_port1_off fence_apc_snmp ipaddr=apc2.example.com port=1 login=user passwd='7a4D#1j!pz864' pcmk_reboot_action="off"
# pcs stonith create node1_apc1_port1_on fence_apc_snmp ipaddr=apc1.example.com port=1 login=user passwd='7a4D#1j!pz864' pcmk_reboot_action="on"
# pcs stonith create node1_apc2_port1_on fence_apc_snmp ipaddr=apc2.example.com port=1 login=user passwd='7a4D#1j!pz864' pcmk_reboot_action="on"
# pcs stonith create node2_apc1_port2_off fence_apc_snmp ipaddr=apc1.example.com port=2 login=user passwd='7a4D#1j!pz864' pcmk_reboot_action="off"
# pcs stonith create node2_apc2_port2_off fence_apc_snmp ipaddr=apc2.example.com port=2 login=user passwd='7a4D#1j!pz864' pcmk_reboot_action="off"
# pcs stonith create node2_apc1_port2_on fence_apc_snmp ipaddr=apc1.example.com port=2 login=user passwd='7a4D#1j!pz864' pcmk_reboot_action="on"
# pcs stonith create node2_apc2_port2_on fence_apc_snmp ipaddr=apc2.example.com port=2 login=user passwd='7a4D#1j!pz864' pcmk_reboot_action="on"

Adding Devices to New Level


Now that the devices are created, add the devices for each node to a new level in an off, off, on, on pattern. 
# pcs stonith level add <level> <node> <devices>

Continuing the example above:

# pcs stonith level add 1 node1.example.com node1_apc1_port1_off,node1_apc2_port1_off,node1_apc1_port1_on,node1_apc2_port2_on
# pcs stonith level add 1 node2.example.com node2_apc1_port2_off,node2_apc2_port2_off,node2_apc1_port2_on,node2_apc2_port2_on

Adding More Devices


If a device ever needs to be added to a level after it is created, such as if another power supplied is attached to the system, then the entire level should be deleted and recreated, rather than simply adding to the existing level again. 
# pcs stonith level remove <level> [node id]

For example, to add a third power supply to node 1 from the above example:

# pcs stonith create node1_apc3_port1_off fence_apc_snmp ipaddr=apc3.example.com port=1 login=user passwd='7a4D#1j!pz864' pcmk_reboot_action="off"
# pcs stonith create node1_apc3_port1_on fence_apc_snmp ipaddr=apc3.example.com port=1 login=user passwd='7a4D#1j!pz864' pcmk_reboot_action="on"
# pcs stonith level remove 1 node1.example.com
# pcs stonith level add 1 node1_apc1_port1_off,node1_apc2_port1_off,node1_apc3_port1_off,node1_apc1_port1_on,node1_apc2_port2_on,node1_apc3_port1_on

Root Cause

Configuring redundant power supplies adds further resiliency for nodes, but requires a more complex setup in stonith to ensure that all devices are powered off and on in the right order. The above steps and examples represent the most common configuration for such environments, but other considerations may exist for non-standard setups. Contact Red Hat Global Support Services if there are questions or concerns with this approach.

Note the last section from the Resolution above should always be followed if adding new devices to an existing configuration even if ordering does not matter. The reason is that when devices are added to a level for the first time, the order in which they're specified is preserved and they will be executed in that order. However, if another device is later added to the same example, in the cib (cluster XML configuration) this will be represented as two separate level entries each referencing the same numeric level. stonith-ng cannot be guaranteed to execute these in any particular order, and as such, the correct solution is to delete the level and re-add it with all of the necessary devices specified in the correct order.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.