Can Pacemaker fence the cluster node with the fewest running resources?
Environment
- Red Hat Enterprise Linux 7, 8, 9 (with the High Availability Add-on)
Issue
- Can Pacemaker avoid fencing the node with the highest total resource priority in the event of a split brain?
Resolution
Red Hat Enterprise Linux 7
This feature request was tracked in [Bug 1840407 - RFE: A way to implement a more dynamic fence delay so that in the event of a network split, the cluster will fence the node running the fewest resources](https://bugzilla.redhat.com/show_bug.cgi?id=1840407). As of 27 May 2020, this RFE is in CLOSED WONTFIX state.
This feature will not be added because RHEL 7 is in the Maintenance Support 2 phase of its life cycle and because of some upgrade compatibility issues.
Red Hat Enterprise Linux 8
As of [`pacemaker-2.0.4-6.el8`](/errata/RHEA-2020:4804), Pacemaker supports the new [`priority-fencing-delay`](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/8.3_release_notes/rhel-8-3-0-release#enhancement_high-availability-and-clusters) cluster property. This property allows you to configure a two-node cluster so that in a split-brain situation a fencing delay is added for the node with the highest total resource priority. That is, the node with the lowest total resource priority is more likely to be fenced first (fence delays help when both nodes see the loss of the other and schedule fencing for it at the same time, but that doesn't always happen at the same time).
For more information then see: 10.4. Fencing delays | Configuring fencing in a Red Hat High Availability cluster.
Red Hat Enterprise Linux 9
RHEL 9 supports the priority-fencing-delay cluster property in all minor versions.
For more information then see 10.4. Fencing delays | Configuring fencing in a Red Hat High Availability cluster.
Note that this mechanism is only useful for 2 node clusters.
The priority can be set for all resources via defaults to give each resource equal weight.
Alternatively, the priority can be set for an individual resource via the priority meta attribute. The delay is applied before fencing the node with the highest total resource priority. So if one resource is assigned a much higher priority than other resources, that resource contributes more to the total resource priority on the node where it's running. You can take advantage of this if one resource is much more important than the others by setting a high priority for that resource, ensuring that the delay is applied to the node where that resource is running. This can help maintain uptime for a critical resource that stops and/or starts slowly.
For example, to set the priority of all resources to 1 so that all resources are equally weighted:
# pcs resource defaults update priority=1
To set the priority of a particular resource called sid_ascs00 to 10 (overriding any priority value set in resource defaults):
# pcs resource update sid_ascs00 meta priority=10
Note: For cloned resources the meta attribute should be applied to the clone resource and promoted clones resources will have a priority that is 1 higher than the configured priority.
Finally, to set the priority-fencing-delay to 15s (for example):
# pcs property set priority-fencing-delay=15s
For more information on fencing delays in general, see the following solution: Delaying Fencing in a Two Node Cluster to Prevent Fence Races or "Fence Death" Scenarios.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.