When using a quorum device ping heuristic to prevent fence races in a two-node RHEL cluster, both nodes still fence each other

Environment

• Red Hat Enterprise Linux (RHEL) 5 with the High Availability Add On
• Cluster configured to use a quorum device (<quorumd> in /etc/cluster/cluster.conf)
  • One or more ping heuristics (<heuristic program="ping ..." .../> in `/etc/cluster/cluster.conf)

Issue

• Why do both cluster nodes race to fence each other after removing all cluster interconnect network cables in a two node cluster with a quorum disk and ping heuristic?
• Shouldn't a quorum disk prevent fence races?

Resolution

Configure a ping heuristic to operate on the cluster interconnect network. For example, if the cluster uses the bond1 interface on network 192.168.0.0/24 for communication, then set up the heuristic to ping the gateway of that network:

<heuristic program="ping -w1 -c1 192.168.0.1" tko="5" interval="2" score="1"/>

NOTE: If this heuristic is not the only one, then the <quorumd min_score> attribute and/or <heuristic score> attribute should be adjusted such that the failure of this ping heuristic will result in a negative transition in score.

Alternative solutions for avoiding fence races are also available.

Root Cause

A quorum device in itself is not enough to avoid fence races. It must either use master_wins mode, or be configured with heuristics that will break the tie in all relevant scenarios.

When using a quorum device with a heuristic to avoid fence races in a 2-node cluster, that heuristic must serve as the tie-breaker when there is a network split between nodes. In other words, when the cluster interconnect network goes down or is split between the nodes such that they cannot communicate, one node's heuristics must fall below min_score while the other node remains above min_score; the node that has had a negative transition will reboot itself, thereby "breaking the tie".

If the status of the heuristic is not affected by the failure of the cluster interconnect, then it will not serve as a tie-breaker, and thus both nodes can still race to fence each other. This problem must be solved by either moving the ping to the cluster interconnect network, by adding another ping heuristic for the interconnect network (and ensuring the failure of that heuristic will cause a drop below min_score), or using one of the other methods for avoiding fence races (master_wins mode or a fence delay).

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.