Both nodes in a two-node cluster race to fence each other even though they are using a quorum device without a heuristic which should enable master_wins in RHEL 6

Environment

• Red Hat Enterprise Linux (RHEL) 6 with the High Availability Add On
• A cluster consisting of exactly two nodes
• Cluster utilizing a quorum device (<quorumd> in /etc/cluster/cluster.conf)
  • The intended configuration is for master_wins to be enabled
  • There are no <heuristics> specified in /etc/cluster/cluster.conf
  • The <quorumd> tag does not specify master_wins="1", but rather is left blank with the expectation that it will be auto-enabled
  • The <quorumd> tag specifies a votes="<count>" attribute

Issue

• In a 2 node cluster both nodes fenced each other causing a reboot, even though we have a qdisk with master_wins enabled.
• I have a quorum device without any heuristics in a two node cluster, and so I expect master_wins to be automatically enabled, however both nodes are showing that they have a quorum device vote in cman_tool status output:

[root@node1 ~]# cman_tool status
[...]
Expected votes: 3
Quorum device votes: 1
Total votes: 3
Node votes: 1
[...]

[root@node2 ~]# cman_tool status
[...]
Expected votes: 3
Quorum device votes: 1
Total votes: 3
Node votes: 1
[...]

• Both nodes in the cluster raced to fence each other even though I'm using a master_wins configuration

Resolution

Either:

• Remove the votes="<count>" attribute from the <quorumd> tag in /etc/cluster/cluster.conf, OR
• Specify master_wins="1" in the <quorumd> tag in /etc/cluster/cluster.conf.
• Apply updates from advisory This content is not included.RHBA-2015:1363 - update packages to versions below:
  • clusterlib-3.0.12.1-73.el6
  • cman-3.0.12.1-73.el6

Root Cause

Normally if master_wins is not specified in the <quorumd> tag in /etc/cluster/cluster.conf in a two-node cluster with no heuristics, this mode will be automatically enabled. However, if the "votes" attribute is specified, it prevents auto-enabling of master_wins. As such, either votes must be removed from the configuration allowing it to be auto-calculated, or master_wins must be specified manually.

Red Hat investigated in Bugzilla #1099223 whether qdiskd can be modified in future releases to automatically enable master_wins even if votes="1" is specified.
The versions mentioned in This content is not included.RHBA-2015:1363 advisory contains this feature.

Diagnostic Steps

• Check cman_tool status after the cluster has been started for a few minutes (qdiskd needs time to initialize and register), and see if "Quorum device votes" are listed for both nodes, as shown in the Issue section above. If so, master_wins mode has not been properly enabled.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.