How do I configure cluster nodes to use a dedicated heartbeat interface?
Environment
- Red Hat Cluster Suite 4+
- Red Hat Enterprise Linux Server 5 (with the High Availability Add On)
- Red Hat Enterprise Linux Server 6 (with the High Availability Add On)
- Red Hat Enterprise Linux Server 7 (With the High Availability Add On)
Issue
- How do I configure cluster nodes to use a dedicated heartbeat interface?
- How to use separate network for cluster heartbeat?
Resolution
Many administrators prefer to set up a second, dedicated interface on their clustered systems for the heartbeat service. This is generally a good idea, especially if the primary network interface has a lot of traffic. The way that the network interface is choosen for cluster communication is explained in the following article: What should I name my cluster nodes?.
This article assumes you have a working cluster configuration where the node names are not the same as the machine's hostname. In addition, it is assumed each cluster node has two properly configured network interfaces.
In following example, we have a two node cluster. The hostnames of these systems are cluster1.example.com and cluster2.example.com. The node names of the two systems, as defined in the cluster configuration file /etc/cluster/cluster.conf are node1-priv and node2-priv.
# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="276" name="testcluster">
<clusternodes>
<clusternode name="node1-priv" nodeid="1">
<fence>
<method name="Method2">
<device name="rhevm_fence" port="node1"/>
</method>
</fence>
</clusternode>
<clusternode name="node2-priv" nodeid="2">
<fence>
<method name="Method2">
<device name="rhevm_fence" port="node2"/>
</method>
</fence>
</clusternode>
</clusternodes>
[...]
To configure the heartbeat to use a second interface instead of the primary interface, modify the /etc/hosts file on each node so that the hostname points to the IP address of the primary interface. The node name should point to the secondary interface. Here is the /etc/hosts file for both nodes:
$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
# node1 settings
10.10.xx.xx cluster1.example.com cluster1
192.168.xx.xx node1-priv #### Cluster node name
# node2 settings:
10.10.xx.xx cluster2.example.com cluster2
192.168.xx.xx node2-priv #### Cluster node name
In the above configuration, the heartbeat would be sent over eth1 in the 192.168.xx. subnet. All other traffic would be sent over eth0, which has is the 10.10.xx. subnet.
Note: The change in cluster heartbeat network will require a restart of entire cluster, please refer to the following article for detailed information about which configuration changes would need a restart of cluster: Which configuration changes require a restart in a Red Hat Enterprise Linux cluster?.
If you are wanting to configure a redundant heartbeat network then see the following article: How can I configure a redundant heartbeat network for Red Hat Clustering?.
RHEL 6 HA Add-On with Pacemaker
To setup a cluster on a private network using the /etc/hosts shown above use:
pcs cluster setup --name my_cluster node1-priv node2-priv
RHEL 7 HA Add-On with Pacemaker
This is the same as RHEL 6 w/ Pacemaker during initial configuration, however if you're configuring a new heartbeat interface on an established RHEL 7 cluster, please refer to the following guide:
How do I change my currently configured heartbeat interface for RHEL 7 Pacemaker?
Diagnostic Steps
To find out which network interface that is used for cluster communication then run the following commands below. This example will demonstrate that eth1 is the network interface used for cluster communication. This example assumes that cman started successfully on the cluster node.
-
Find the IP that is used for cluster communication for RHEL 5 or RHEL 6.
# cman_tool status | grep "Node addresses" Node addresses: 192.168.122.151 -
Find the IP that is used for cluster communication for RHEL 6 or RHEL 7 which will list IP addresses for all the cluster nodes.
# corosync-quorumtool -li Nodeid Name 1 192.168.122.151 2 192.168.122.152 3 192.168.122.153 -
Find the network interface that uses that particular IP.
# ip addr | grep "192\.168\.122\.151" -B 2 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 52:54:00:05:01:01 brd ff:ff:ff:ff:ff:ff inet 192.168.122.151/24 brd 192.168.122.255 scope global eth1
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.