How can I configure my cluster to use a specific network interface for communication with other nodes?

Updated

Overview

This article describes how to configure the cluster software to use a specific network interface and IP address or hostname for communication amongst the nodes in a Red Hat Enterprise Linux (RHEL) 5, 6, or 7 High Availability cluster using either cman and/or corosync.

How is the network interface chosen for cluster communication?

In both cman and corosync-based clusters, the cluster software determines which interface and IP address to bind to for cluster communications by reading the configured nodename or node address and then mapping that back to an IP through one of a few different methods. If this nodename or address is given in the form of an IP address, then the process is simple: bind to that IP. If, however, the name or address is not an IP, then this name is resolved in either /etc/hosts or DNS to determine its mapping to an IP, and that address is used to communicate over.

In summary: the simplest and most effective way of configuring which network interface is used to communicate over is to:

a) Create host-mappings for every node's IP and name in /etc/hosts on all nodes, and
b) Use that name when defining the cluster nodes in the configuration

Note: the use of DHCP for obtaining an ip address for an interface being used by corosync for heartbeat communication is unsupported

RHEL 7: Configuring a corosync-based cluster to use a specific network interface.

1. Define all nodes in /etc/hosts. If these nodes do not have any pre-existing hostname on that network, then names can be assigned to them in this file, as long as those names don't conflict with others hosts on the network.

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.143.61 node1.example.com node1
192.168.143.62 node2.example.com node2

2. Set up the cluster with pcs, the pcsd web interface, or directly in /etc/corosync/corosync.conf, specifying names matching those that were added in /etc/hosts. For example:

# pcs cluster setup --name myCluster node1.example.com node2.example.com

Which would produce a nodelist definition in /etc/corosync/corosync.conf with those names as the "ring0_addr" for each node:

nodelist {
  node {
        ring0_addr: node1.example.com
        nodeid: 1
       }
  node {
        ring0_addr: node2.example.com
        nodeid: 2
       }
}

NOTE: RHEL 7 corosync supports the usage of Redundant Ring Protocol in which multiple redundant interfaces can be used to communicate in the cluster. In such configurations, the "ring1_addr" should be defined in /etc/hosts the same way the primary name was.

3. Upon starting the cluster with pcs cluster start [--all], check the output of corosync-cfgtool on each node to see what addresses are being used for node communication:

[root@node1 ~]# corosync-cfgtool -s
Printing ring status.
Local node ID 1
RING ID 0
	id	= 192.168.143.61
	status	= ring 0 active with no faults

[root@node1 ~]# corosync-cfgtool -s
Printing ring status.
Local node ID 1
RING ID 0
	id	= 192.168.143.62
	status	= ring 0 active with no faults

RHEL 5 or 6: Configuring a cman-based cluster to use a specific network interface

1. Define all nodes in /etc/hosts. If these nodes do not have any pre-existing hostname on that network, then names can be assigned to them in this file, as long as those names don't conflict with others hosts on the network.

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.143.61 node1.example.com node1
192.168.143.62 node2.example.com node2

2. Configure the cluster with Conga, ccs (RHEL 6 only), or directly in /etc/cluster/cluster.conf, specifying clusternode names matching those entries added to /etc/hosts:

<cluster config_version="1" name="myCluster">
	<clusternodes>
		<clusternode name="node1.example.com" nodeid="1">
                       [...]
   		</clusternode>
		<clusternode name="node2.example.com" nodeid="2">
                       [...]
		</clusternode>
	</clusternodes>

NOTE: In RHEL 6, Redundant Ring Protocol allows for configuring an additional redundant interface to communicate over. This "altname" should be configured in /etc/hosts the same way as the primary node name.

3. After completing configuration of the rest of the necessary components and starting the cman service, check cman_tool status to see what addresses the nodes are communicating over:

[root@node1 ~]# cman_tool status | grep "Node addresses"
Node addresses: 192.168.143.61 

[root@node2 ~]# cman_tool status | grep "Node addresses"
Node addresses: 192.168.143.62
Product(s)
Category
Components
Tags