Exploring RHEL High Availability's Components, Concepts, and Features - Overview of Transport Protocols

Updated 7 Jul 2023

Overview

Applicable Environments

Red Hat Enterprise Linux (RHEL) 6, 7, 8 or 9 with the High Availability Add-On

Useful References and Guides

Design guidance - Selecting the Transport Protocol

Introduction

This guide offers a breakdown of the messaging protocols that a RHEL High Availability cluster may use to send data between members, and the essential knowledge for effectively using those protocols.

Uses

The transport setting defines the protocol through which nodes of a cluster send messages to each other. When a node must transmit some piece of information cluster-wide, it follows the transport protocol to ensure delivery of that message to all available nodes.

Messages are used by nodes to:

Initiate a membership join when a node is first starting its cluster processes
Establish a new membership after a node has stopped responding, or something has disrupted the ongoing heartbeat communications in the cluster
Communicate states and information transmitted by applications through CPG

Available Transport Protocols

Protocol: knet (a.k.a. kronosnet)

Description: Nodes transmit cluster-wide messages individually to each node in the membership list. knet can use up to 8 communication addresses (links) per node.

Available Releases:

RHEL 8 and 9: Default protocol
RHEL 5, 6 and 7: Not available

How to configure:

RHEL 8 and 9: In /etc/corosync/corosync.conf:
```
totem {
    transport: knet
}
```

Protocol: udp Over Multicast

Description: Nodes transmit cluster-wide messages to a specific multicast address chosen for this cluster, causing the network switch to deliver that message to any other nodes which have subscribed to the address. All nodes that are up and started would be subscribed to that address and thus will receive that message.

Available Releases:

RHEL 8 and 9: Not supported
RHEL 7: Available
RHEL 5 and 6: Default protocol

How to configure:

RHEL 7: In /etc/corosync/corosync.conf:

totem {
    transport: udp
    broadcast: no  # Optional - broadcast defaults to "no", which enables multicast
}

RHEL 5 and 6: The "udp over multicast" protocol is the default, and thus does not need to be explicitly configured, but could be specified in /etc/cluster/cluster.conf as:
```
<cluster [...] >
    <cman transport="udp" broadcast="no"/>
    [...]
</cluster>
```

Protocol: udp Over Broadcast

Description: Nodes transmit cluster-wide messages to the broadcast address for their local subnet, causing the network switch to deliver that message to all other hosts on the same subnet, which should include all nodes of the cluster. RRP is not supported when enabling broadcast with udp (as it could lead to the cluster behaving in unexpected ways).

Available Releases:

RHEL 8 and 9: Not supported
RHEL 5, 6, and 7: Available

How to Configure:

RHEL 7: In /etc/corosync/corosync.conf:
```
totem {
    transport: udp
    broadcast: yes
}
```
- RHEL 6: The udp transport is the default in this release, but broadcast defaults to disabled so would have to be explicitly enabled. Specifying both settings in /etc/cluster/cluster.conf would look like:
```
<cluster [...] >
    <cman transport="udp" broadcast="yes"/>
    [...]
</cluster>
```
- RHEL 5: There is no transport setting available in RHEL 5, as udp is built-in as the only choice. broadcast would need to be enabled explicitly in /etc/cluster/cluster.conf:
```
<cluster [...] >
    <cman broadcast="yes"/>
    [...]
</cluster>
```

Protocol: udpu (a.k.a. UDP-Unicast)

Description: Nodes transmit cluster-wide messages individually to each node in the membership list.

Available Releases:

RHEL 8 and 9: Not supported
RHEL 7: Default protocol
RHEL 6 Update 2 and later: Available, starting with corosync-1.4.1-4.el6 and cman-3.0.12.1-23.el6
RHEL 5, RHEL 6 Prior to Update 2: Not available - use udp with multicast or broadcast.

How to Configure:

RHEL 7: udpu is the default transport, and thus does not need to be configured explicitly, but can be specified in /etc/corosync/corosync.conf with:
```
totem {
    transport: udpu
}
```
- RHEL 6 Update 2 and later: In /etc/cluster/cluster.conf:
```
<cluster [...] >
    <cman transport="udpu"/>
    [...]
</cluster>
```

Technical Implementation and Usage Details

udp over multicast

The udp transport handles cluster-wide message delivery by relying on a network facility to do the heavy lifting of sending the message out to its various targets. The default mode that udp operates in is to utilize IP multicasting on the network. Multicast is a facility implemented on networks to allow hosts to subscribe to an address in a special IP range (224.0.0.1 - 239.255.255.255), then receive any subsequent packets trasmitted to that address.

In a RHEL cluster, this multicast facility used by corosync or openais to distribute messages to the various nodes in that cluster. When a node starts up in the cluster, it will first determine what multicast address this cluster should use - which is either explicitly specified in the configuration, or will be derived as a hash of the cluster name; since all nodes in the cluster should share the same configuration, all of them will know to interact with the same multicast address. A node that is starting in the cluster will use the IGMP protocol to join the multicast group for that address, which then alerts the switch that it should forward messages destined for that that group to this particular host. Any nodes that have started will have done the same, so now as they send messages out, all of them should receive them.

udp over broadcast

The udp transport over broadcast works very similarly to the "udp over multicast" mode described above, but instead of using multicast groups to deliver messages to just the nodes in the cluster, it uses the local subnet's broadcast address to transmit those messages to all hosts. There is no need for corosync or openais to subscribe to a group so the switch knows to deliver messages to that node; rather, the switch will deliver any packets sent to the broadcast address to all hosts with an address on that subnet. Any cluster nodes that are active in the cluster or are trying to join should be listening for such messages.

udpu

udpu works differently than the udp transport in that it does not rely on any special network facility to distribute messages to the desired hosts. Rather, the udpu transport will take a message and craft packets to be sent directly to each node in the configured membership list, one-on-one. This is useful when multicast or broadcast facilities may not be available on the network, or the complexity of setting them up is not desired.

With udpu, no special facilities are needed, as a node that has a cluster-wide message will simply distribute it to those other nodes itself. However this comes with a tradeoff, in that more processing will be done on the host as opposed to dedicated network hardware. The volume of messages and the load on the network hardware are the primary factors that usually dictate whether that tradeoff is acceptable or could be suboptimal.

knet

Knet is a lightweight TCP transport protocol that supports cross-platform multi-threading. As such udpu, knet doesn't do multicasting. The knet transport supports IPv4 and IPv6 addresses concurrently, provided they are consistent on each link. knet supports multi-homing up to 8 communication addresses for each node, called links which can also be configured on different priorities.

To understand further on This content is not included.how to configure corosync with kronosnet, please check the guide linked here with different parameters and the man corosync.conf which will include information on each option that can be used.