Exploring RHEL High Availability's Components - corosync

Updated

Applicable Environments

  • Red Hat Enterprise Linux (RHEL) 6, 7, 8 or 9 with the High Availability Add-On

Component Overview

Uses

corosync is the component - and a daemon of the same name - that serves the core membership and member-communication needs for RHEL 6, 7, 8 and 9 High Availability clusters. It is required for the High Availability Add-On to function in these releases.

In addition to those membership and messaging functions, it also:

  • Manages quorum rules and determination in RHEL 7, 8 and 9 clusters through a layer known as votequorum, whereas this function is provided by a separate component - cman - in RHEL 6 clusters.
  • Provides messaging capabilities for applications that coordinate or operate across multiple members of the cluster and thus must communicate stateful or other information between instances. This layer is known as CPG - Closed Process Groups.
  • Provides a checkpoint layer known as CKPT which allows for maintaining stateful information in memory across nodes, so that it can be played back if needed. This layer was retired on RHEL 7 in favor of using features in CPG, but some components of RHEL High Availability and Resilient Storage still use it - most notably cmirror.

Technical Implementation and Usage Details

General

The corosync daemon is the foundation on which the rest of the components in the High Availability Add-On rely on to perform their work - the components of a cluster tend to have some basis in operating across nodes, so they need to be looped in on changes in membership, and may need methods to communicate with their peer-instances distributed around the cluster. corosync provides this base membership and communication layer in a way that gives the other components a consistent view of membership, rather than having them all reinvent the wheel to track which nodes are online and need to be considered or communicated with.

When the corosync daemon is started, it loads the configuration which defines the methods for communicating between members, the list of known members, details of how quorum should be decided, and more. With this information in hand, it sets out to establish a cluster membership by communicating using the configured methods with any other nodes that are available to operate in this particular cluster. Using that shared membership those nodes have established, they communicate with each other to relay their individual states, to monitor each other through a heartbeat mechanism, and to pass application data between themselves via CPG messages. The communication protocols in use work to ensure that messages get delivered and processed in the correct order, and that any faults are handled in the appropriate manner.

The other components of the RHEL High Availability Add-On which carry out most of the work a user might be aware of - resource management, coordinating shared LVM access, fencing, etc - base their decisions on information coming from corosync. These components provides hooks, or "callbacks", into corosync which results in them being notified when there is a membership change or a quorum change. Those components then take action in response to such changes when they happen, such as fencing a member that has left the cluster, or recovering resources from a fenced member, or coordinating with a newly-joined member's components.

RHEL 8 and 9

In RHEL 8 and 9 was introduced the corosync 3 cluster engine that has the functions very similarly to RHEL 7's corosync 2.

It introduces a new type of transport protocol, the Kronosnet (knet) network abstraction layer for cluster communication with up to 8 communication address per node, which does not work on RHEL 7's clusters, being this one of the reasons in-place upgrades are not supported within major versions and why clusters nodes with different major versions can not run within one single cluster.

Corosync 3 as such corosync 2 from RHEL 7 continues with the same functions to establish membership and for continuous cluster nodes communication. The major differences in cluster creation and administration between RHEL 7 and RHEL 8 are listed on the considerations in adopting RHEL 8.

RHEL 7

RHEL 7's corosync is configured directly and can operate as a standalone unit - although tools like pcs abstract some of the details of that arrangement. corosync's configuration file resides at /etc/corosync/corosync.conf, and defines important items like: the member list, the transport protocol, redundancy protocol, quorum features and rules, communication encryption, timers and settings for communication monitoring and reactions, and more.

In this release, the corosync daemon is started using the systemd corosync.service unit. Again, using pcs may abstract this detail, in that a pcs cluster start will call systemctl start corosync.service under the hood, but corosync can also be started manually through systemctl. When it starts up, it establishes the membership with any other available members, and then it is up to other components - like pacemaker - to tie in to corosync to utilize the membership that has been established and carry out additional work.

A key feature available through RHEL 7 corosync is its votequorum subcomponent and API. votequorum is the name of the portion of corosync that manages quorum - which is a term describing the rules upon which the cluster members decide if they are capable of providing service, usually by being amongst a majority of members that can communicate with each other. votequorum offers various features for controlling how the cluster makes these decisions, such as allowing for half of members in a split to remain functional, or allowing a single "last man standing" node to remain functional even if the others all go missing.

RHEL 6

RHEL 6's corosync functions very similarly to RHEL 7's at its core layers, as far as establishing membership and passing communications. However, it ties into the structure of the cluster components differently in this release. The main cluster configuration is at /etc/cluster/cluster.conf, and is generally bound by rules and implementation specific to the cman component. corosync itself is available as a standalone service and process - similar to how it exists in RHEL 7 - but Red Hat's High Availability Add-On only supports utilizing it by way of cman. Internally, corosync offers a number of features and settings, but those are configured through cman's configuration using its settings and terminology, and then cman interprets those and loads them into corosync. So the point is: the corosync daemon will still be started in RHEL 6 clusters and it will provide similar or the same functions as other releases, but it is controlled by way of cman.

corosync gets loaded when the cman init service is started. This service loads various components and modules, then calls the cman_tool join utility to get everything loaded - this causes cman's configuration parser to load corosync settings into memory, then kicks off the corosync daemon to carry out all of the membership and communication work noted above. cman goes thorugh a similar process to stop corosync.

Notable Sub-components and Provided Files

  • /usr/sbin/corosync: The main cluster engine daemon
  • /etc/corosync/corosync.conf: Main cluster configuration file (applicable only in RHEL 7 HA clusters)
  • /etc/sysconfig/corosync: Environment configuration for corosync - startup settings, daemon settings, etc
  • /usr/sbin/corosync-cmapctl: Displays the configuration database for the active cluster instance
  • /usr/sbin/corosync-cpgtool: Displays active CPG groups, their connected PIDs, and the members currently joined to each.
  • /usr/sbin/corosync-quorumtool: Displays details of membership and quorum
  • /usr/sbin/corosync-blackbox: Outputs corosync's in-memory ringbuffer of all recorded log messages up to the highest debug level. Useful for debugging internal or problems or unexpected behaviors.

Additional Knowledge Content

SBR
Product(s)
Category
Components
Tags
Article Type