corosync does not run with a realtime scheduling priority in a RHEL 7 High Availability cluster

Environment

• Red Hat Enterprise Linux (RHEL) 7 with the High Availability Add On
• corosync releases prior to 2.3.4-4.el7_1.3
  • On those releases, COROSYNC_OPTIONS in /etc/sysconfig/corosync does not include -r

Issue

• corosync ran with the SCHED_RR realtime scheduling policy in RHEL 6, but now runs with the SCHED_OTHER policy using a nice value of 20.
• My cluster nodes are experiencing frequent fencing or "processor failures" in the cluster whenever there is a minor amount of load placed on the nodes
• Cluster nodes are being fenced frequently

Resolution

• Update to corosync-2.3.4-4.el7_1.3 or later.

  • NOTE: This change will not take effect until corosync is restarted, or the node is stopped and started with pcs cluster stop and pcs cluster start.
  • NOTE: When you see the issue even if the system runs the fixed version or later, you should check whether cgconfig.service is enabled and active on the system. If cgconfig.service is enabled, you need to disable the service and restart the system.

• Workaround: In /etc/sysconfig/corosync, add -r to the COROSYNC_OPTIONS:

  COROSYNC_OPTIONS="-r"
  

  • NOTE: This change will not take effect until corosync is restarted, or the node is stopped and started with pcs cluster stop and pcs cluster start.

Root Cause

This issue is being investigated by Red Hat in Bugzilla #1170347 and was corrected in an asynchronous erratum for RHEL 7 Update 1 in Bugzilla #1260002.

The implementation of corosync in RHEL 7 did not include the same measures it did in RHEL 6 to ensure it runs with a realtime priority, so it is configured by default to run as a normal process in the SCHED_OTHER round-robin time-sharing policy. This may result in many more processes being scheduled ahead of corosync where they would not have been in earlier releases, and this may make a cluster node more likely to fail to send its token to another node in time, thus resulting in it being fenced more frequently. corosync does allow setting it to run in realtime priority with SCHED_RR by having the -r option passed into the daemon when it is started. The /etc/sysconfig/corosync file offers a way to set this option so that corosync uses that policy.

Diagnostic Steps

• Review ps -eo cmd,rtprio output to determine if corosync lists a dash ('-') under RTPRIO, then the system is susceptible. If it lists a number, then corosync is being properly scheduled with a realtime priority.

## Example: corosync not being scheduled properly with a realtime priority
#  ps -eo cmd,rtprio | grep -e [c]orosync -e RTPRIO
CMD                         RTPRIO
corosync                        -

## Example: corosync being scheduled properly with a realtime priority
#  ps -eo cmd,rtprio | grep -e [c]orosync -e RTPRIO
CMD                         RTPRIO
corosync                        99

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.