Increased softirq usage in RHEL 7.3 and RHEL 7.4 kernels when compared to prior RHEL 7 kernels; related to setting CPU affinity (pinning)

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 7
  • kernel-3.10.0-327.el7
  • kernel-3.10.0-514.el7

Issue

  • After updating the kernel from 3.10.0-327.el7 series to a newer 3.10.0-514.el7 or 3.10.0-693.el7 series kernel, application performance suffers. This is seen in an environment where IRQs are pinned to specific CPUs.

Resolution

  • Set the rcu_nocbs kernel command line parameter to encompass all available CPU cores. For example, for a 16 core system use rcu_nocbs=0-15.

Root Cause

  • This change is a side effect of a build-time change in the kernel between RHEL 7.2 and RHEL 7.3. Previously, the CONFIG_RCU_NOCB_CPU_ALL kernel configuration option was set for RHEL 7 kernels. From 7.3 onwards, CONFIG_RCU_NOCB_CPU_ALL is no longer set.

  • Setting the rcu_nocbs kernel command line option at boot for all available CPU cores should mimic the same behaviour as having CONFIG_RCU_NOCB_CPU_ALL set at build time.

  • Historically, the CONFIG_RCU_NOCB_CPU_ALL kernel configuration compile-time option was only enabled in RHEL kernels not because it was desired, but because it was required by another dependency. In the interim between RHEL 7.2 and RHEL 7.3, this dependency was resolved, allowing the CONFIG_RCU_NOCB_CPU_ALL option to be disabled at build time.

  • CONFIG_RCU_NOCB_CPU_ALL is a build-time option which forces RCU callback processing for every CPU core to be "offloaded" as kernel threads (rcuo threads). While this is desirable for some workloads, it is not good for others. The major downside of the CONFIG_RCU_NOCB_CPU_ALL option is that there is no boot-time option to disable it, so it is imposing a configuration choice on every system which cannot be overridden.

  • When CONFIG_RCU_NOCB_CPU_ALL is not set at compile time, then RCU callback process offloading can still be enabled for some or all cores with the boot-time rcu_nocbs kernel command line option. In other words, rcu_nocbs is a boot-time configuration choice which can enable the kernel to behave in exactly the same way as if the CONFIG_RCU_NOCB_CPU_ALL option had been enabled at compile time.

  • Given the above, it should be clear that having CONFIG_RCU_NOCB_CPU_ALL disabled at compile time produces a more configurable kernel.

  • Please see the kernel's own documentation for a more thorough explanation of both the CONFIG_RCU_NOCB_CPU_ALL compile-time option and rcu_nocbs boot option:

  • This issue is now described in the RHEL 7.3 Release Notes Known Issues section at Chapter 57:

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.