Reducing the performance impact of 'Meltdown' CVE fixes for OpenStack guests with "PCID" CPU feature flag

Solution Unverified - Updated

Environment

  • Red Hat OpenStack Platform.
  • x86_64 architecture.

Issue

Fixes for This content is not included.Meltdown security vulnerability variants can result in some performance slowdown for certain workloads, given the nature of those vulnerabilities and their fixes. This is well documented in the following related article:

Is it possible to reduce the performance impact 'Meltdown' fixes have on OpenStack Platform guests? Yes, it is — by ensuring the presence of a particular CPU feature flag called 'PCID' (Process Context ID) within the guest. More details on that in the following sections.

Resolution

How can PCID help lessen the impact on performance?

Ensuring that PCID is successfully enabled within OpenStack Platform guests can help to reduce the performance impact of 'Meltdown' CVE fixes. Without PCID in place, some CPU cache is discarded when switching from user space to kernel space and vice-versa (every system call does both of these). PCID enables indexing of CPU cache to a certain context (i.e. user space cache or kernel space cache). So, instead of CPU cache being discarded, CPU context changes and cache related to the previous context persists for use when its corresponding context is once again returned to.

On entering the kernel, context pcid is set to 1 and on returning to user space it is set to 0. This way, the cache keeps running, but when running in user space, cache loaded in kernel space cannot be used and vice-versa. This is more efficient than having to discard all translation caches at every round trip.

The net result of having PCID active is that overall, data cache pipelines can be utilized more efficiently, resulting in the need for fewer calls to memory or disk by comparison to not having PCID in place. As such, having PCID enabled within OpenStack Platform guests can help to lessen the impact 'Meltdown' CVE fixes may have on performance within those guests directly.

PCID support in CPUs, RHEL, libvirt, QEMU and Nova

Firstly, PCID must be available within the hardware.

RHEL supports PCID with two Address Space Identifiers (ASIDs) only, one for kernel space one for user space, so there is no Translation Lookaside Buffer (TLB) flush in the kernel entry/exit.

The ability to use the PCID feature to make user and kernel switches faster has been introduced into RHEL 6 and 7 respectively. This feature was tracked via the following Red Hat Bugzillas and corresponding advisories:

OpenStack Nova supports three distinct CPU modes:

  1. cpu_mode = host-model: OpenStack Nova defaults to using this, when
    virt_type=kvm|qemu is set in /etc/nova/nova/conf. Using CPU
    mode 'host-model' will do the right thing if you have the relevant
    libvirt and QEMU versions with 'Meltdown' fixes.

    NB: After updating libvirt and QEMU, for the fixes to take
    effect, you must do a 'cold reboot', meaning; explicitly stop the
    guest and start it.

  2. cpu_mode = host-passthrough

    When using host-passthrough CPU mode, if the PCID feature flag is
    available in the hardware, it will be passed through to the Nova
    guests.

  3. A specific custom CPU model — this can be enabled using the
    configuration attributes cpu_mode = custom plus cpu_model = IvyBridge (i.e. a named CPU model).

    (The list of all valid named CPU models are available in
    /usr/share/libvirt/cpu_map.xml.)

    When using this custom CPU mode, the only Intel CPU models that include the PCID capability are: 'Haswell', 'Broadwell', and 'Skylake' variants. Please note that the libvirt / QEMU Intel CPU models: 'Nehalem', 'Westmere', 'SandyBridge', and 'IvyBridge' will not expose the PCID capability, even if the host CPUs by the same name include it.

Other considerations

Please note; if host pass-through is enabled on OpenStack Platform compute nodes, guest live migration will only succeed across identical CPUs in the pool.

The following have been created upstream in order to request the ability to configure extra CPU flags for named CPU models in Nova, to allow for more granular specification of CPU feature flags. A mechanism to specify granular CPU flags already exists within libvirt and QEMU. The feature being reviewed via the links below relates to the the ability to configure those extra CPU flags within Nova, with granularity. Please note that these links are external to Red Hat and as such, we cannot validate their contents:

Root Cause

Because 'Meltdown' kernel side-channel attack vulnerabilities target optimizations inherent in modern CPU architectures, which were designed to make more efficient use of data cache pipelines, mitigation against those vulnerabilities can impact the performance of some workloads.

Diagnostic Steps

For guests running Red Hat Enterprise Linux (RHEL), please run the following command in order to verify whether PCID is active within the guest directly. If a response is received, then it has been enabled successfully:

grep '\bpcid\b' /proc/cpuinfo
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.