Windows Guest migrated from VMware shows high CPU utilization

Environment

• Microsoft Windows
  • Virtualization Based Security (VBS) enabled
• Red Hat OpenShift Container Platform 4.x
  • OpenShift Virtualization 4.x

Issue

• Windows Guest has High CPU usage, not responding well
• Even when idle CPU usage in the Guest is very high
• High CPU usage in the OCP node
• Windows Guest migrated from VMware has high CPU utilization when running in OpenShift.
• Intel CPUs.

Resolution

• Enabling VBS causes a performance overhead due to the nested virtualization. Using HVCI (memory integrity) also causes a performance hit. If these are not required please consider disabling them for best performance.

• If VBS and/or HVCI are required, enable the EVMCS hyper-v enlightenment in the Virtual Machine spec:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
spec:
  template:
    spec:
      domain:
        features:
          hyperv:
            evmcs: {}      <--- add to the list

• Further hyper-v enlightenments are needed to alleviate the performance impact (such as Direct TLB Flush) but they are not available yet in OpenShift Virtualization (see Root Cause section).

NOTE: adding this to a Virtual Machine that does not run any hypervisor itself should not cause any harm. So it can be added to standard user templates and boot sources if needed.

Root Cause

• Lack of EVMCS enlightenment makes nested virtualization required by VBS sub-optimal as it triggers too many exists on the L0 host when accessing VMCS.
• Improvements to automatically enable this are being tracked in This content is not included.CNV-69067 - Sub-optimal performance of Windows with VBS enabled due to lack of EVMCS.
• On top of enabling EVMCS, further performance improvements are available via enabling Direct and Extended TLB Flushes enlightenments. These are not currently available in the VM configuration, and are being tracked in This content is not included.CNV-69327 - Missing knob to control hv-tlbflush-direct and hv-tlbflush-extended. Once these are available, enable them as well. For alternate ways of enabling these (for testing only), please contact Red Hat Support.

Diagnostic Steps

1. Confirm in the Windows settings that it was VBS enabled (or WSL or Hyper-V), otherwise this does not apply.
2. Using kvm_stat on the qemu-kvm PID of the VM for 1 second shows a high number of VM exists due to VMCS access:

kvm_exit(VMREAD)                              131321    130961
kvm_exit(VMRESUME)                             36432     36330
kvm_exit(VMWRITE)                             311990    311135

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.