KVM SR-IOV devices with high latency and messages like "Spurious APIC interrupt (vector 0xFF) on CPU#0, should never happen"

Environment

• Red Hat Enterprise Linux 9.2 KVM Host
• nVidia GPU Passthrough to KVM Guest
• SR-IOV NIC passthrough to KVM Guest (eg: Intel, Mellanox, Broadcom, etc)

Issue

• Virtual Machine dmesg shows many errors such as below:

Spurious APIC interrupt (vector 0xFF) on CPU#0, should never happen.

• Virtual Machine takes a very long time to initialise SR-IOV devices on reboot, or hangs.
• SR-IOV Network Interfaces have high latency after a Guest reboot

Resolution

Upgrade the KVM hypervisor to one of the following packages or later:

There are also several workarounds available.

Workaround - Disable APICv on the hypervisor

Disable APIC Virtualization (APICv) in the Intel KVM module configuration on the hypervisor:

/etc/modprobe.d/kvm_intel.conf

options kvm_intel enable_apicv=0

Rebuild the initramfs and reboot.

Restart the guest.

Workaround - Don't reboot the guest

When restarting an SR-IOV guest, fully "power off" the guest and start it again.

Avoid rebooting the guest from within (eg: using the reboot command).

This will ensure each time the guest boots it is running on a new qemu-kvm process, avoiding this problem.

Root Cause

This is a problem with interrupt routing through KVM.

It was investigated on Red Hat Private Bug 2149989 - VM dmesg report "Spurious APIC interrupt (vector 0xFF) on CPU#2, should never happen" and found to be resolved with a rebase of kernel KVM with Red Hat Bug 2177720 - Rebase KVM x86 to 6.3-rcX. This was included in development package kernel-5.14.0-334.el9 and later.

Just the 33 commits which fix the high-latency in SR-IOV were backported to RHEL 9.2 on Red Hat Private Bug 2218871 - kvm/sriov: high latency after soft reboot [rhel-9.2.0.z] and included in package kernel-5.14.0-284.25.1.el9_2 and later.

Diagnostic Steps

Guest logs shows delays and failures on soft reboot of Guest, drivers unable to initialise properly and quickly:

[    4.809349] mlx5_core 0000:08:00.0: firmware version: 16.30.1004
[    4.981777] mlx5_core 0000:08:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 24414Mbps
[    4.997313] mlx5_core 0000:08:00.0: Assigned random MAC address aa:aa:aa:aa:aa:aa
[    4.998904] mlx5_core 0000:08:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[   ***] A start job is running for Wait for…ce Initialization (13s / no limit)
[   18.013061] mlx5_core 0000:08:00.0: Supported tc offload range - chains: 1, prios: 1
[   18.025080] mlx5_core 0000:09:00.0: firmware version: 16.30.1004
[    **] A start job is running for Wait for…ce Initialization (14s / no limit)
[   18.925496] mlx5_core 0000:09:00.0: Assigned random MAC address aa:aa:aa:bb:bb:bb
[  *** ] A start job is running for Wait for…ce Initialization (23s / no limit)
[***   ] A start job is running for Wait for…ce Initialization (24s / no limit)
[   28.744989] mlx5_core 0000:0a:00.0: firmware version: 16.30.1004
[**    ] A start job is running for Wait for…ce Initialization (24s / no limit)
[   29.688120] mlx5_core 0000:0a:00.0: Assigned random MAC address aa:aa:aa:cc:cc:cc
[ ***  ] A start job is running for Wait for…ce Initialization (47s / no limit)
[   52.305569] mlx5_core 0000:0a:00.0: Supported tc offload range - chains: 1, prios: 1
[   52.317172] mlx5_core 0000:0b:00.0: firmware version: 16.30.1004
[        54.188479] mlx5_core 0000:0b:00.0: Assigned random MAC address aa:aa:aa:dd:dd:dd
[   54.190740] mlx5_core 0000:0b:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  *** ] A start job is running for Wait for…nitialization (1min 9s / no limit)
[   ***] A start job is running for Wait for…itialization (1min 11s / no limit)
[   76.205704] mlx5_core 0000:0a:00.0 enp10s0: renamed from eth2
[   76.217822] mlx5_core 0000:0b:00.0 enp11s0: renamed from eth3
[   76.228624] mlx5_core 0000:08:00.0 enp8s0: renamed from eth0

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.