About the impact of virtio-net multi-queue on tx performance

The Red Hat Enterprise Linux 7 Virtualization Tuning and Optimization Guide says that multi-queue virtio-net can have a negative effect on your network performance. This article provides some further clarification with regards to this matter.

How multi-queue can hurt performance for outgoing traffic

Tests show a regression with up to 50% reduction in throughput for small packet (64B - 1024B) parallel TCP transmissions using netperf for multi-queue (mq) compared to single-queue (sq).

The reason for this phenomenon is that the virtio-net driver does not depend on tx interrupts to free transmitted packets. Instead, it orphans the socket buffer (SKB) during the transmit. This confuses socket accounting and leads to buffer bloats, since the stack cannot estimate the pending packets in both qdics and the NIC.

In cases of TCP transmission with small packets, TCP small queue does not work since socket accounting does not work. Due to this, multi-queue in fact transmits much more packets but of a much smaller size, leading to a lower overall throughput than in the case of single queue transmission.

Small packet sending in TCP streams is rare, as user-space applications usually execute large write/sendmsg calls for streaming traffic and these performance reductions do not affect UDP.

Due to the fact that these performance regressions can be treated as a corner case, Red Hat enables multi-queue by default in various setups.

(Future) Workarounds

At time of this writing, the upstream kernel supports napi_tx, which brings tx interrupts back. This could be a possible solution for this problem. Since the feature has not been backported to Red Hat Enterprise Linux 7, Red Hat did not test whether it solves the issue or not.

Transmit Packet Steering (XPS) could be a workaround: administrators can drive all tx traffic to a single tx queue. However, this is not a generic solution.