After network MTU change, some VMs and bridges have the old MTU and seeing packet drops

Solution Verified - Updated

Environment

  • Red Hat Virtualization 4.x

Issue

  • Some of the hosts are showing the MTU as 1500 causing problems for VMs using MTU of 9000 as they cannot use jumbo frames.
  • After changing the MTU of the network, some VMs and Hosts still have the old MTU
  • MTU settings on a network is inconsistent.

Resolution

Ideally update the MTU of the network with all Virtual Machines using it down or with their NICs unplugged. Only start them again (or plug the NICs) after the MTU has been updated.

If the MTU was already updated with VMs running, find the Virtual Machines running with old MTU (see Diagnostic Steps). For these VMs, e each Virtual NIC must be unplugged and plugged again in order to refresh to the new MTU, or power cycle the Virtual Machine. Before doing this ensure the bridge is set to the new MTU as well before plugging or power cycling the VM on a different host.

Do not migrate VMs that are still using the old MTU, as it will cause more problems.

Root Cause

  • Virtual Machines will only get the new MTU once they are power cycled or their vNICs refreshed by unplug + plug again.
  • Migrating a Virtual Machine still running with old MTU will set the bridge on the destination host to the same MTU, regardless of they MTU being updated to a new value before.
  • Bridge MTU is not updated in case there are VMs running attached to it.

There are 2 improvements in RHV being tracked in the following Bugzillas for this issue:
This content is not included.Hint after updating mtu on networks connected to running VMs
This content is not included.Document that updating a MTU of a network requires restarting the VM

Diagnostic Steps

  • The ip link show command on each hypervisor shows different MTU to what is configured in RHV on tap and bridge devices.
  • An easy way to find the current MTU of each Virtual Machine is to run this command on each host:
$ egrep "source bridge=|mtu size" /var/run/libvirt/qemu/*.xml
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.