Why does the multicast traffic among virtual machines quit working?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 6.5
  • Red Hat Enterprise Linux 5.9 onwards
  • KVM virtualization or Red Hat Enterprise Virtualization
  • Bridged virtual machine network interfaces
  • Multicast traffic between guests

Issue

  • Why does the multicast traffic among virtual machines quit working after upgrade to RHEL 6.5?
  • In a virtualized environment based on KVM hypervisor the flow of multicast traffic between the Virtual Machines stops after a constant amount of time.
  • Multicast traffic fails to reach the intended destinations after a fixed time has passed.
  • Multicast packets can not be received on the guest from other guests

Resolution

RHEL-based resolution

Upgrade to the following packages:

  • RHEL 6.6 - kernel-2.6.32-504.el6
  • RHEL 6.5.z - kernel-2.6.32-431.23.1.el6
  • RHEL 5.11.z - kernel-2.6.18-400.el5

Switch-based resolution

Run an IGMP querier elsewhere on the LAN.

On a Cisco switch, this can be achieved with the following configuration:

interface vlan VLANID
 ip address IPADDR NETMASK
 ip igmp snooping querier

IGMP querying can be confirmed with the command:

show ip igmp interface vlan VLANID | include querier

Workaround

Enable Multicast Querying on the bridge. This can be done in several different ways.

In each method, replace BRIDGENAME with the actual name of the bridge where the virtual machines are connected, eg: br0 or rhevm.

Runtime

# echo 1 > /sys/class/net/BRIDGENAME/bridge/multicast_querier

Permanent - ifup-local

Create /sbin/ifup-local as described at How do I run a script or program immediately after my network interface goes up? and include contents such as:

#!/bin/bash
if [ "$1" == "BRIDGENAME" ]; then
  /bin/echo "1" > /sys/class/net/$1/bridge/multicast_querier
fi

Permanent - ifcfg

In the bridge configuration file /etc/sysconfig/network-scripts/ifcfg-BRIDGENAME add BRIDGING_OPTS to enable the multicast querier, for example:

DEVICE=BRIDGENAME
TYPE=Bridge
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=dhcp
DELAY=0
BRIDGING_OPTS="multicast_querier=1"

Permanent - udev

Create a udev rule at /etc/udev/rules.d/61-br-querier.rules containing:

ACTION=="add", SUBSYSTEM=="net", RUN+="/etc/sysconfig/network-scripts/vnet_querier_enable"

Create the helper-script at /etc/sysconfig/network-scripts/vnet_querier_enable containing:

#!/bin/sh
if [[ $INTERFACE == BRIDGENAME ]]; then
    /bin/echo 1 > /sys/devices/virtual/net/$INTERFACE/bridge/multicast_querier
fi

Root Cause

The upstream Linux kernel disabled multicast querying by default with patch Content from git.kernel.org is not included.c5c2326.

This patch was introduced in RHEL kernel 2.6.32-431 (RHEL 6.5). The change is mentioned in the RHEL 6.5 Release Notes:

kernel component
    In cluster environment, the multicast traffic from the guest to a host can be 
    unreliable. To work around this problem, enable multicast_querier for the
    bridge. The setting is located in the 
    /sys/class/net/<bridge_name>/bridge/multicast_querier file. Note that
    if the setting is not available, the problem should not occur.

The consequence is that the multicast group membership, not being refreshed, expires and the multicast traffic is not forwarded anymore to the intended destinations.

Setting the multicast_querier flag to 1 enables the bridge to send multicast queries again.

This change affects Red Hat Enterprise Virtualization hypervisors if the kernel is updated to one of the mentioned versions.

This was later changed upstream with patch Content from git.kernel.org is not included.b00589a, which enables multicast traffic to be delivered if no querier is present, or querying is not enabled.

This patch has been included in RHEL 5.11.z.

This patch has been included in RHEL 6.5.z and RHEL 6.6.

RHEL 7 never faced this issue, as b00589a was included in 7.0.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.