How can I test multicast traffic between two or more hosts?

Introduction

This article describes some of the basic debugging steps that can be used to troubleshoot multicast related issues within a cluster.

Environment

• Red Hat Cluster Suite 4+
• Red Hat Enterprise Linux 5 (with the High Availability Add on)
• Red Hat Enterprise Linux 6 (with the High Availability Add on)
• Red Hat Enterprise Linux 7 (with the High Availability Add on)
• Red Hat Enterprise Linux 8 (via rhel-8-for-x86_64-appstream-rpms)

What are the basic debugging commands that can be used?

• netstat -g should be used to verify the interfaces' multicast group memberships
• netstat -s shows a packet counter for multicast packets; this should increase when multicast traffic is received/sent
• tcpdump, 'wireshark' and other sniffing tools can be used to inspect multicast traffic
• ifconfig should show interfaces flagged MULTICAST; 'route' should show the multicast ranges

Using omping to generate and verify multicast traffic

• On Red hat Enterprise Linux 6.1+ the package 'omping' is available as part of the distribution. See 'man omping' for documentation.
• omping should be started on both nodes, taking the the ip of the remote node and the server's own ip as parameters:

# omping <remote IPs> <own ip>

Example output of omping with multicast traffic:

# omping 192.168.4.1 192.168.4.21  
 192.168.4.1 : waiting for response msg  
 192.168.4.1 : joined (S,G) = (\*, 232.43.211.234), pinging  
 192.168.4.1 :  unicast,  seq=1, size=69 bytes, dist=0, time=0.251ms  
 192.168.4.1 :  unicast,  seq=2, size=69 bytes, dist=0, time=0.373ms  
 192.168.4.1 : multicast, seq=2, size=69 bytes, dist=0, time=0.385ms  
 192.168.4.1 :  unicast,  seq=3, size=69 bytes, dist=0, time=0.476ms  
 192.168.4.1 : multicast, seq=3, size=69 bytes, dist=0, time=0.490ms  
 192.168.4.1 :  unicast,  seq=4, size=69 bytes, dist=0, time=0.423ms

• omping checks could fail if iptables rules are enabled on the nodes. Please disable iptables before doing the tests or make sure the port used by omping (4321 by default) is open (omping can be used with a custom port with the option -p)

Using multicast.py to generate and verify multicast traffic

• An alternative to omping, for versions of RHEL prior to 6.1, is multicast.py (attached).
• This script can be used on RHEL with the python interpreter installed.
• multicast.py -h documents the parameters. Please note the default values in square brackets [ ] if the option is not specified.
• Please note the default port 5405 is used by cman. So starting this script with the default port while cman is running will result in the script exiting with 'port allocation failure'. Please use port 5406 if cman is running.
• Run multicast.py on each node to send and receive multicast messages.

Set up a server (with option "-s") to receive multicast messages. Once the client sends the messages then output like the following will be displayed. In the example below -I refers to the interface to listen on, which should be the interface that the cluster traffic goes over, and -M is the multicast address (press Control+c to exit):

# ./multicast.py -I 192.168.0.1 -M 239.0.0.1 -s -p 5405  
 Mon, 04 Oct 2010 17:15:18 DEBUG  Bound: 127.0.0.1:5405  
 Mon, 04 Oct 2010 17:15:18 DEBUG  Joined the multicast network: 239.0.0.1 on 192.168.0.1  
 Mon, 04 Oct 2010 17:16:06 INFO   Received 'hello' (\#1) from 127.0.0.1:56339, time=0.001844  
 Mon, 04 Oct 2010 17:16:07 INFO   Received 'hello' (\#2) from 127.0.0.1:56339, time=0.003399  
 Mon, 04 Oct 2010 17:16:17 INFO   Exiting...  
 Mon, 04 Oct 2010 17:16:17 INFO   192.168.0.1:33585: Received = 2, (maybe) Lost = 0  
 Mon, 04 Oct 2010 17:16:17 DEBUG  Left the multicast network: 239.0.0.1 on 192.168.0.1

Set up one or more clients to send multicast messages (press Control+c to exit):

# ./multicast.py -I 192.168.0.2 -M 239.0.0.1 hello -p 5405  
 Mon, 04 Oct 2010 17:16:06 DEBUG  Joined the multicast network: 239.0.0.1 on 192.168.0.2  
 Mon, 04 Oct 2010 17:16:06 INFO   Sent data: '000000001 1286183766.53 hello'  
 Mon, 04 Oct 2010 17:16:07 INFO   Sent data: '000000002 1286183767.53 hello'  
 Mon, 04 Oct 2010 17:16:15 INFO   Exiting...  
 Mon, 04 Oct 2010 17:16:15 DEBUG  Left the multicast network: 192.168.0.2 on 127.0.0.1

Verify that multicast ttl is appropriate for the customers network environment

In RHEL 6 cluster it is possible to adjust the ttl for multicast packets. This may be larger for some networks that include multiple switches, and may need to be directly adjusted as shown here