fence_xvm command unable to list VMs (running on separate KVM hosts) and reports "Timed out waiting for response" message

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 7 (with High Availability or Resilient Storage Add On)

Resolution

  1. Firstly, check the IP subnet range assigned to the VM (running on the KVM host). Refer the HostMin and HostMax parameter values for the IP subnet range.

    # ipcalc <VM IP>/<VM IP netmask>
Address:	<VM IP>
Network:	<VM network start IP>/<VM IP netmask>
Netmask:	<VM IP netmask in CIDR> = <VM IP netmask>
Broadcast:	<Broadcast IP>
Address space:	Private Use
HostMin:	<IP subnet range start IP>		<<<====
HostMax:	<IP subnet range end IP>		<<<====
Hosts/Net:	<Total No of IPs in the subnet>

  2. Then, from the above IP subnet range, assign a unique IP to the bridge connection.
    NOTE : This unique IP assigned to the bridge connection must not be assigned to any other server or device in the same subnet. Please double-check the same to avoid any duplicate IP conflicts.

  3. The unique IP can be assigned to the bridge connection using the below command:

    # ip addr add <unique IP>/<same VM IP netmask> dev <bridge connection name>
where,
    replace <unique IP> with the *unique* IP assigned to the bridge connection
    replace <bridge connection name> with the bridge connection name
    replace <same VM IP netmask> with the correct netmask that is used by the VM

  4. Then, run the below commands, and check if successful TCP connection attempts from the KVM host to the VM are being reported using the "fence_virtd -Fd99" command ?
    Run below command on the KVM host:

    # systemctl stop fence_virtd.service
# fence_virtd -Fd99

    While the debug mode is on for the "fence_virtd" service (as above) on the KVM host, run the below command from the VMs in another terminal.

    # fence_xvm -a <multicast IP 1> -k /etc/cluster/<key 1> -o list
# fence_xvm -a <multicast IP 2> -k /etc/cluster/<key 2> -o list
where,
    replace <multicast IP 1> with the multicast IP address configured in /etc/fence_virtd.conf of the KVM host 1
    replace <multicast IP 2> with the multicast IP address configured in /etc/fence_virtd.conf of the KVM host 2
    replace <key 1> with the fence_xvm key file for KVM host 1
    replace <key 2> with the fence_xvm key file for KVM host 2

  5. Then, check the output of the "fence_virtd -Fd99" command for TCP connection which should get established between the KVM host and the VM.

  6. Next, press 'Ctrl-C' on the KVM host terminal to stop the debug mode for the "fence_virtd" service on KVM host.

  7. Then, start the "fence_virtd" service on KVM host again.

    # systemctl start fence_virtd.service

Root Cause

  • In general, it is not mandatory to assign IP to the bridge interface. However, in case of fence_xvm, the TCP connection from the KVM host to the VM could not be established since there was no direct path to connect to the VM. By assigning a unique IP to the bridge interface, we ensure that this TCP connection gets established further.

Diagnostic Steps

  1. Check that the firewall in disabled on the KVM hosts and VMs.

    # systemctl status firewalld.service

If firewall is not disabled, then add the firewall rules as below, on the KVM hosts and VMs, else ignore the below commands.

```
# firewall-cmd --permanent --add-port=1229/tcp
# firewall-cmd --permanent --add-port=1229/udp
# firewall-cmd --reload
```

  1. The "fence_xvm" commands are reporting error as "Timed out waiting for response.. Operation failed".

    [root@xxxx cluster]# fence_xvm -a <multicast IP 1> -k /etc/cluster/<key 1> -o list
Timed out waiting for response
Operation failed
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.