fence_ipmilan fails with "Chassis power = Unknown" in RHEL 5 or 6 or RHEV

Environment

• Red Hat Enterprise Linux (RHEL) 5 or 6 with the High Availability Add On
  • One or more nodes configured to use fence_ipmilan
• Red Hat Enterprise Virtualization (RHEV)
  • One or more hypervisors configured to use fence_ipmilan

Issue

• Running fence_ipmilan results in a failure
• Cluster nodes fail to fence another using fence_ipmilan

# fence_ipmilan -a 192.168.2.10 -l admin -p password -o status
Getting status of IPMI:192.168.2.10...Chassis power = Unknown
Failed

• HP ILO4, fence_ipmilan fencing is not working
• RedHat Cluster Suite - ILO fencing is not working

Resolution

• Open UDP port 623 in the Firewall.

Root Cause

• Unable to establish IPMI v2 / RMCP+ session . Server not reachable or UDP port 623 is blocked in FW.

Diagnostic Steps

• Determine if the node that is doing the fencing can connect to the device in question. Usually ping is the easiest way to do this:

# ping -c1 node2-ilo.example.com
PING node2-ilo.example.com (192.168.2.10) 56(84) bytes of data.
64 bytes from node2-ilo.example.com (192.168.2.10): icmp_seq=1 ttl=64 time=1.55 ms

--- node2-ilo.example.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 3ms
rtt min/avg/max/mdev = 1.556/1.556/1.556/0.000 ms

• If this fails, check that the hostname/IP is correct, that the device has network connectivity, that there is no firewall blocking access, that the network configuration is correct, etc.

• Ensure IPMI v2 / RMCP+ session getting established. This communication happens over UDP port 623.

# tshark -i - <  tcpdump_UDP_623.pcap
  0.000000 xx.xx.xxy.yy -> yyy.zzz.xx.yy IPMI/ATCA Req, Get Channel Authentication Capabilities, seq 0x00
  0.011319 yyy.zzz.xx.yy -> xx.xx.xxy.yy IPMI/ATCA Rsp, Get Channel Authentication Capabilities, seq 0x00
  0.011394 xx.xx.xxy.yy -> yyy.zzz.xx.yy RMCP+ Session ID 0x0, payload type: RMCP+ Open Session Request
  0.012560 yyy.zzz.xx.yy -> xx.xx.xxy.yy RMCP+ Session ID 0x0, payload type: RMCP+ Open Session Response
  0.012713 xx.xx.xxy.yy -> yyy.zzz.xx.yy RMCP+ Session ID 0x0, payload type: RAKP Message 1
  0.013910 yyy.zzz.xx.yy -> xx.xx.xxy.yy RMCP+ Session ID 0x0, payload type: RAKP Message 2
  0.013972 xx.xx.xxy.yy -> yyy.zzz.xx.yy RMCP+ Session ID 0x0, payload type: RAKP Message 3
  0.015423 yyy.zzz.xx.yy -> xx.xx.xxy.yy RMCP+ Session ID 0x0, payload type: RAKP Message 4
  0.015498 xx.xx.xxy.yy -> yyy.zzz.xx.yy RMCP+ Session ID 0x277b5055, payload type: IPMI Message
  0.017784 yyy.zzz.xx.yy -> xx.xx.xxy.yy RMCP+ Session ID 0xa0a2a3a4, payload type: IPMI Message
  0.017863 xx.xx.xxy.yy -> yyy.zzz.xx.yy RMCP+ Session ID 0x277b5055, payload type: IPMI Message
  0.020882 yyy.zzz.xx.yy -> xx.xx.xxy.yy RMCP+ Session ID 0xa0a2a3a4, payload type: IPMI Message
  0.020996 xx.xx.xxy.yy -> yyy.zzz.xx.yy RMCP+ Session ID 0x277b5055, payload type: IPMI Message
  0.024338 yyy.zzz.xx.yy -> xx.xx.xxy.yy RMCP+ Session ID 0xa0a2a3a4, payload type: IPMI Message

• If tcpdump shows Invalid user name, ensure management user has been created. Following is the tcpdump output when no such user is created. This may happens with blade servers. CMM ( Chassis Management) user is configured and admin is trying with that user. System must have individual IMM (Blade Management) console user.

  0.000000  xxx.yy.zz.xx -> zzz.xx.yy.zz ASF 54 Presence Ping
  0.000526 zzz.xx.yy.zz -> xxx.yy.zz.xx  ASF 70 Presence Pong
  0.000673  xxx.yy.zz.xx -> zzz.xx.yy.zz IPMI/ATCA 65 Req, Get Channel Authentication Capabilities, seq 0x01
  0.001325 zzz.xx.yy.zz -> xxx.yy.zz.xx  IPMI/ATCA 72 Rsp, Get Channel Authentication Capabilities, seq 0x01
  0.001492  xxx.yy.zz.xx -> zzz.xx.yy.zz IPMI/ATCA 80 Req, Get Session Challenge, seq 0x02
  0.002021 zzz.xx.yy.zz -> xxx.yy.zz.xx  IPMI/ATCA 64 Rsp, Get Session Challenge, seq 0x02, Invalid user name

• Ensure that IPMI over LAN access is enabled in the administrative interface for the device

• Ensure that the user account in question is configured with administrative privileges in the device configuration

• Determine if fence_ipmilan can obtain status information from the device:

# fence_ipmilan -a node2-ilo.example.com -l Administrator -p password -o status

• If this does not work, some devices require use of the "lanplus" option (-P):

# fence_ipmilan -a node2-ilo.example.com -l Administrator -p password -P -o status

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.