fence_ipmilan fails with 'ExpectToken returned -1' when using -vv or higher verbosity in RHEL 5

Environment

• Red Hat Enterprise Linux (RHEL) 5 with the High Availability Add On
• Executing fence_ipmilan from the command line with -vv (or higher verbosity) specified.
  • This issue should not affect the agent when called automatically by fenced, or when called via fence_node

Issue

• When I run fence_ipmilan with -vv or higher verbosity, it fails. When I run fence_ipmitool directly, it works.

# fence_ipmilan -a 192.168.2.10 -l fence -p administrator -vvv -P
Rebooting machine @ IPMI:192.168.2.10...Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.2.10' -U 'fence' -P 'administrator' -v -v -v chassis power status'...
Spawned: '/usr/bin/ipmitool -I lanplus -H '192.168.2.10' -U 'fence' -P 'administrator' -v -v -v chassis power status' - PID 3821
Looking for: 
    'Password:', val = 1
    'Unable to establish LAN', val = 11
    'IPMI mutex', val = 14
    'Unsupported cipher suite ID', val = 2048
    'read_rakp2_message: no support for', val = 2048
    'Chassis Power is off', val = 4096
    'Chassis Power is on', val = 8192
ExpectToken returned -1
ExpectToken failed.  Info returned:
>>>>>
IPMI LAN host 192.168.2.10 port 623

>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 
[...]
Chass
>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x3c
>>    data    : 0xee 0xa1 0x08 0x00 

BUILDING A v2 COMMAND
>> Initialization vector (16 bytes)
 d2 17 b0 cd e8 6f 96 03 8e bd 63 84 9a ba 22 15
authcode input (48 bytes)
 06 c0 ee a1 08 00 05 00 00 00 20 00 d2 17 b0 cd
 e8 6f 96 03 8e bd 63 84 9a ba 22 15 8f 1b 81 23
 f0 2f 94 51 a6 68 54 84 16 a1 fd bd ff ff 02 07
authcode output (12 bytes)
 5f 95 e0 64 3b 65 d3 9d 74 e1 f4 9b
<< IPMI Response Session Header
<<   Authtype                : Unknown (0x06)
<<   Payload type            : IPMI (0)
<<   Session ID              : 0xa0a2a3a4
<<   Sequence                : 0x00000003
<<   IPMI Msg/Payload Length : 32
<< IPMI Response Message Header
<<   Rq Addr    : 81
<<   NetFn      : 07
<<   Rq LUN     : 0
<<   Rs Addr    : 20
<<   Rq Seq     : 03
<<   Rs Lun     : 0
<<   Command    : 3c
<<   Compl Code : 0x00
Closed Session 0008a1ee

is Power is on

<<<<<
Error = 2 (No such file or directory)
Reaping pid 3821
Failed

• ipmitool seems to split some output lines and mix in debug output when using -vvv.

Resolution

• Do not specify any more than a single -v:

# fence_ipmilan -a 192.168.2.10 -l admin -p admin -P -v -o status
Getting status of IPMI:192.168.2.10...Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.2.10' -U 'admin' -P 'admin' -v chassis power status'...
Chassis power = On
Done

Root Cause

• It seems that when fence_ipmilan -vv or higher verbosity is specified, the resulting ipmitool -v -v will intersperse debug output with regular output. Notice in the example output in the Issue section above that the "Chassis Power is on" line is split in two, with "Chass" being on one line, followed by verbose debug output, followed by "is Power is on". - Because fence_ipmilan actually parses the ipmitool output for expected strings, this breaks the parsing and fence_ipmilan cannot determine the power status.

When configured in /etc/cluster/cluster.conf this is not an issue, because only one level of verbosity is available in the STDIN-versions of the options (verbose="1"). In other words, when called by fenced or fence_node, there is no way to specify it to run with -vv or higher, and thus the issue does not occur.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.